Mercurial > hgsubversion
view tests/fixtures/renames.svndump @ 931:e1dbd9646d6a
svnwrap: use custom StringIO class in get_file()
The wrappers were calling ra.get_file() with a cStringIO object.
Empirically, svn 1.7.5 is writing 16kB blocks to the stream object, and
cStringIO reallocates its internal buffer and doubles its size whenever
it is filled. With large committed files this requires two large
memory blocks at the same time.
SimpleStringIO implements the mimimum StringIO interface used by
ra.get_file() but instead stores all the blocks and "join" them at the
end. It means more fragmentation but requires only one large block,
without overallocation. Also, 16kB blocks should be friendly to most
allocators.
In practice, this simple change let me convert a revision containing
multiple moderately large files, the largest being around 450MB, with a
32-bits Windows setup, python 2.7, swig svn 1.7.5, in stupid mode, while
it was previously aborting with "not enough memory". The same revision
still fails in replay mode.
| author | Patrick Mezard <patrick@mezard.eu> |
|---|---|
| date | Sun, 16 Sep 2012 19:31:49 +0200 |
| parents | c2a84d436202 |
| children | 0d0132cba155 |
line wrap: on
line source
SVN-fs-dump-format-version: 2 UUID: 113560bd-ec36-42a6-acef-e4688a33b129 Revision-number: 0 Prop-content-length: 56 Content-length: 56 K 8 svn:date V 27 2008-12-05T22:48:38.139917Z PROPS-END Revision-number: 1 Prop-content-length: 114 Content-length: 114 K 7 svn:log V 12 init project K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:38.525864Z PROPS-END Node-path: branches Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Revision-number: 2 Prop-content-length: 113 Content-length: 113 K 7 svn:log V 11 add a and b K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:39.313010Z PROPS-END Node-path: trunk/a Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 60b725f10c9c85c70d97880dfe8191b3 Content-length: 12 PROPS-END a Node-path: trunk/b Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 3b5d5c3712955042212316173ccf37be Content-length: 12 PROPS-END b Node-path: trunk/changed Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 8 Text-content-md5: ec1bebaea2c042beb68f7679ddd106a4 Content-length: 18 PROPS-END changed Node-path: trunk/changeddir Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk/changeddir/f Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 9 Text-content-md5: 2dfdfd8492a2c558ec838d69f73f5f6b Content-length: 19 PROPS-END changed2 Node-path: trunk/da Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk/da/daf Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 2cd6ee2c70b0bde53fbe6cac3c8b8bb1 Content-length: 12 PROPS-END c Node-path: trunk/da/db Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk/da/db/dbf Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: e29311f6f1bf1af907f9ef9f44b8328b Content-length: 12 PROPS-END d Node-path: trunk/deleteddir Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk/deleteddir/f Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 11 Text-content-md5: 49b72b575e26ecddb296dd59b24c3e67 Content-length: 21 PROPS-END deleteddir Node-path: trunk/deletedfile Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 8 Text-content-md5: 4d742b2f247bec99b41a60acbebc149a Content-length: 18 PROPS-END deleted Node-path: trunk/groupdir Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk/groupdir/a Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 60b725f10c9c85c70d97880dfe8191b3 Content-length: 12 PROPS-END a Node-path: trunk/groupdir/b Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 3b5d5c3712955042212316173ccf37be Content-length: 12 PROPS-END b Node-path: trunk/unchanged Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 10 Text-content-md5: 85ae5b04dd0a666efad8633d142a4635 Content-length: 20 PROPS-END unchanged Node-path: trunk/unchangeddir Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk/unchangeddir/f Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 11 Text-content-md5: a11092875079a002afb9ecef07f510e7 Content-length: 21 PROPS-END unchanged2 Revision-number: 3 Prop-content-length: 123 Content-length: 123 K 7 svn:log V 21 delete files and dirs K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:40.224632Z PROPS-END Node-path: trunk/changed Node-kind: file Node-action: change Text-content-length: 16 Text-content-md5: 1725f40a29aad369a39b0f96c82d50f9 Content-length: 16 changed changed Node-path: trunk/changeddir/f Node-kind: file Node-action: change Text-content-length: 18 Text-content-md5: 984b8c4ab9193b7659b9f914897a949c Content-length: 18 changed2 changed2 Node-path: trunk/groupdir/a Node-kind: file Node-action: change Text-content-length: 4 Text-content-md5: 0d227f1abf8c2932d342e9b99cc957eb Content-length: 4 a a Node-path: trunk/deleteddir Node-action: delete Node-path: trunk/deletedfile Node-action: delete Revision-number: 4 Prop-content-length: 116 Content-length: 116 K 7 svn:log V 14 create branch1 K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:42.184704Z PROPS-END Node-path: branches/branch1 Node-kind: dir Node-action: add Node-copyfrom-rev: 1 Node-copyfrom-path: trunk Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: branches/branch1/a Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/a Text-copy-source-md5: 60b725f10c9c85c70d97880dfe8191b3 Node-path: branches/branch1/b Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/b Text-copy-source-md5: 3b5d5c3712955042212316173ccf37be Node-path: branches/branch1/changed Node-kind: file Node-action: add Node-copyfrom-rev: 3 Node-copyfrom-path: trunk/changed Text-copy-source-md5: 1725f40a29aad369a39b0f96c82d50f9 Node-path: branches/branch1/changeddir Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/changeddir Node-path: branches/branch1/changeddir/f Node-kind: file Node-action: delete Node-path: branches/branch1/changeddir/f Node-kind: file Node-action: add Node-copyfrom-rev: 3 Node-copyfrom-path: trunk/changeddir/f Text-copy-source-md5: 984b8c4ab9193b7659b9f914897a949c Node-path: branches/branch1/da Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/da Node-path: branches/branch1/groupdir Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/groupdir Node-path: branches/branch1/groupdir/a Node-kind: file Node-action: delete Node-path: branches/branch1/groupdir/a Node-kind: file Node-action: add Node-copyfrom-rev: 3 Node-copyfrom-path: trunk/groupdir/a Text-copy-source-md5: 0d227f1abf8c2932d342e9b99cc957eb Node-path: branches/branch1/unchanged Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/unchanged Text-copy-source-md5: 85ae5b04dd0a666efad8633d142a4635 Node-path: branches/branch1/unchangeddir Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/unchangeddir Revision-number: 5 Prop-content-length: 106 Content-length: 106 K 7 svn:log V 5 add c K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:43.175723Z PROPS-END Node-path: branches/branch1/c Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 2cd6ee2c70b0bde53fbe6cac3c8b8bb1 Content-length: 12 PROPS-END c Revision-number: 6 Prop-content-length: 129 Content-length: 129 K 7 svn:log V 27 rename and copy a, b and da K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:50.200094Z PROPS-END Node-path: branches/branch1/c1 Node-kind: file Node-action: add Node-copyfrom-rev: 5 Node-copyfrom-path: branches/branch1/c Text-copy-source-md5: 2cd6ee2c70b0bde53fbe6cac3c8b8bb1 Prop-content-length: 34 Text-content-length: 4 Text-content-md5: 63fad9092ad37713ebe26b3193f89c41 Content-length: 38 K 13 svn:mergeinfo V 0 PROPS-END c c Node-path: trunk/a1 Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/a Text-copy-source-md5: 60b725f10c9c85c70d97880dfe8191b3 Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: trunk/a2 Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/a Text-copy-source-md5: 60b725f10c9c85c70d97880dfe8191b3 Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: trunk/b Node-kind: file Node-action: change Text-content-length: 4 Text-content-md5: 06ac26ed8b614fc0b141e4542aa067c2 Content-length: 4 b b Node-path: trunk/b1 Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/b Text-copy-source-md5: 3b5d5c3712955042212316173ccf37be Prop-content-length: 34 Text-content-length: 4 Text-content-md5: 33cb6785d50937d8d307ebb66d6259a7 Content-length: 38 K 13 svn:mergeinfo V 0 PROPS-END b c Node-path: trunk/da1 Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/da Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: trunk/da2 Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/da Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: trunk/a Node-action: delete Node-path: trunk/da Node-action: delete Revision-number: 7 Prop-content-length: 121 Content-length: 121 K 7 svn:log V 19 copy b from branch1 K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:52.154125Z PROPS-END Node-path: trunk/c Node-kind: file Node-action: add Node-copyfrom-rev: 5 Node-copyfrom-path: branches/branch1/c Text-copy-source-md5: 2cd6ee2c70b0bde53fbe6cac3c8b8bb1 Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Revision-number: 8 Prop-content-length: 126 Content-length: 126 K 7 svn:log V 24 copy stuff from the past K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:55.160439Z PROPS-END Node-path: trunk/deleteddir Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/deleteddir Node-path: trunk/deletedfile Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/deletedfile Text-copy-source-md5: 4d742b2f247bec99b41a60acbebc149a Revision-number: 9 Prop-content-length: 140 Content-length: 140 K 7 svn:log V 38 copy stuff from the past before change K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:48:59.163460Z PROPS-END Node-path: trunk/changed2 Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/changed Text-copy-source-md5: ec1bebaea2c042beb68f7679ddd106a4 Node-path: trunk/changed3 Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/changed Text-copy-source-md5: ec1bebaea2c042beb68f7679ddd106a4 Text-content-length: 17 Text-content-md5: 7d93e8c4d61c2a7b05c20b7d8bf11f83 Content-length: 17 changed changed3 Node-path: trunk/changeddir2 Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/changeddir Revision-number: 10 Prop-content-length: 136 Content-length: 136 K 7 svn:log V 34 copy unchanged stuff from the past K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:49:02.160163Z PROPS-END Node-path: trunk/unchanged2 Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/unchanged Text-copy-source-md5: 85ae5b04dd0a666efad8633d142a4635 Node-path: trunk/unchangeddir2 Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/unchangeddir Revision-number: 11 Prop-content-length: 129 Content-length: 129 K 7 svn:log V 27 copy groupdir from the past K 10 svn:author V 7 pmezard K 8 svn:date V 27 2008-12-05T22:49:04.157303Z PROPS-END Node-path: trunk/groupdir2 Node-kind: dir Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/groupdir