Mercurial > hgsubversion
view tests/fixtures/externals.svndump @ 931:e1dbd9646d6a
svnwrap: use custom StringIO class in get_file()
The wrappers were calling ra.get_file() with a cStringIO object.
Empirically, svn 1.7.5 is writing 16kB blocks to the stream object, and
cStringIO reallocates its internal buffer and doubles its size whenever
it is filled. With large committed files this requires two large
memory blocks at the same time.
SimpleStringIO implements the mimimum StringIO interface used by
ra.get_file() but instead stores all the blocks and "join" them at the
end. It means more fragmentation but requires only one large block,
without overallocation. Also, 16kB blocks should be friendly to most
allocators.
In practice, this simple change let me convert a revision containing
multiple moderately large files, the largest being around 450MB, with a
32-bits Windows setup, python 2.7, swig svn 1.7.5, in stupid mode, while
it was previously aborting with "not enough memory". The same revision
still fails in replay mode.
| author | Patrick Mezard <patrick@mezard.eu> |
|---|---|
| date | Sun, 16 Sep 2012 19:31:49 +0200 |
| parents | a5be5ca638de |
| children |
line wrap: on
line source
SVN-fs-dump-format-version: 2 UUID: 2fcff1c7-6cef-40bf-9072-468ceec83032 Revision-number: 0 Prop-content-length: 56 Content-length: 56 K 8 svn:date V 27 2011-02-25T14:33:01.299441Z PROPS-END Revision-number: 1 Prop-content-length: 114 Content-length: 114 K 7 svn:log V 12 init project K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:01.321997Z PROPS-END Node-path: branches Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: externals Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Revision-number: 2 Prop-content-length: 130 Content-length: 130 K 7 svn:log V 28 configure externals projects K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:02.082464Z PROPS-END Node-path: externals/project1 Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: externals/project1/a Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 60b725f10c9c85c70d97880dfe8191b3 Text-content-sha1: 3f786850e387550fdab836ed7e6dc881de23001b Content-length: 12 PROPS-END a Node-path: externals/project2 Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: externals/project2/b Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 60b725f10c9c85c70d97880dfe8191b3 Text-content-sha1: 3f786850e387550fdab836ed7e6dc881de23001b Content-length: 12 PROPS-END a Revision-number: 3 Prop-content-length: 120 Content-length: 120 K 7 svn:log V 18 set externals on . K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:03.076026Z PROPS-END Node-path: trunk Node-kind: dir Node-action: change Prop-content-length: 70 Content-length: 70 K 13 svn:externals V 35 ^/externals/project1 deps/project1 PROPS-END Node-path: trunk/a Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 2 Text-content-md5: 60b725f10c9c85c70d97880dfe8191b3 Text-content-sha1: 3f786850e387550fdab836ed7e6dc881de23001b Content-length: 12 PROPS-END a Revision-number: 4 Prop-content-length: 123 Content-length: 123 K 7 svn:log V 21 update externals on . K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:04.058124Z PROPS-END Node-path: trunk Node-kind: dir Node-action: change Prop-content-length: 169 Content-length: 169 K 13 svn:externals V 133 # A comment, then an empty line, then a blank line ^/externals/project1 deps/project1 -r2 ^/externals/project2@2 deps/project2 PROPS-END Revision-number: 5 Prop-content-length: 115 Content-length: 115 K 7 svn:log V 13 add on subdir K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:05.096545Z PROPS-END Node-path: trunk Node-kind: dir Node-action: change Prop-content-length: 76 Content-length: 76 K 13 svn:externals V 41 -r2 ^/externals/project2@2 deps/project2 PROPS-END Node-path: trunk/subdir Node-kind: dir Node-action: add Prop-content-length: 70 Content-length: 70 K 13 svn:externals V 35 ^/externals/project1 deps/project1 PROPS-END Node-path: trunk/subdir2 Node-kind: dir Node-action: add Prop-content-length: 70 Content-length: 70 K 13 svn:externals V 35 ^/externals/project1 deps/project1 PROPS-END Revision-number: 6 Prop-content-length: 122 Content-length: 122 K 7 svn:log V 20 externals in subtree K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:08.059874Z PROPS-END Node-path: branches/branch1 Node-kind: dir Node-action: add Node-copyfrom-rev: 5 Node-copyfrom-path: trunk Node-path: branches/branch1/subdir2 Node-kind: dir Node-action: change Prop-content-length: 10 Content-length: 10 PROPS-END Revision-number: 7 Prop-content-length: 139 Content-length: 139 K 7 svn:log V 37 externals in subtree, removed on root K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:10.059295Z PROPS-END Node-path: branches/branch2 Node-kind: dir Node-action: add Node-copyfrom-rev: 5 Node-copyfrom-path: trunk Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: branches/branch2/subdir2 Node-kind: dir Node-action: change Prop-content-length: 10 Content-length: 10 PROPS-END Revision-number: 8 Prop-content-length: 125 Content-length: 125 K 7 svn:log V 23 remove externals subdir K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:11.058317Z PROPS-END Node-path: trunk/subdir Node-action: delete Revision-number: 9 Prop-content-length: 126 Content-length: 126 K 7 svn:log V 24 remove externals subdir2 K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:12.058270Z PROPS-END Node-path: trunk/subdir2 Node-kind: dir Node-action: change Prop-content-length: 10 Content-length: 10 PROPS-END Revision-number: 10 Prop-content-length: 127 Content-length: 127 K 7 svn:log V 25 remove externals project2 K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:14.058780Z PROPS-END Node-path: externals/project2 Node-action: delete Revision-number: 11 Prop-content-length: 109 Content-length: 109 K 7 svn:log V 8 change a K 10 svn:author V 7 pmezard K 8 svn:date V 27 2011-02-25T14:33:15.045928Z PROPS-END Node-path: trunk/a Node-kind: file Node-action: change Text-content-length: 4 Text-content-md5: 0d227f1abf8c2932d342e9b99cc957eb Text-content-sha1: d7c8127a20a396cff08af086a1c695b0636f0c29 Content-length: 4 a a