Mercurial > hgsubversion
view tests/fixtures/project_root_at_repo_root.svndump @ 931:e1dbd9646d6a
svnwrap: use custom StringIO class in get_file()
The wrappers were calling ra.get_file() with a cStringIO object.
Empirically, svn 1.7.5 is writing 16kB blocks to the stream object, and
cStringIO reallocates its internal buffer and doubles its size whenever
it is filled. With large committed files this requires two large
memory blocks at the same time.
SimpleStringIO implements the mimimum StringIO interface used by
ra.get_file() but instead stores all the blocks and "join" them at the
end. It means more fragmentation but requires only one large block,
without overallocation. Also, 16kB blocks should be friendly to most
allocators.
In practice, this simple change let me convert a revision containing
multiple moderately large files, the largest being around 450MB, with a
32-bits Windows setup, python 2.7, swig svn 1.7.5, in stupid mode, while
it was previously aborting with "not enough memory". The same revision
still fails in replay mode.
| author | Patrick Mezard <patrick@mezard.eu> |
|---|---|
| date | Sun, 16 Sep 2012 19:31:49 +0200 |
| parents | 537de0300510 |
| children |
line wrap: on
line source
SVN-fs-dump-format-version: 2 UUID: e72213dc-9746-48d2-846a-a19c2c569b0d Revision-number: 0 Prop-content-length: 56 Content-length: 56 K 8 svn:date V 27 2008-10-09T14:46:06.470091Z PROPS-END Revision-number: 1 Prop-content-length: 111 Content-length: 111 K 7 svn:log V 11 Empty dirs. K 10 svn:author V 5 durin K 8 svn:date V 27 2008-10-09T14:46:07.147162Z PROPS-END Node-path: branches Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: tags Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Node-path: trunk Node-kind: dir Node-action: add Prop-content-length: 10 Content-length: 10 PROPS-END Revision-number: 2 Prop-content-length: 114 Content-length: 114 K 7 svn:log V 14 Initial Files. K 10 svn:author V 5 durin K 8 svn:date V 27 2008-10-09T14:46:08.147874Z PROPS-END Node-path: trunk/alpha Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 15 Text-content-md5: 8717538ba2f18a613eaa4892e8d178f7 Content-length: 25 PROPS-END This is alpha. Node-path: trunk/beta Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 14 Text-content-md5: 952a13b33256f343982d0a8f0e0af277 Content-length: 24 PROPS-END This is beta. Node-path: trunk/delta Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 15 Text-content-md5: e48d0bca73f04e800e7bd9fb58d49a62 Content-length: 25 PROPS-END This is delta. Revision-number: 3 Prop-content-length: 110 Content-length: 110 K 7 svn:log V 10 Tag rev 1. K 10 svn:author V 5 durin K 8 svn:date V 27 2008-10-09T14:46:10.131807Z PROPS-END Node-path: tags/rev1 Node-kind: dir Node-action: add Node-copyfrom-rev: 1 Node-copyfrom-path: trunk Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: tags/rev1/alpha Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/alpha Text-copy-source-md5: 8717538ba2f18a613eaa4892e8d178f7 Node-path: tags/rev1/beta Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/beta Text-copy-source-md5: 952a13b33256f343982d0a8f0e0af277 Node-path: tags/rev1/delta Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/delta Text-copy-source-md5: e48d0bca73f04e800e7bd9fb58d49a62 Revision-number: 4 Prop-content-length: 116 Content-length: 116 K 7 svn:log V 16 Branch to crazy. K 10 svn:author V 5 durin K 8 svn:date V 27 2008-10-09T14:46:12.131174Z PROPS-END Node-path: branches/crazy Node-kind: dir Node-action: add Node-copyfrom-rev: 1 Node-copyfrom-path: trunk Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: branches/crazy/alpha Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/alpha Text-copy-source-md5: 8717538ba2f18a613eaa4892e8d178f7 Node-path: branches/crazy/beta Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/beta Text-copy-source-md5: 952a13b33256f343982d0a8f0e0af277 Node-path: branches/crazy/delta Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/delta Text-copy-source-md5: e48d0bca73f04e800e7bd9fb58d49a62 Revision-number: 5 Prop-content-length: 108 Content-length: 108 K 7 svn:log V 9 Add gamma K 10 svn:author V 5 durin K 8 svn:date V 27 2008-10-09T14:46:13.143196Z PROPS-END Node-path: trunk/gamma Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 15 Text-content-md5: 255b705986e84fbb13db9c5a832739ae Content-length: 25 PROPS-END This is gamma. Revision-number: 6 Prop-content-length: 108 Content-length: 108 K 7 svn:log V 9 Add omega K 10 svn:author V 5 durin K 8 svn:date V 27 2008-10-09T14:46:14.144244Z PROPS-END Node-path: branches/crazy/omega Node-kind: file Node-action: add Prop-content-length: 10 Text-content-length: 15 Text-content-md5: 22219e00c8dbb47ce790f0eb658e8014 Content-length: 25 PROPS-END This is omega. Revision-number: 7 Prop-content-length: 121 Content-length: 121 K 7 svn:log V 21 Branch to more_crazy. K 10 svn:author V 5 durin K 8 svn:date V 27 2008-10-09T14:46:16.135900Z PROPS-END Node-path: branches/more_crazy Node-kind: dir Node-action: add Node-copyfrom-rev: 1 Node-copyfrom-path: trunk Prop-content-length: 34 Content-length: 34 K 13 svn:mergeinfo V 0 PROPS-END Node-path: branches/more_crazy/alpha Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/alpha Text-copy-source-md5: 8717538ba2f18a613eaa4892e8d178f7 Node-path: branches/more_crazy/beta Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/beta Text-copy-source-md5: 952a13b33256f343982d0a8f0e0af277 Node-path: branches/more_crazy/delta Node-kind: file Node-action: add Node-copyfrom-rev: 2 Node-copyfrom-path: trunk/delta Text-copy-source-md5: e48d0bca73f04e800e7bd9fb58d49a62 Node-path: branches/more_crazy/gamma Node-kind: file Node-action: add Node-copyfrom-rev: 5 Node-copyfrom-path: trunk/gamma Text-copy-source-md5: 255b705986e84fbb13db9c5a832739ae