annotate tests/test_fetch_truncated.py @ 931:e1dbd9646d6a

svnwrap: use custom StringIO class in get_file() The wrappers were calling ra.get_file() with a cStringIO object. Empirically, svn 1.7.5 is writing 16kB blocks to the stream object, and cStringIO reallocates its internal buffer and doubles its size whenever it is filled. With large committed files this requires two large memory blocks at the same time. SimpleStringIO implements the mimimum StringIO interface used by ra.get_file() but instead stores all the blocks and "join" them at the end. It means more fragmentation but requires only one large block, without overallocation. Also, 16kB blocks should be friendly to most allocators. In practice, this simple change let me convert a revision containing multiple moderately large files, the largest being around 450MB, with a 32-bits Windows setup, python 2.7, swig svn 1.7.5, in stupid mode, while it was previously aborting with "not enough memory". The same revision still fails in replay mode.
author Patrick Mezard <patrick@mezard.eu>
date Sun, 16 Sep 2012 19:31:49 +0200
parents 20e73b5ab6f7
children d741f536f23a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
643
d2ef7220a079 tests: import test_util as the first module in all relevant tests
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 576
diff changeset
1 import test_util
d2ef7220a079 tests: import test_util as the first module in all relevant tests
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 576
diff changeset
2
331
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
3 import unittest
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
4
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
5 from mercurial import commands
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
6 from mercurial import hg
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
7
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
8 class TestFetchTruncatedHistory(test_util.TestBase):
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
9 def test_truncated_history(self, stupid=False):
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
10 # Test repository does not follow the usual layout
866
20e73b5ab6f7 test_util: merge load_svndump_fixture() into TestBase
Patrick Mezard <patrick@mezard.eu>
parents: 833
diff changeset
11 repo_path = self.load_svndump('truncatedhistory.svndump')
20e73b5ab6f7 test_util: merge load_svndump_fixture() into TestBase
Patrick Mezard <patrick@mezard.eu>
parents: 833
diff changeset
12 svn_url = test_util.fileurl(repo_path + '/project2')
576
d96aa92d9ad9 tests: silence test suite by using quiet UIs everywhere
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 331
diff changeset
13 commands.clone(self.ui(stupid), svn_url, self.wc_path, noupdate=True)
d96aa92d9ad9 tests: silence test suite by using quiet UIs everywhere
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 331
diff changeset
14 repo = hg.repository(self.ui(stupid), self.wc_path)
331
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
15
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
16 # We are converting /project2/trunk coming from:
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
17 #
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
18 # Changed paths:
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
19 # D /project1
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
20 # A /project2/trunk (from /project1:2)
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
21 #
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
22 # Here a full fetch should be performed since we are starting
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
23 # the conversion on an already filled branch.
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
24 tip = repo['tip']
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
25 files = tip.manifest().keys()
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
26 files.sort()
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
27 self.assertEqual(files, ['a', 'b'])
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
28 self.assertEqual(repo['tip']['a'].data(), 'a\n')
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
29
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
30 def test_truncated_history_stupid(self):
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
31 self.test_truncated_history(True)
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
32
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
33 def suite():
833
312b37bc5e20 tests: avoid shadowing Python builtin all()
Yonggang Luo <luoyonggang@gmail.com>
parents: 643
diff changeset
34 all_tests = [unittest.TestLoader().loadTestsFromTestCase(TestFetchTruncatedHistory),
331
75f082b5897e Switch to using url scheme wrappers instead of duplicating each command we wrap.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 257
diff changeset
35 ]
833
312b37bc5e20 tests: avoid shadowing Python builtin all()
Yonggang Luo <luoyonggang@gmail.com>
parents: 643
diff changeset
36 return unittest.TestSuite(all_tests)