hgsubversion: tests/comprehensive/test_verify_and

annotate tests/comprehensive/test_verify_and_startrev.py @ 931:e1dbd9646d6a

svnwrap: use custom StringIO class in get_file() The wrappers were calling ra.get_file() with a cStringIO object. Empirically, svn 1.7.5 is writing 16kB blocks to the stream object, and cStringIO reallocates its internal buffer and doubles its size whenever it is filled. With large committed files this requires two large memory blocks at the same time. SimpleStringIO implements the mimimum StringIO interface used by ra.get_file() but instead stores all the blocks and "join" them at the end. It means more fragmentation but requires only one large block, without overallocation. Also, 16kB blocks should be friendly to most allocators. In practice, this simple change let me convert a revision containing multiple moderately large files, the largest being around 450MB, with a 32-bits Windows setup, python 2.7, swig svn 1.7.5, in stupid mode, while it was previously aborting with "not enough memory". The same revision still fails in replay mode.

author	Patrick Mezard <patrick@mezard.eu>
date	Sun, 16 Sep 2012 19:31:49 +0200
parents	7f90bb48c9de
children	f9014e28721b

rev	line source
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	1 import os
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	2 import pickle
766 124cd25de4ef tests: fix running the comprehensive tests using nose. Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 499 diff changeset	3 import sys
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	4 import unittest
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	5
426 72e63999722f tests: make comprehensive tests work properly with nose and run.py Augie Fackler <durin42@gmail.com> parents: 395 diff changeset	6 # wrapped in a try/except because of weirdness in how
72e63999722f tests: make comprehensive tests work properly with nose and run.py Augie Fackler <durin42@gmail.com> parents: 395 diff changeset	7 # run.py works as compared to nose.
72e63999722f tests: make comprehensive tests work properly with nose and run.py Augie Fackler <durin42@gmail.com> parents: 395 diff changeset	8 try:
72e63999722f tests: make comprehensive tests work properly with nose and run.py Augie Fackler <durin42@gmail.com> parents: 395 diff changeset	9 import test_util
72e63999722f tests: make comprehensive tests work properly with nose and run.py Augie Fackler <durin42@gmail.com> parents: 395 diff changeset	10 except ImportError:
766 124cd25de4ef tests: fix running the comprehensive tests using nose. Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 499 diff changeset	11 sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
124cd25de4ef tests: fix running the comprehensive tests using nose. Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 499 diff changeset	12 import test_util
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	13
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	14 from mercurial import hg
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	15 from mercurial import ui
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	16
897 6bc8046e3d0a move verify to a file of its own Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 894 diff changeset	17 from hgsubversion import verify
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	18
886 d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	19 # these fixtures contain no files at HEAD and would result in empty clones
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	20 _skipshallow = set([
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	21 'binaryfiles.svndump',
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	22 'binaryfiles-broken.svndump',
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	23 'emptyrepo.svndump',
892 3bfb7e985c47 svn verify: add a test for corrupt repositories. Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 888 diff changeset	24 'correct.svndump',
3bfb7e985c47 svn verify: add a test for corrupt repositories. Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 888 diff changeset	25 'corrupt.svndump',
886 d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	26 ])
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	27
888 c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	28 _skipall = set([
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	29 'project_root_not_repo_root.svndump',
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	30 ])
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	31
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	32 _skipstandard = set([
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	33 'subdir_is_file_prefix.svndump',
892 3bfb7e985c47 svn verify: add a test for corrupt repositories. Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 888 diff changeset	34 'correct.svndump',
3bfb7e985c47 svn verify: add a test for corrupt repositories. Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 888 diff changeset	35 'corrupt.svndump',
888 c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	36 ])
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	37
499 1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	38 def _do_case(self, name, stupid, layout):
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	39 subdir = test_util.subdir.get(name, '')
886 d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	40 repo, svnpath = self.load_and_fetch(name, subdir=subdir, stupid=stupid,
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	41 layout=layout)
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	42 assert len(self.repo) > 0
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	43 for i in repo:
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	44 ctx = repo[i]
899 7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	45 self.assertEqual(verify.verify(repo.ui, repo, rev=ctx.node(),
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	46 stupid=True), 0)
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	47 self.assertEqual(verify.verify(repo.ui, repo, rev=ctx.node(),
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	48 stupid=False), 0)
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	49
886 d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	50 # check a startrev clone
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	51 if layout == 'single' and name not in _skipshallow:
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	52 self.wc_path += '_shallow'
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	53 shallowrepo = self.fetch(svnpath, subdir=subdir, stupid=stupid,
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	54 layout='single', startrev='HEAD')
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	55
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	56 self.assertEqual(len(shallowrepo), 1,
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	57 "shallow clone should have just one revision, not %d"
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	58 % len(shallowrepo))
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	59
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	60 fulltip = repo['tip']
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	61 shallowtip = shallowrepo['tip']
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	62
894 9562f606c4aa make assertion failures slightly more helpful by including the failing verification Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 892 diff changeset	63 repo.ui.pushbuffer()
897 6bc8046e3d0a move verify to a file of its own Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 894 diff changeset	64 self.assertEqual(0, verify.verify(repo.ui, shallowrepo,
899 7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	65 rev=shallowtip.node(),
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	66 stupid=True))
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	67 self.assertEqual(0, verify.verify(repo.ui, shallowrepo,
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	68 rev=shallowtip.node(),
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	69 stupid=False))
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	70
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	71 stupidui = ui.ui(repo.ui)
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	72 stupidui.config('hgsubversion', 'stupid', True)
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	73 self.assertEqual(verify.verify(stupidui, repo, rev=ctx.node(),
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	74 stupid=True), 0)
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	75 self.assertEqual(verify.verify(stupidui, repo, rev=ctx.node(),
7f90bb48c9de svn verify: use a custom editor and get_revision() Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 897 diff changeset	76 stupid=False), 0)
886 d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	77
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	78 # viewing diff's of lists of files is easier on the eyes
894 9562f606c4aa make assertion failures slightly more helpful by including the failing verification Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 892 diff changeset	79 self.assertMultiLineEqual('\n'.join(fulltip), '\n'.join(shallowtip),
9562f606c4aa make assertion failures slightly more helpful by including the failing verification Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 892 diff changeset	80 repo.ui.popbuffer())
886 d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	81
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	82 for f in fulltip:
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	83 self.assertMultiLineEqual(fulltip[f].data(), shallowtip[f].data())
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	84
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 833 diff changeset	85
499 1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	86 def buildmethod(case, name, stupid, layout):
1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	87 m = lambda self: self._do_case(case, stupid, layout)
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	88 m.__name__ = name
499 1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	89 bits = case, stupid and 'stupid' or 'real', layout
1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	90 m.__doc__ = 'Test verify on %s with %s replay. (%s)' % bits
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	91 return m
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	92
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	93 attrs = {'_do_case': _do_case}
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	94 fixtures = [f for f in os.listdir(test_util.FIXTURES) if f.endswith('.svndump')]
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	95 for case in fixtures:
888 c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	96 if case in _skipall:
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	97 continue
499 1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	98 bname = 'test_' + case[:-len('.svndump')]
888 c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	99 if case not in _skipstandard:
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	100 attrs[bname] = buildmethod(case, bname, False, 'standard')
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	101 name = bname + '_stupid'
c6388ed0ec0a svnmeta: only remove directory components in normalize() Dan Villiom Podlaski Christiansen <dan@cabo.dk> parents: 886 diff changeset	102 attrs[name] = buildmethod(case, name, True, 'standard')
499 1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	103 name = bname + '_single'
1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	104 attrs[name] = buildmethod(case, name, False, 'single')
1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	105 # Disabled because the "stupid and real are the same" tests
1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	106 # verify this plus even more.
1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	107 # name = bname + '_single_stupid'
1fd3cfa47c5e Support for single-directory clones. Augie Fackler <durin42@gmail.com> parents: 441 diff changeset	108 # attrs[name] = buildmethod(case, name, True, 'single')
395 636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	109
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	110 VerifyTests = type('VerifyTests', (test_util.TestBase,), attrs)
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	111
636e9bf5d49c svncommands: add verify command Dirkjan Ochtman <dirkjan@ochtman.nl> parents: diff changeset	112 def suite():
833 312b37bc5e20 tests: avoid shadowing Python builtin all() Yonggang Luo <luoyonggang@gmail.com> parents: 766 diff changeset	113 all_tests = [unittest.TestLoader().loadTestsFromTestCase(VerifyTests)]
312b37bc5e20 tests: avoid shadowing Python builtin all() Yonggang Luo <luoyonggang@gmail.com> parents: 766 diff changeset	114 return unittest.TestSuite(all_tests)

Mercurial > hgsubversion

annotate tests/comprehensive/test_verify_and_startrev.py @ 931:e1dbd9646d6a