annotate tests/comprehensive/test_verify_and_startrev.py @ 890:78db88de9622

Partial metadata rebuilding For highly active subversion repositories, it can be excruciatingly slow to pull updates one at a time from subversion. One way around this is to setup another mercurial repo that pulls new commits from svn periodicly (say every 5 minutes). When you want to update your repository, you can pull commits from this mercurial repository via native mercurial protocols, which will be much faster than pulling directly from svn. Unfortunately, your metadata will be out of date after doing so. Highly active repositories also tend to be very large, which means that it takes a long time to rebuild your metadata from scratch. To address this, this adds support to do a partial rebuild on the metadata by processing only revisions that have been added to the repository after the last revision we processed. With the rev map 1k revisions (~2 days) behind tip updatemeta is dramatically faster than rebuild meta: $ hg --time svn updatemeta Time: real 0.570 secs (user 0.480+0.000 sys 0.060+0.000) $ hg --time svn rebuildmeta Time: real 129.160 secs (user 128.570+0.000 sys 0.320+0.000)
author David Schleimer <dschleimer@fb.com>
date Sat, 12 May 2012 07:28:23 -0700
parents c6388ed0ec0a
children 3bfb7e985c47
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
1 import os
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
2 import pickle
766
124cd25de4ef tests: fix running the comprehensive tests using nose.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 499
diff changeset
3 import sys
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
4 import unittest
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
5
426
72e63999722f tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents: 395
diff changeset
6 # wrapped in a try/except because of weirdness in how
72e63999722f tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents: 395
diff changeset
7 # run.py works as compared to nose.
72e63999722f tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents: 395
diff changeset
8 try:
72e63999722f tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents: 395
diff changeset
9 import test_util
72e63999722f tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents: 395
diff changeset
10 except ImportError:
766
124cd25de4ef tests: fix running the comprehensive tests using nose.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 499
diff changeset
11 sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
124cd25de4ef tests: fix running the comprehensive tests using nose.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 499
diff changeset
12 import test_util
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
13
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
14 from mercurial import hg
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
15 from mercurial import ui
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
16
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
17 from hgsubversion import svncommands
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
18
886
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
19 # these fixtures contain no files at HEAD and would result in empty clones
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
20 _skipshallow = set([
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
21 'binaryfiles.svndump',
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
22 'binaryfiles-broken.svndump',
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
23 'emptyrepo.svndump',
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
24 ])
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
25
888
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
26 _skipall = set([
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
27 'project_root_not_repo_root.svndump',
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
28 ])
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
29
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
30 _skipstandard = set([
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
31 'subdir_is_file_prefix.svndump',
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
32 ])
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
33
499
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
34 def _do_case(self, name, stupid, layout):
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
35 subdir = test_util.subdir.get(name, '')
886
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
36 repo, svnpath = self.load_and_fetch(name, subdir=subdir, stupid=stupid,
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
37 layout=layout)
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
38 assert len(self.repo) > 0
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
39 for i in repo:
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
40 ctx = repo[i]
441
de085126dbd4 svncommands: use rev instead of verifynode -- should not have to be a node
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents: 427
diff changeset
41 self.assertEqual(svncommands.verify(repo.ui, repo, rev=ctx.node()), 0)
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
42
886
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
43 # check a startrev clone
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
44 if layout == 'single' and name not in _skipshallow:
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
45 self.wc_path += '_shallow'
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
46 shallowrepo = self.fetch(svnpath, subdir=subdir, stupid=stupid,
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
47 layout='single', startrev='HEAD')
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
48
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
49 self.assertEqual(len(shallowrepo), 1,
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
50 "shallow clone should have just one revision, not %d"
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
51 % len(shallowrepo))
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
52
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
53 fulltip = repo['tip']
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
54 shallowtip = shallowrepo['tip']
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
55
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
56 self.assertEqual(0, svncommands.verify(repo.ui, shallowrepo,
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
57 rev=shallowtip.node()))
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
58
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
59 # viewing diff's of lists of files is easier on the eyes
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
60 self.assertMultiLineEqual('\n'.join(fulltip), '\n'.join(shallowtip))
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
61
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
62 for f in fulltip:
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
63 self.assertMultiLineEqual(fulltip[f].data(), shallowtip[f].data())
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
64
d3ff5807f1bd fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 833
diff changeset
65
499
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
66 def buildmethod(case, name, stupid, layout):
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
67 m = lambda self: self._do_case(case, stupid, layout)
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
68 m.__name__ = name
499
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
69 bits = case, stupid and 'stupid' or 'real', layout
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
70 m.__doc__ = 'Test verify on %s with %s replay. (%s)' % bits
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
71 return m
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
72
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
73 attrs = {'_do_case': _do_case}
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
74 fixtures = [f for f in os.listdir(test_util.FIXTURES) if f.endswith('.svndump')]
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
75 for case in fixtures:
888
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
76 if case in _skipall:
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
77 continue
499
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
78 bname = 'test_' + case[:-len('.svndump')]
888
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
79 if case not in _skipstandard:
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
80 attrs[bname] = buildmethod(case, bname, False, 'standard')
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
81 name = bname + '_stupid'
c6388ed0ec0a svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 886
diff changeset
82 attrs[name] = buildmethod(case, name, True, 'standard')
499
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
83 name = bname + '_single'
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
84 attrs[name] = buildmethod(case, name, False, 'single')
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
85 # Disabled because the "stupid and real are the same" tests
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
86 # verify this plus even more.
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
87 # name = bname + '_single_stupid'
1fd3cfa47c5e Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents: 441
diff changeset
88 # attrs[name] = buildmethod(case, name, True, 'single')
395
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
89
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
90 VerifyTests = type('VerifyTests', (test_util.TestBase,), attrs)
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
91
636e9bf5d49c svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff changeset
92 def suite():
833
312b37bc5e20 tests: avoid shadowing Python builtin all()
Yonggang Luo <luoyonggang@gmail.com>
parents: 766
diff changeset
93 all_tests = [unittest.TestLoader().loadTestsFromTestCase(VerifyTests)]
312b37bc5e20 tests: avoid shadowing Python builtin all()
Yonggang Luo <luoyonggang@gmail.com>
parents: 766
diff changeset
94 return unittest.TestSuite(all_tests)