Mercurial > hgsubversion
annotate tests/comprehensive/test_verify_and_startrev.py @ 890:78db88de9622
Partial metadata rebuilding
For highly active subversion repositories, it can be excruciatingly
slow to pull updates one at a time from subversion. One way around
this is to setup another mercurial repo that pulls new commits from
svn periodicly (say every 5 minutes). When you want to update your
repository, you can pull commits from this mercurial repository via
native mercurial protocols, which will be much faster than pulling
directly from svn.
Unfortunately, your metadata will be out of date after doing so.
Highly active repositories also tend to be very large, which means
that it takes a long time to rebuild your metadata from scratch. To
address this, this adds support to do a partial rebuild on the
metadata by processing only revisions that have been added to the
repository after the last revision we processed.
With the rev map 1k revisions (~2 days) behind tip updatemeta is
dramatically faster than rebuild meta:
$ hg --time svn updatemeta
Time: real 0.570 secs (user 0.480+0.000 sys 0.060+0.000)
$ hg --time svn rebuildmeta
Time: real 129.160 secs (user 128.570+0.000 sys 0.320+0.000)
author | David Schleimer <dschleimer@fb.com> |
---|---|
date | Sat, 12 May 2012 07:28:23 -0700 |
parents | c6388ed0ec0a |
children | 3bfb7e985c47 |
rev | line source |
---|---|
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
1 import os |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
2 import pickle |
766
124cd25de4ef
tests: fix running the comprehensive tests using nose.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
499
diff
changeset
|
3 import sys |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
4 import unittest |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
5 |
426
72e63999722f
tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents:
395
diff
changeset
|
6 # wrapped in a try/except because of weirdness in how |
72e63999722f
tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents:
395
diff
changeset
|
7 # run.py works as compared to nose. |
72e63999722f
tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents:
395
diff
changeset
|
8 try: |
72e63999722f
tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents:
395
diff
changeset
|
9 import test_util |
72e63999722f
tests: make comprehensive tests work properly with nose and run.py
Augie Fackler <durin42@gmail.com>
parents:
395
diff
changeset
|
10 except ImportError: |
766
124cd25de4ef
tests: fix running the comprehensive tests using nose.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
499
diff
changeset
|
11 sys.path.insert(0, os.path.dirname(os.path.dirname(__file__))) |
124cd25de4ef
tests: fix running the comprehensive tests using nose.
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
499
diff
changeset
|
12 import test_util |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
13 |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
14 from mercurial import hg |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
15 from mercurial import ui |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
16 |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
17 from hgsubversion import svncommands |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
18 |
886
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
19 # these fixtures contain no files at HEAD and would result in empty clones |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
20 _skipshallow = set([ |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
21 'binaryfiles.svndump', |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
22 'binaryfiles-broken.svndump', |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
23 'emptyrepo.svndump', |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
24 ]) |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
25 |
888
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
26 _skipall = set([ |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
27 'project_root_not_repo_root.svndump', |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
28 ]) |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
29 |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
30 _skipstandard = set([ |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
31 'subdir_is_file_prefix.svndump', |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
32 ]) |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
33 |
499
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
34 def _do_case(self, name, stupid, layout): |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
35 subdir = test_util.subdir.get(name, '') |
886
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
36 repo, svnpath = self.load_and_fetch(name, subdir=subdir, stupid=stupid, |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
37 layout=layout) |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
38 assert len(self.repo) > 0 |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
39 for i in repo: |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
40 ctx = repo[i] |
441
de085126dbd4
svncommands: use rev instead of verifynode -- should not have to be a node
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
427
diff
changeset
|
41 self.assertEqual(svncommands.verify(repo.ui, repo, rev=ctx.node()), 0) |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
42 |
886
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
43 # check a startrev clone |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
44 if layout == 'single' and name not in _skipshallow: |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
45 self.wc_path += '_shallow' |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
46 shallowrepo = self.fetch(svnpath, subdir=subdir, stupid=stupid, |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
47 layout='single', startrev='HEAD') |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
48 |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
49 self.assertEqual(len(shallowrepo), 1, |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
50 "shallow clone should have just one revision, not %d" |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
51 % len(shallowrepo)) |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
52 |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
53 fulltip = repo['tip'] |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
54 shallowtip = shallowrepo['tip'] |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
55 |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
56 self.assertEqual(0, svncommands.verify(repo.ui, shallowrepo, |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
57 rev=shallowtip.node())) |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
58 |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
59 # viewing diff's of lists of files is easier on the eyes |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
60 self.assertMultiLineEqual('\n'.join(fulltip), '\n'.join(shallowtip)) |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
61 |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
62 for f in fulltip: |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
63 self.assertMultiLineEqual(fulltip[f].data(), shallowtip[f].data()) |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
64 |
d3ff5807f1bd
fold test_startrev and test_verify into a new test; test_verify_and_startrev
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents:
833
diff
changeset
|
65 |
499
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
66 def buildmethod(case, name, stupid, layout): |
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
67 m = lambda self: self._do_case(case, stupid, layout) |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
68 m.__name__ = name |
499
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
69 bits = case, stupid and 'stupid' or 'real', layout |
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
70 m.__doc__ = 'Test verify on %s with %s replay. (%s)' % bits |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
71 return m |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
72 |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
73 attrs = {'_do_case': _do_case} |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
74 fixtures = [f for f in os.listdir(test_util.FIXTURES) if f.endswith('.svndump')] |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
75 for case in fixtures: |
888
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
76 if case in _skipall: |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
77 continue |
499
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
78 bname = 'test_' + case[:-len('.svndump')] |
888
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
79 if case not in _skipstandard: |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
80 attrs[bname] = buildmethod(case, bname, False, 'standard') |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
81 name = bname + '_stupid' |
c6388ed0ec0a
svnmeta: only remove directory components in normalize()
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents:
886
diff
changeset
|
82 attrs[name] = buildmethod(case, name, True, 'standard') |
499
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
83 name = bname + '_single' |
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
84 attrs[name] = buildmethod(case, name, False, 'single') |
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
85 # Disabled because the "stupid and real are the same" tests |
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
86 # verify this plus even more. |
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
87 # name = bname + '_single_stupid' |
1fd3cfa47c5e
Support for single-directory clones.
Augie Fackler <durin42@gmail.com>
parents:
441
diff
changeset
|
88 # attrs[name] = buildmethod(case, name, True, 'single') |
395
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
89 |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
90 VerifyTests = type('VerifyTests', (test_util.TestBase,), attrs) |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
91 |
636e9bf5d49c
svncommands: add verify command
Dirkjan Ochtman <dirkjan@ochtman.nl>
parents:
diff
changeset
|
92 def suite(): |
833
312b37bc5e20
tests: avoid shadowing Python builtin all()
Yonggang Luo <luoyonggang@gmail.com>
parents:
766
diff
changeset
|
93 all_tests = [unittest.TestLoader().loadTestsFromTestCase(VerifyTests)] |
312b37bc5e20
tests: avoid shadowing Python builtin all()
Yonggang Luo <luoyonggang@gmail.com>
parents:
766
diff
changeset
|
94 return unittest.TestSuite(all_tests) |