annotate tests/comprehensive/test_updatemeta.py @ 1233:0d0132cba155

editor: fix edge case with in memory file-store size limit There are a few cases where we will set a single file into to the editor's FileStore object more than once. Notably, for copied and then modified files, we will set it at least twice. Three times if editing fails (which it can for symlinks). If we pass the in-memory storage limit in between the first (or second if editing fails) time we set the file and the last time we set the file, we will write the data to the in memory store the first time and the file store the last time. We didn't remove it form the in-memory store though, and we always prefer reading from the in-memory store. This means we can sometimes end up with the wrong version of a file. This is fairly unlikely to happen in normal use since you need to hit the memory limit between two writes to the store for the same file. We only write a file multiple times if a) the file (and not one of it's parent directories) is copied and then modified or b) editing fails. From what I can tell, it's only common for editing to fail for symlinks, and they ten to be relatively small data that is unlikely to push over the limit. Finally, the default limit is 100MB which I would expect to be most often either well over (source code) or well under (binaries or automated changes) the size of the changes files in a single commit. The easiest way to reproduce this is to set the in-memory cache size to 0 and then commit a copied and modified symlink. The empty-string version from the failed editing will be the one that persists. I happened to stumble upon this while trying (and failing) to test a bug-fix for a related bug with identical symptoms (empty simlink). I have seen this in the wild, once, but couldn't reproduce it at the time. The repo in question is quite large and quite active, so I am quite confident in my estimation that this is a real, but very rare, problem. The test changes attached to this was mneant to test a related bug, but turned out not to actually cover the bug in question. They did trigger this bug though, and are worthwhile to test, so I kept them.
author David Schleimer <dschleimer@fb.com>
date Mon, 07 Apr 2014 17:51:59 -0700
parents cd0d14e25757
children c6b01fd34694
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
1 import os
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
2 import pickle
1085
48379ebd2763 tests: unbreak running test_updatemeta individually
David Schleimer <dschleimer@fb.com>
parents: 1057
diff changeset
3 import sys
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
4 import unittest
1042
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
5
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
6 # wrapped in a try/except because of weirdness in how
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
7 # run.py works as compared to nose.
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
8 try:
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
9 import test_util
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
10 except ImportError:
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
11 sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
12 import test_util
af84ef787d93 tests: move updatemeta & rebuildmeta tests into comprehensive
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 930
diff changeset
13
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
14 import test_rebuildmeta
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
15
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
16 from mercurial import context
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
17 from mercurial import extensions
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
18 from mercurial import hg
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
19 from mercurial import ui
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
20
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
21 from hgsubversion import svncommands
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
22 from hgsubversion import svnmeta
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
23
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
24
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
25
1092
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
26 def _do_case(self, name, layout):
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
27 subdir = test_util.subdir.get(name, '')
1092
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
28 single = layout == 'single'
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
29 u = ui.ui()
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
30 config = {}
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
31 if layout == 'custom':
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
32 config['hgsubversion.layout'] = 'custom'
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
33 u.setconfig('hgsubversion', 'layout', 'custom')
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
34 for branch, path in test_util.custom.get(name, {}).iteritems():
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
35 config['hgsubversionbranch.%s' % branch] = path
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
36 u.setconfig('hgsubversionbranch', branch, path)
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
37
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
38 repo, repo_path = self.load_and_fetch(name,
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
39 subdir=subdir,
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
40 layout=layout,
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
41 config=config)
1048
903c9c9dfe6a tests: count revisions explicitly
Dan Villiom Podlaski Christiansen <danchr@gmail.com>
parents: 1044
diff changeset
42 assert test_util.repolen(self.repo) > 0
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
43 wc2_path = self.wc_path + '_clone'
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
44 src, dest = test_util.hgclone(u, self.wc_path, wc2_path, update=False)
930
5bacb9c63e3e Fix more peer breakage with old hg versions
Patrick Mezard <patrick@mezard.eu>
parents: 922
diff changeset
45 src = test_util.getlocalpeer(src)
5bacb9c63e3e Fix more peer breakage with old hg versions
Patrick Mezard <patrick@mezard.eu>
parents: 922
diff changeset
46 dest = test_util.getlocalpeer(dest)
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
47
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
48 # insert a wrapper that prevents calling changectx.children()
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
49 def failfn(orig, ctx):
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
50 self.fail('calling %s is forbidden; it can cause massive slowdowns '
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
51 'when rebuilding large repositories' % orig)
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
52
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
53 origchildren = getattr(context.changectx, 'children')
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
54 extensions.wrapfunction(context.changectx, 'children', failfn)
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
55
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
56 # test updatemeta on an empty repo
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
57 try:
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
58 svncommands.updatemeta(u, dest,
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
59 args=[test_util.fileurl(repo_path +
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
60 subdir), ])
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
61 finally:
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
62 # remove the wrapper
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
63 context.changectx.children = origchildren
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
64
1057
cd256960b622 comprehensive tests: consolidate stupidity into test_util
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 1048
diff changeset
65 self._run_assertions(name, single, src, dest, u)
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
66
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
67
1057
cd256960b622 comprehensive tests: consolidate stupidity into test_util
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 1048
diff changeset
68 def _run_assertions(self, name, single, src, dest, u):
cd256960b622 comprehensive tests: consolidate stupidity into test_util
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 1048
diff changeset
69 test_rebuildmeta._run_assertions(self, name, single, src, dest, u)
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
70
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
71
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
72 skip = set([
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
73 'project_root_not_repo_root.svndump',
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
74 'corrupt.svndump',
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
75 ])
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
76
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
77 attrs = {'_do_case': _do_case,
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
78 '_run_assertions': _run_assertions,
1057
cd256960b622 comprehensive tests: consolidate stupidity into test_util
Dan Villiom Podlaski Christiansen <dan@cabo.dk>
parents: 1048
diff changeset
79 'stupid_mode_tests': True,
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
80 }
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
81 for case in [f for f in os.listdir(test_util.FIXTURES) if f.endswith('.svndump')]:
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
82 # this fixture results in an empty repository, don't use it
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
83 if case in skip:
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
84 continue
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
85 bname = 'test_' + case[:-len('.svndump')]
1092
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
86 attrs[bname] = test_rebuildmeta.buildmethod(case, bname, 'auto')
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
87 attrs[bname + '_single'] = test_rebuildmeta.buildmethod(case,
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
88 bname + '_single',
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
89 'single')
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
90 if case in test_util.custom:
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
91 attrs[bname + '_custom'] = test_rebuildmeta.buildmethod(case,
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
92 bname + '_custom',
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
93 'custom')
cd0d14e25757 layouts: add custom layout for those of us that need weird mappings
David Schleimer <dschleimer@fb.com>
parents: 1085
diff changeset
94
922
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
95
6b7ac659c855 updatemeta: correctly handle empty metadata
Jun Fang <junfang@fb.com>
parents:
diff changeset
96 UpdateMetaTests = type('UpdateMetaTests', (test_util.TestBase,), attrs)