hgsubversion: tests/test_fetch

annotate tests/test_fetch_renames.py @ 1233:0d0132cba155

editor: fix edge case with in memory file-store size limit There are a few cases where we will set a single file into to the editor's FileStore object more than once. Notably, for copied and then modified files, we will set it at least twice. Three times if editing fails (which it can for symlinks). If we pass the in-memory storage limit in between the first (or second if editing fails) time we set the file and the last time we set the file, we will write the data to the in memory store the first time and the file store the last time. We didn't remove it form the in-memory store though, and we always prefer reading from the in-memory store. This means we can sometimes end up with the wrong version of a file. This is fairly unlikely to happen in normal use since you need to hit the memory limit between two writes to the store for the same file. We only write a file multiple times if a) the file (and not one of it's parent directories) is copied and then modified or b) editing fails. From what I can tell, it's only common for editing to fail for symlinks, and they ten to be relatively small data that is unlikely to push over the limit. Finally, the default limit is 100MB which I would expect to be most often either well over (source code) or well under (binaries or automated changes) the size of the changes files in a single commit. The easiest way to reproduce this is to set the in-memory cache size to 0 and then commit a copied and modified symlink. The empty-string version from the failed editing will be the one that persists. I happened to stumble upon this while trying (and failing) to test a bug-fix for a related bug with identical symptoms (empty simlink). I have seen this in the wild, once, but couldn't reproduce it at the time. The repo in question is quite large and quite active, so I am quite confident in my estimation that this is a real, but very rare, problem. The test changes attached to this was mneant to test a related bug, but turned out not to actually cover the bug in question. They did trigger this bug though, and are worthwhile to test, so I kept them.

author	David Schleimer <dschleimer@fb.com>
date	Mon, 07 Apr 2014 17:51:59 -0700
parents	dc516f85b6c3
children	d3c79072bc6a

rev	line source
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	1 import test_util
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	2
643 d2ef7220a079 tests: import test_util as the first module in all relevant tests Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 513 diff changeset	3 import sys
d2ef7220a079 tests: import test_util as the first module in all relevant tests Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 513 diff changeset	4 import unittest
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	5
91 7d10165cf3d9 tests: Mock the mercurial.ui.ui class like we really should to capture output. Augie Fackler <durin42@gmail.com> parents: 78 diff changeset	6 class TestFetchRenames(test_util.TestBase):
1064 dc516f85b6c3 test_fetch_renames: use stupid mode metaclass Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 1044 diff changeset	7 stupid_mode_tests = True
dc516f85b6c3 test_fetch_renames: use stupid mode metaclass Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 1044 diff changeset	8
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	9 def _debug_print_copies(self, repo):
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	10 w = sys.stderr.write
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	11 for rev in repo:
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	12 ctx = repo[rev]
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	13 w('%d - %s\n' % (ctx.rev(), ctx.branch()))
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	14 for f in ctx:
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	15 fctx = ctx[f]
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	16 w('%s: %r %r\n' % (f, fctx.data(), fctx.renamed()))
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	17
1064 dc516f85b6c3 test_fetch_renames: use stupid mode metaclass Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 1044 diff changeset	18 def test_rename(self):
944 d6db289f1548 pull: add hgsubversion.filestoresize to control memory consumption Patrick Mezard <patrick@mezard.eu> parents: 833 diff changeset	19 config = {
d6db289f1548 pull: add hgsubversion.filestoresize to control memory consumption Patrick Mezard <patrick@mezard.eu> parents: 833 diff changeset	20 'hgsubversion.filestoresize': '0',
d6db289f1548 pull: add hgsubversion.filestoresize to control memory consumption Patrick Mezard <patrick@mezard.eu> parents: 833 diff changeset	21 }
1064 dc516f85b6c3 test_fetch_renames: use stupid mode metaclass Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 1044 diff changeset	22 repo = self._load_fixture_and_fetch('renames.svndump', config=config)
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	23
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	24 # Map revnum to mappings of dest name to (source name, dest content)
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	25 copies = {
68 e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past Patrick Mezard <pmezard@gmail.com> parents: 67 diff changeset	26 4: {
91 7d10165cf3d9 tests: Mock the mercurial.ui.ui class like we really should to capture output. Augie Fackler <durin42@gmail.com> parents: 78 diff changeset	27 'a1': ('a', 'a\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	28 'linka1': ('linka', 'a'),
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	29 'a2': ('a', 'a\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	30 'linka2': ('linka', 'a'),
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	31 'b1': ('b', 'b\nc\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	32 'linkb1': ('linkb', 'bc'),
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	33 'da1/daf': ('da/daf', 'c\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	34 'da1/dalink': ('da/dalink', 'daf'),
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	35 'da1/db/dbf': ('da/db/dbf', 'd\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	36 'da1/db/dblink': ('da/db/dblink', '../daf'),
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	37 'da2/daf': ('da/daf', 'c\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	38 'da2/dalink': ('da/dalink', 'daf'),
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	39 'da2/db/dbf': ('da/db/dbf', 'd\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	40 'da2/db/dblink': ('da/db/dblink', '../daf'),
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	41 },
68 e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past Patrick Mezard <pmezard@gmail.com> parents: 67 diff changeset	42 5: {
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	43 'c1': ('c', 'c\nc\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	44 'linkc1': ('linkc', 'cc'),
69 63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest Patrick Mezard <pmezard@gmail.com> parents: 68 diff changeset	45 },
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest Patrick Mezard <pmezard@gmail.com> parents: 68 diff changeset	46 9: {
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest Patrick Mezard <pmezard@gmail.com> parents: 68 diff changeset	47 'unchanged2': ('unchanged', 'unchanged\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	48 'unchangedlink2': ('unchangedlink', 'unchanged'),
69 63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest Patrick Mezard <pmezard@gmail.com> parents: 68 diff changeset	49 'unchangeddir2/f': ('unchangeddir/f', 'unchanged2\n'),
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	50 'unchangeddir2/link': ('unchangeddir/link', 'f'),
496 5e0dfe59d4c3 copies: fix under-reporting of copies in hg Augie Fackler <durin42@gmail.com> parents: 132 diff changeset	51 },
5e0dfe59d4c3 copies: fix under-reporting of copies in hg Augie Fackler <durin42@gmail.com> parents: 132 diff changeset	52 10: {
1233 0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	53 'groupdir2/b': ('groupdir/b', 'b\n'),
0d0132cba155 editor: fix edge case with in memory file-store size limit David Schleimer <dschleimer@fb.com> parents: 1064 diff changeset	54 'groupdir2/linkb': ('groupdir/linkb', 'b'),
496 5e0dfe59d4c3 copies: fix under-reporting of copies in hg Augie Fackler <durin42@gmail.com> parents: 132 diff changeset	55 },
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	56 }
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	57 for rev in repo:
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	58 ctx = repo[rev]
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	59 copymap = copies.get(rev, {})
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	60 for f in ctx.manifest():
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	61 cp = ctx[f].renamed()
69 63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest Patrick Mezard <pmezard@gmail.com> parents: 68 diff changeset	62 self.assertEqual(bool(cp), bool(copymap.get(f)),
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest Patrick Mezard <pmezard@gmail.com> parents: 68 diff changeset	63 'copy records differ for %s in %d' % (f, rev))
67 e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	64 if not cp:
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	65 continue
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	66 self.assertEqual(cp[0], copymap[f][0])
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	67 self.assertEqual(ctx[f].data(), copymap[f][1])
e319c9168910 hg_delta_editor: register svn file copies Patrick Mezard <pmezard@gmail.com> parents: diff changeset	68
130 c2a84d436202 test_fetch_rename: test content of files coming from the past Patrick Mezard <pmezard@gmail.com> parents: 101 diff changeset	69 self.assertEqual(repo['tip']['changed3'].data(), 'changed\nchanged3\n')
c2a84d436202 test_fetch_rename: test content of files coming from the past Patrick Mezard <pmezard@gmail.com> parents: 101 diff changeset	70
1064 dc516f85b6c3 test_fetch_renames: use stupid mode metaclass Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 1044 diff changeset	71 def test_case(self):
dc516f85b6c3 test_fetch_renames: use stupid mode metaclass Dan Villiom Podlaski Christiansen <danchr@gmail.com> parents: 1044 diff changeset	72 repo = self._load_fixture_and_fetch('filecase.svndump')
132 3a9d6cd18332 test_fetch_renames: test file and directory case changes Patrick Mezard <pmezard@gmail.com> parents: 130 diff changeset	73 files = {
3a9d6cd18332 test_fetch_renames: test file and directory case changes Patrick Mezard <pmezard@gmail.com> parents: 130 diff changeset	74 0: ['A', 'a', 'e/a', 'b', 'd/a', 'D/a', 'f/a', 'F'],
3a9d6cd18332 test_fetch_renames: test file and directory case changes Patrick Mezard <pmezard@gmail.com> parents: 130 diff changeset	75 1: ['A', 'a', 'E/a', 'B', 'd/A', 'D/a', 'f/a', 'F'],
3a9d6cd18332 test_fetch_renames: test file and directory case changes Patrick Mezard <pmezard@gmail.com> parents: 130 diff changeset	76 }
3a9d6cd18332 test_fetch_renames: test file and directory case changes Patrick Mezard <pmezard@gmail.com> parents: 130 diff changeset	77 for rev in repo:
3a9d6cd18332 test_fetch_renames: test file and directory case changes Patrick Mezard <pmezard@gmail.com> parents: 130 diff changeset	78 self.assertEqual(sorted(files[rev]), sorted(repo[rev].manifest()))

Mercurial > hgsubversion

annotate tests/test_fetch_renames.py @ 1233:0d0132cba155