Mercurial > hgsubversion

annotate notes/metadata.txt @ 1233:0d0132cba155

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
editor: fix edge case with in memory file-store size limit There are a few cases where we will set a single file into to the editor's FileStore object more than once. Notably, for copied and then modified files, we will set it at least twice. Three times if editing fails (which it can for symlinks). If we pass the in-memory storage limit in between the first (or second if editing fails) time we set the file and the last time we set the file, we will write the data to the in memory store the first time and the file store the last time. We didn't remove it form the in-memory store though, and we always prefer reading from the in-memory store. This means we can sometimes end up with the wrong version of a file. This is fairly unlikely to happen in normal use since you need to hit the memory limit between two writes to the store for the same file. We only write a file multiple times if a) the file (and not one of it's parent directories) is copied and then modified or b) editing fails. From what I can tell, it's only common for editing to fail for symlinks, and they ten to be relatively small data that is unlikely to push over the limit. Finally, the default limit is 100MB which I would expect to be most often either well over (source code) or well under (binaries or automated changes) the size of the changes files in a single commit. The easiest way to reproduce this is to set the in-memory cache size to 0 and then commit a copied and modified symlink. The empty-string version from the failed editing will be the one that persists. I happened to stumble upon this while trying (and failing) to test a bug-fix for a related bug with identical symptoms (empty simlink). I have seen this in the wild, once, but couldn't reproduce it at the time. The repo in question is quite large and quite active, so I am quite confident in my estimation that this is a real, but very rare, problem. The test changes attached to this was mneant to test a related bug, but turned out not to actually cover the bug in question. They did trigger this bug though, and are worthwhile to test, so I kept them.
author David Schleimer <dschleimer@fb.com>
date Mon, 07 Apr 2014 17:51:59 -0700
parents ba801f44d240
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
134
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
1 Branches
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
2 --------
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
3 In order to handle branches that are not immediate children of /branches/, the
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
4 following information must be stored in the revmap:
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
5
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
6 revision path
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
7
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
8 Where path is the actual relative path of the branch in svn. An example, with
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
9 the previous format for clarification:
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
10 New | Old
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
11 3 <hash> trunk | 3 <hash>
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
12 4 <hash> branches/foo | 4 <hash> foo
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
13
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
14 Tags
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
15 ----
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
16 Note that if a tag is committed to, we can handle that case by making a branch
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
17 and then marking it as deleted. The revmap line would look something like this:
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
18 10 <hash> tags/the_tag
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
19 And the commit would be done on the hg branch 'modified-tag/the_tag'. Note that
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
20 if this was 'tags/releases/1.0.0', then the branch would be
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
21 'modified-tag/releases/1.0.0'.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
22
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
23 Detecting Closing of Branches
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
24 -----------------------------
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
25 Subversion users typically remove branches when done with them. This means that
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
26 if a commit performs a delete operation on the '' path inside a branch, we can
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
27 be sure that the branch no longer exists. The branch should then be marked as
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
28 inactive.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
29
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
30 Closing Branches
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
31 ----------------
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
32 As of this writing, Mercurial marks branches as inactive by merging them so they
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
33 have no active heads. In order to mark a branch as closed, the active head on
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
34 the branch will be used as the first parent of the new changeset, and the second
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
35 changeset will be either the active head on the hg branch 'closed-branches' or
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
36 be the nullrev. The commit to mark the branch as inactive will happen on the
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
37 'closed-branches' branch in Mercurial.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
38
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
39 Recovery
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
40 --------
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
41 hgsubversion stores several pieces of essential metadata in .hg/svn/. In order
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
42 to rebuild this data, the key 'convert_revision' should be stored in the extra
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
43 dictionary of the converted revision. The key should contain data in the format:
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
44 'svn:<uuid>/abs/path@<rev>' where <uuid> is the repo UUID, /abs/path is the
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
45 absolute path to the location the edits were made in Subversion (that is, if it
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
46 was a trunk commit on /foo/trunk, then /foo/trunk is what gets stored here, even
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
47 though the project root does not equal the repo root), and <rev> is the revision
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
48 number of the change in Subversion. This key (and its contents) have been
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
49 chosen to be compatible with the convert extension so that repos originally
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
50 converted to hg using convert can be maintained using hgsubversion if desired.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
51
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
52 Tags (tag_info) can be reconstructed by listing the tags directory and then
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
53 running log on each tag to determine its parent changeset.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
54
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
55 The last revision converted (last_rev) can be converted simply by using the
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
56 highest revision number encountered while rebuilding the revision map.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
57
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
58 The legacy tag_locations file does not need to be replaced - it will be
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
59 obsoleted as part of the long-term branch refactor.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
60
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
61 The url will have to be provided by the user. The uuid can be re-requested from
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
62 the repository.
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
63
22248b34b15a Add notes on how metadata is stored and recovered. Note that at this point,
Augie Fackler <durin42@gmail.com>
parents:
diff changeset
64 branch_info can be rebuilt during the rebuild of the revision map by recording
155
ba801f44d240 utility_commands: Implement rebuildmeta so that metadata can be rebuilt.
Augie Fackler <durin42@gmail.com>
parents: 134
diff changeset
65 the revisions of all active heads of server-side branches. branch_info contains
ba801f44d240 utility_commands: Implement rebuildmeta so that metadata can be rebuilt.
Augie Fackler <durin42@gmail.com>
parents: 134
diff changeset
66 a map from branch: (parent_branch, parent_branch_rev, branch_created_rev)