annotate tests/fixtures/renames.sh @ 931:e1dbd9646d6a

svnwrap: use custom StringIO class in get_file() The wrappers were calling ra.get_file() with a cStringIO object. Empirically, svn 1.7.5 is writing 16kB blocks to the stream object, and cStringIO reallocates its internal buffer and doubles its size whenever it is filled. With large committed files this requires two large memory blocks at the same time. SimpleStringIO implements the mimimum StringIO interface used by ra.get_file() but instead stores all the blocks and "join" them at the end. It means more fragmentation but requires only one large block, without overallocation. Also, 16kB blocks should be friendly to most allocators. In practice, this simple change let me convert a revision containing multiple moderately large files, the largest being around 450MB, with a 32-bits Windows setup, python 2.7, swig svn 1.7.5, in stupid mode, while it was previously aborting with "not enough memory". The same revision still fails in replay mode.
author Patrick Mezard <patrick@mezard.eu>
date Sun, 16 Sep 2012 19:31:49 +0200
parents c2a84d436202
children 0d0132cba155
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
67
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
1 #!/bin/sh
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
2 #
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
3 # Generate renames.svndump
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
4 #
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
5
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
6 mkdir temp
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
7 cd temp
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
8
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
9 mkdir project-orig
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
10 cd project-orig
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
11 mkdir trunk
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
12 mkdir branches
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
13 cd ..
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
14
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
15 svnadmin create testrepo
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
16 svnurl=file://`pwd`/testrepo
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
17 svn import project-orig $svnurl -m "init project"
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
18
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
19 svn co $svnurl project
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
20 cd project/trunk
69
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
21 # Entries for regular tests
67
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
22 echo a > a
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
23 echo b > b
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
24 mkdir -p da/db
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
25 echo c > da/daf
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
26 echo d > da/db/dbf
69
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
27 # Entries to test delete + copy
68
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
28 echo deleted > deletedfile
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
29 mkdir deleteddir
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
30 echo deleteddir > deleteddir/f
69
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
31 # Entries to test copy before change
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
32 echo changed > changed
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
33 mkdir changeddir
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
34 echo changed2 > changeddir/f
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
35 # Entries unchanged in the rest of history
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
36 echo unchanged > unchanged
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
37 mkdir unchangeddir
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
38 echo unchanged2 > unchangeddir/f
71
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
39 # One of the files will be changed afterwards, to test
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
40 # group copies detection
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
41 mkdir groupdir
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
42 echo a > groupdir/a
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
43 echo b > groupdir/b
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
44 svn add a b da deletedfile deleteddir changed changeddir unchanged unchangeddir groupdir
67
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
45 svn ci -m "add a and b"
69
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
46 # Remove files to be copied later
68
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
47 svn rm deletedfile
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
48 svn rm deleteddir
69
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
49 # Update files to be copied before this change
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
50 echo changed >> changed
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
51 echo changed2 >> changeddir/f
71
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
52 # Update one of the groupdir files
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
53 echo a >> groupdir/a
68
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
54 svn ci -m "delete files and dirs"
67
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
55 cd ../branches
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
56 svn cp ../trunk branch1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
57 svn ci -m "create branch1"
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
58 cd branch1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
59 echo c > c
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
60 svn add c
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
61 svn ci -m "add c"
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
62 cd ../../trunk
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
63 # Regular copy and rename
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
64 svn cp a a1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
65 svn mv a a2
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
66 # Copy and update of source and dest
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
67 svn cp b b1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
68 echo b >> b
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
69 echo c >> b1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
70 # Directory copy and renaming
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
71 svn cp da da1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
72 svn mv da da2
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
73 # Test one copy operation in branch
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
74 cd ../branches/branch1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
75 svn cp c c1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
76 echo c >> c1
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
77 cd ../..
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
78 svn ci -m "rename and copy a, b and da"
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
79 cd trunk
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
80 # Copy across branch
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
81 svn cp ../branches/branch1/c c
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
82 svn ci -m "copy b from branch1"
68
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
83 # Copy deleted stuff from the past
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
84 svn cp $svnurl/trunk/deletedfile@2 deletedfile
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
85 svn cp $svnurl/trunk/deleteddir@2 deleteddir
e0c86ebe05e3 test_fetch_renames: test copy of deleted stuff from the past
Patrick Mezard <pmezard@gmail.com>
parents: 67
diff changeset
86 svn ci -m "copy stuff from the past"
69
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
87 # Copy data from the past before it was changed
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
88 svn cp $svnurl/trunk/changed@2 changed2
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
89 svn cp $svnurl/trunk/changeddir@2 changeddir2
130
c2a84d436202 test_fetch_rename: test content of files coming from the past
Patrick Mezard <pmezard@gmail.com>
parents: 71
diff changeset
90 # Harder, copy from the past before change and change it again
c2a84d436202 test_fetch_rename: test content of files coming from the past
Patrick Mezard <pmezard@gmail.com>
parents: 71
diff changeset
91 # This confused the stupid diff path
c2a84d436202 test_fetch_rename: test content of files coming from the past
Patrick Mezard <pmezard@gmail.com>
parents: 71
diff changeset
92 svn cp $svnurl/trunk/changed@2 changed3
c2a84d436202 test_fetch_rename: test content of files coming from the past
Patrick Mezard <pmezard@gmail.com>
parents: 71
diff changeset
93 echo changed3 >> changed3
69
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
94 svn ci -m "copy stuff from the past before change"
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
95 # Copy unchanged stuff from the past. Since no changed occured in these files
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
96 # between the source and parent revision, we record them as copy from parent
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
97 # instead of source rev.
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
98 svn cp $svnurl/trunk/unchanged@2 unchanged2
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
99 svn cp $svnurl/trunk/unchangeddir@2 unchangeddir2
63ece4ea25c9 hg_delta_editor: register copies only if files are unchanged between source and dest
Patrick Mezard <pmezard@gmail.com>
parents: 68
diff changeset
100 svn ci -m "copy unchanged stuff from the past"
71
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
101 # Copy groupdir, unfortunately one file was changed after r2 so the
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
102 # copy should not be recorded at all
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
103 svn cp $svnurl/trunk/groupdir@2 groupdir2
bf1e8b8ed452 test_fetch_renames: test directory copy with children files changed in-between
Patrick Mezard <pmezard@gmail.com>
parents: 69
diff changeset
104 svn ci -m "copy groupdir from the past"
67
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
105 cd ../..
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
106
e319c9168910 hg_delta_editor: register svn file copies
Patrick Mezard <pmezard@gmail.com>
parents:
diff changeset
107 svnadmin dump testrepo > ../renames.svndump