Mercurial > hg > mercurial-crew-with-dirclash

annotate mercurial/streamclone.py @ 5369:7530334bf301

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
revlog: generate trivial deltas against null revision To avoid extra memory usage and performance issues with large files, generate a trivial delta header for deltas against the null revision rather than calling the usual delta generator. We append the delta header to meta rather than prepending it to data to avoid a large allocate and copy.
author Matt Mackall <mpm@selenic.com>
date Wed, 03 Oct 2007 17:17:27 -0500
parents 97b734fb9c6f
children 5105b119edd2
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
1 # streamclone.py - streaming clone server support for mercurial
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
2 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
3 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
4 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
6 # of the GNU General Public License, incorporated herein by reference.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
7
3893
6b4127c7d52a Simplify i18n imports
Matt Mackall <mpm@selenic.com>
parents: 3886
diff changeset
8 from i18n import _
3886
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3856
diff changeset
9 import os, stat, util, lock
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
10
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
11 # if server supports streaming clone, it advertises "stream"
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
12 # capability with value that is version+flags of repo it is serving.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
13 # client only streams if it can read that repo format.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
14
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
15 def walkrepo(root):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
16 '''iterate over metadata files in repository.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
17 walk in natural (sorted) order.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
18 yields 2-tuples: name of .d or .i file, size of file.'''
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
19
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
20 strip_count = len(root) + len(os.sep)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
21 def walk(path, recurse):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
22 ents = os.listdir(path)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
23 ents.sort()
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
24 for e in ents:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
25 pe = os.path.join(path, e)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
26 st = os.lstat(pe)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
27 if stat.S_ISDIR(st.st_mode):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
28 if recurse:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
29 for x in walk(pe, True):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
30 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
31 else:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
32 if not stat.S_ISREG(st.st_mode) or len(e) < 2:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
33 continue
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
34 sfx = e[-2:]
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
35 if sfx in ('.d', '.i'):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
36 yield pe[strip_count:], st.st_size
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
37 # write file data first
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
38 for x in walk(os.path.join(root, 'data'), True):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
39 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
40 # write manifest before changelog
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
41 meta = list(walk(root, False))
2623
d1cbfe9e13cd fix problem with uncompressed clone and python 2.3.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2622
diff changeset
42 meta.sort()
d1cbfe9e13cd fix problem with uncompressed clone and python 2.3.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2622
diff changeset
43 meta.reverse()
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
44 for x in meta:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
45 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
46
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
47 # stream file format is simple.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
48 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
49 # server writes out line that says how many files, how many total
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
50 # bytes. separator is ascii space, byte counts are strings.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
51 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
52 # then for each file:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
53 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
54 # server writes out line that says file name, how many bytes in
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
55 # file. separator is ascii nul, byte count is string.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
56 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
57 # server writes out raw file data.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
58
4865
439e2f2fde42 Fix inconsistency for the stream_out capability in hgweb
Edouard Gomez <ed.gomez@free.fr>
parents: 4134
diff changeset
59 def stream_out(repo, fileobj, untrusted=False):
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
60 '''stream out all metadata files in repository.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
61 writes to file-like object, must support write() and optional flush().'''
2621
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
62
4865
439e2f2fde42 Fix inconsistency for the stream_out capability in hgweb
Edouard Gomez <ed.gomez@free.fr>
parents: 4134
diff changeset
63 if not repo.ui.configbool('server', 'uncompressed', untrusted=untrusted):
2621
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
64 fileobj.write('1\n')
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
65 return
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
66
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
67 # get consistent snapshot of repo. lock during scan so lock not
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
68 # needed while we stream, and commits can happen.
4959
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
69 lock = None
3687
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
70 try:
4959
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
71 try:
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
72 repolock = repo.lock()
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
73 except (lock.LockHeld, lock.LockUnavailable), inst:
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
74 repo.ui.warn('locking the repository failed: %s\n' % (inst,))
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
75 fileobj.write('2\n')
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
76 return
3687
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
77
4959
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
78 fileobj.write('0\n')
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
79 repo.ui.debug('scanning\n')
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
80 entries = []
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
81 total_bytes = 0
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
82 for name, size in walkrepo(repo.spath):
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
83 name = repo.decodefn(util.pconvert(name))
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
84 entries.append((name, size))
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
85 total_bytes += size
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
86 finally:
97b734fb9c6f Use try/finally pattern to cleanup locks and transactions
Matt Mackall <mpm@selenic.com>
parents: 4865
diff changeset
87 del repolock
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
88
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
89 repo.ui.debug('%d files, %d bytes to transfer\n' %
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
90 (len(entries), total_bytes))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
91 fileobj.write('%d %d\n' % (len(entries), total_bytes))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
92 for name, size in entries:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
93 repo.ui.debug('sending %s (%d bytes)\n' % (name, size))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
94 fileobj.write('%s\0%d\n' % (name, size))
3791
8643b9f90b51 introduce localrepo.spath for the store path, sopener fixes
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3721
diff changeset
95 for chunk in util.filechunkiter(repo.sopener(name), limit=size):
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
96 fileobj.write(chunk)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
97 flush = getattr(fileobj, 'flush', None)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
98 if flush: flush()