annotate mercurial/streamclone.py @ 4865:439e2f2fde42

Fix inconsistency for the stream_out capability in hgweb During some experiments of mine, the uncompressed cloning could not be enabled for hgweb.cgi nor hgwebdir.cgi though the server claimed to be stream_out capable. The only solution was to enable it using the user's .hgrc file. This solution is not acceptable when publishing the repos through an HTTP server because the CGI runs as a www dedicated user whose's home hgrc file may not be accessible to users publishing their repos through their userdir. For such cases we could end up with this typical debug output: hg --debug clone --uncompressed http://server/hg/project destination directory: project sending capabilities command capabilities: lookup changegroupsubset stream=1 unbundle=HG10GZ,HG10BZ,HG10UN sending stream_out command abort: operation forbidden by server The error lies in the fact the hgweb object defines new accessors to the repo configuration that trust things by default (untrusted=True) but the streamclone:stream_out function uses the usual accessors to the repo.ui object, which do not trust by default (untrusted=False) Fix this inconsistency, adding a new parameter to the stream_out function. hgweb then forces a "trust by default" behavior.
author Edouard Gomez <ed.gomez@free.fr>
date Sat, 12 May 2007 00:41:30 +0200
parents 9dc64c8414ca
children 97b734fb9c6f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
1 # streamclone.py - streaming clone server support for mercurial
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
2 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
3 # Copyright 2006 Vadim Gelfer <vadim.gelfer@gmail.com>
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
4 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
5 # This software may be used and distributed according to the terms
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
6 # of the GNU General Public License, incorporated herein by reference.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
7
3893
6b4127c7d52a Simplify i18n imports
Matt Mackall <mpm@selenic.com>
parents: 3886
diff changeset
8 from i18n import _
3886
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3856
diff changeset
9 import os, stat, util, lock
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
10
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
11 # if server supports streaming clone, it advertises "stream"
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
12 # capability with value that is version+flags of repo it is serving.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
13 # client only streams if it can read that repo format.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
14
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
15 def walkrepo(root):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
16 '''iterate over metadata files in repository.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
17 walk in natural (sorted) order.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
18 yields 2-tuples: name of .d or .i file, size of file.'''
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
19
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
20 strip_count = len(root) + len(os.sep)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
21 def walk(path, recurse):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
22 ents = os.listdir(path)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
23 ents.sort()
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
24 for e in ents:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
25 pe = os.path.join(path, e)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
26 st = os.lstat(pe)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
27 if stat.S_ISDIR(st.st_mode):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
28 if recurse:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
29 for x in walk(pe, True):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
30 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
31 else:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
32 if not stat.S_ISREG(st.st_mode) or len(e) < 2:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
33 continue
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
34 sfx = e[-2:]
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
35 if sfx in ('.d', '.i'):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
36 yield pe[strip_count:], st.st_size
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
37 # write file data first
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
38 for x in walk(os.path.join(root, 'data'), True):
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
39 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
40 # write manifest before changelog
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
41 meta = list(walk(root, False))
2623
d1cbfe9e13cd fix problem with uncompressed clone and python 2.3.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2622
diff changeset
42 meta.sort()
d1cbfe9e13cd fix problem with uncompressed clone and python 2.3.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2622
diff changeset
43 meta.reverse()
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
44 for x in meta:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
45 yield x
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
46
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
47 # stream file format is simple.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
48 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
49 # server writes out line that says how many files, how many total
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
50 # bytes. separator is ascii space, byte counts are strings.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
51 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
52 # then for each file:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
53 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
54 # server writes out line that says file name, how many bytes in
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
55 # file. separator is ascii nul, byte count is string.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
56 #
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
57 # server writes out raw file data.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
58
4865
439e2f2fde42 Fix inconsistency for the stream_out capability in hgweb
Edouard Gomez <ed.gomez@free.fr>
parents: 4134
diff changeset
59 def stream_out(repo, fileobj, untrusted=False):
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
60 '''stream out all metadata files in repository.
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
61 writes to file-like object, must support write() and optional flush().'''
2621
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
62
4865
439e2f2fde42 Fix inconsistency for the stream_out capability in hgweb
Edouard Gomez <ed.gomez@free.fr>
parents: 4134
diff changeset
63 if not repo.ui.configbool('server', 'uncompressed', untrusted=untrusted):
2621
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
64 fileobj.write('1\n')
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
65 return
5a5852a417b1 clone: disable stream support on server side by default.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2612
diff changeset
66
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
67 # get consistent snapshot of repo. lock during scan so lock not
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
68 # needed while we stream, and commits can happen.
3687
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
69 try:
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
70 repolock = repo.lock()
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
71 except (lock.LockHeld, lock.LockUnavailable), inst:
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
72 repo.ui.warn('locking the repository failed: %s\n' % (inst,))
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
73 fileobj.write('2\n')
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
74 return
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
75
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
76 fileobj.write('0\n')
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
77 repo.ui.debug('scanning\n')
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
78 entries = []
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
79 total_bytes = 0
3791
8643b9f90b51 introduce localrepo.spath for the store path, sopener fixes
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3721
diff changeset
80 for name, size in walkrepo(repo.spath):
4127
a3a6f71edf2e streamclone: stream_out tried to decodefilename non-normalized paths
Patrick Mezard <pmezard@gmail.com>
parents: 3856
diff changeset
81 name = repo.decodefn(util.pconvert(name))
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
82 entries.append((name, size))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
83 total_bytes += size
3687
d5dd0a2a44bc Handle locking exceptions if streaming clone can't lock the repo. (Issue324)
Thomas Arendsen Hein <thomas@intevation.de>
parents: 2623
diff changeset
84 repolock.release()
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
85
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
86 repo.ui.debug('%d files, %d bytes to transfer\n' %
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
87 (len(entries), total_bytes))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
88 fileobj.write('%d %d\n' % (len(entries), total_bytes))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
89 for name, size in entries:
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
90 repo.ui.debug('sending %s (%d bytes)\n' % (name, size))
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
91 fileobj.write('%s\0%d\n' % (name, size))
3791
8643b9f90b51 introduce localrepo.spath for the store path, sopener fixes
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3721
diff changeset
92 for chunk in util.filechunkiter(repo.sopener(name), limit=size):
2612
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
93 fileobj.write(chunk)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
94 flush = getattr(fileobj, 'flush', None)
ffb895f16925 add support for streaming clone.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
95 if flush: flush()