mercurial/changegroup.py
author Maxim Dounin <mdounin@mdounin.ru>
Sat, 27 Oct 2007 16:27:55 +0400
changeset 5483 0c43f87baba3
parent 5370 61462e7d62ed
permissions -rw-r--r--
Fix file-changed-to-dir and dir-to-file commits (issue660). Allow adding to dirstate files that clash with previously existing but marked for removal. Protect from reintroducing clashes by revert. This change doesn't address related issues with update. Current workaround is to do "clean" update by manually removing conflicting files/dirs from working directory.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     1
"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     2
changegroup.py - Mercurial changegroup manipulation functions
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     3
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     4
 Copyright 2006 Matt Mackall <mpm@selenic.com>
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     5
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     6
This software may be used and distributed according to the terms
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     7
of the GNU General Public License, incorporated herein by reference.
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     8
"""
3886
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
     9
3893
6b4127c7d52a Simplify i18n imports
Matt Mackall <mpm@selenic.com>
parents: 3886
diff changeset
    10
from i18n import _
3886
abaee83ce0a6 Replace demandload with new demandimport
Matt Mackall <mpm@selenic.com>
parents: 3859
diff changeset
    11
import struct, os, bz2, zlib, util, tempfile
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    12
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    13
def getchunk(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    14
    """get a chunk from a changegroup"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    15
    d = source.read(4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    16
    if not d:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    17
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    18
    l = struct.unpack(">l", d)[0]
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    19
    if l <= 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    20
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    21
    d = source.read(l - 4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    22
    if len(d) < l - 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    23
        raise util.Abort(_("premature EOF reading chunk"
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    24
                           " (got %d bytes, expected %d)")
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    25
                          % (len(d), l - 4))
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    26
    return d
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    27
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    28
def chunkiter(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    29
    """iterate through the chunks in source"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    30
    while 1:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    31
        c = getchunk(source)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    32
        if not c:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    33
            break
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    34
        yield c
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    35
5370
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    36
def chunkheader(length):
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    37
    """build a changegroup chunk header"""
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    38
    return struct.pack(">l", length + 4)
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    39
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    40
def closechunk():
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    41
    return struct.pack(">l", 0)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    42
3666
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    43
class nocompress(object):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    44
    def compress(self, x):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    45
        return x
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    46
    def flush(self):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    47
        return ""
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    48
3669
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    49
bundletypes = {
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    50
    "": ("", nocompress),
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    51
    "HG10UN": ("HG10UN", nocompress),
3762
b9d3e12da485 changegroup.py: delay the loading of the bz2 and zlib modules
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3706
diff changeset
    52
    "HG10BZ": ("HG10", lambda: bz2.BZ2Compressor()),
b9d3e12da485 changegroup.py: delay the loading of the bz2 and zlib modules
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 3706
diff changeset
    53
    "HG10GZ": ("HG10GZ", lambda: zlib.compressobj()),
3669
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    54
}
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    55
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    56
def writebundle(cg, filename, bundletype):
3666
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    57
    """Write a bundle file and return its filename.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    58
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    59
    Existing files will not be overwritten.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    60
    If no filename is specified, a temporary file is created.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    61
    bz2 compression can be turned off.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    62
    The bundle file will be deleted in case of errors.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    63
    """
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    64
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    65
    fh = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    66
    cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    67
    try:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    68
        if filename:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    69
            fh = open(filename, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    70
        else:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    71
            fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    72
            fh = os.fdopen(fd, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    73
        cleanup = filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    74
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    75
        header, compressor = bundletypes[bundletype]
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    76
        fh.write(header)
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    77
        z = compressor()
3669
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    78
3666
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    79
        # parse the changegroup data, otherwise we will block
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    80
        # in case of sshrepo because we don't know the end of the stream
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    81
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    82
        # an empty chunkiter is the end of the changegroup
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    83
        empty = False
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    84
        while not empty:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    85
            empty = True
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    86
            for chunk in chunkiter(cg):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    87
                empty = False
5370
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    88
                fh.write(z.compress(chunkheader(len(chunk))))
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    89
                pos = 0
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    90
                while pos < len(chunk):
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    91
                    next = pos + 2**20
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    92
                    fh.write(z.compress(chunk[pos:next]))
61462e7d62ed changegroup: avoid large copies
Matt Mackall <mpm@selenic.com>
parents: 3932
diff changeset
    93
                    pos = next
3666
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    94
            fh.write(z.compress(closechunk()))
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    95
        fh.write(z.flush())
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    96
        cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    97
        return filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    98
    finally:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    99
        if fh is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   100
            fh.close()
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   101
        if cleanup is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   102
            os.unlink(cleanup)
3667
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   103
3859
8c24b6fd5866 fix errors spotted by pychecker
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3762
diff changeset
   104
def readbundle(fh, fname):
3667
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   105
    header = fh.read(6)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   106
    if not header.startswith("HG"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   107
        raise util.Abort(_("%s: not a Mercurial bundle file") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   108
    elif not header.startswith("HG10"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   109
        raise util.Abort(_("%s: unknown bundle version") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   110
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   111
    if header == "HG10BZ":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   112
        def generator(f):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   113
            zd = bz2.BZ2Decompressor()
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   114
            zd.decompress("BZ")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   115
            for chunk in util.filechunkiter(f, 4096):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   116
                yield zd.decompress(chunk)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   117
        return util.chunkbuffer(generator(fh))
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   118
    elif header == "HG10UN":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   119
        return fh
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   120
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   121
    raise util.Abort(_("%s: unknown bundle compression type")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   122
                     % fname)