mercurial/changegroup.py
author Thomas Arendsen Hein <thomas@intevation.de>
Fri, 01 Dec 2006 23:27:53 +0100
changeset 3755 05120e210c65
parent 3706 0d810798acb1
child 3762 b9d3e12da485
permissions -rw-r--r--
Use unsigned version format. This way can use one additional bit, and when encountering invalid revlogs with the first bit set don't produce python warnings or strange error messages.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     1
"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     2
changegroup.py - Mercurial changegroup manipulation functions
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     3
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     4
 Copyright 2006 Matt Mackall <mpm@selenic.com>
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     5
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     6
This software may be used and distributed according to the terms
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     7
of the GNU General Public License, incorporated herein by reference.
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
     8
"""
1997
802e8a029d99 Added missing gettext import to changegroup.py.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1981
diff changeset
     9
from i18n import gettext as _
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    10
from demandload import *
3669
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    11
demandload(globals(), "struct os bz2 zlib util tempfile")
1981
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    12
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    13
def getchunk(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    14
    """get a chunk from a changegroup"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    15
    d = source.read(4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    16
    if not d:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    17
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    18
    l = struct.unpack(">l", d)[0]
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    19
    if l <= 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    20
        return ""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    21
    d = source.read(l - 4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    22
    if len(d) < l - 4:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    23
        raise util.Abort(_("premature EOF reading chunk"
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    24
                           " (got %d bytes, expected %d)")
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    25
                          % (len(d), l - 4))
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    26
    return d
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    27
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    28
def chunkiter(source):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    29
    """iterate through the chunks in source"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    30
    while 1:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    31
        c = getchunk(source)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    32
        if not c:
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    33
            break
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    34
        yield c
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    35
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    36
def genchunk(data):
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    37
    """build a changegroup chunk"""
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    38
    header = struct.pack(">l", len(data)+ 4)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    39
    return "%s%s" % (header, data)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    40
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    41
def closechunk():
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    42
    return struct.pack(">l", 0)
736b6c96bbbc make incoming work via ssh (issue139); move chunk code into separate module.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
diff changeset
    43
3666
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    44
class nocompress(object):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    45
    def compress(self, x):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    46
        return x
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    47
    def flush(self):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    48
        return ""
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    49
3669
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    50
bundletypes = {
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    51
    "": ("", nocompress),
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    52
    "HG10UN": ("HG10UN", nocompress),
3705
e9d339d5d2d7 BZ2Compressor already defaults to compresslevel=9.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3704
diff changeset
    53
    "HG10BZ": ("HG10", bz2.BZ2Compressor),
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    54
    "HG10GZ": ("HG10GZ", zlib.compressobj),
3669
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    55
}
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    56
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    57
def writebundle(cg, filename, bundletype):
3666
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    58
    """Write a bundle file and return its filename.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    59
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    60
    Existing files will not be overwritten.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    61
    If no filename is specified, a temporary file is created.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    62
    bz2 compression can be turned off.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    63
    The bundle file will be deleted in case of errors.
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    64
    """
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    65
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    66
    fh = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    67
    cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    68
    try:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    69
        if filename:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    70
            if os.path.exists(filename):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    71
                raise util.Abort(_("file '%s' already exists") % filename)
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    72
            fh = open(filename, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    73
        else:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    74
            fd, filename = tempfile.mkstemp(prefix="hg-bundle-", suffix=".hg")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    75
            fh = os.fdopen(fd, "wb")
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    76
        cleanup = filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    77
3706
0d810798acb1 Use 'bundletype' instead of 'type' to not shadow built-in function.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 3705
diff changeset
    78
        header, compressor = bundletypes[bundletype]
3704
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    79
        fh.write(header)
9c1737a3e254 fix writebundle for bz2 bundles
Benoit Boissinot <benoit.boissinot@ens-lyon.org>
parents: 3669
diff changeset
    80
        z = compressor()
3669
f4dc02d7fb71 unduplicate bundle writing code from httprepo
Matt Mackall <mpm@selenic.com>
parents: 3667
diff changeset
    81
3666
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    82
        # parse the changegroup data, otherwise we will block
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    83
        # in case of sshrepo because we don't know the end of the stream
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    84
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    85
        # an empty chunkiter is the end of the changegroup
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    86
        empty = False
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    87
        while not empty:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    88
            empty = True
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    89
            for chunk in chunkiter(cg):
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    90
                empty = False
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    91
                fh.write(z.compress(genchunk(chunk)))
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    92
            fh.write(z.compress(closechunk()))
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    93
        fh.write(z.flush())
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    94
        cleanup = None
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    95
        return filename
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    96
    finally:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    97
        if fh is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    98
            fh.close()
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
    99
        if cleanup is not None:
025f68f22ae2 move write_bundle to changegroup.py
Matt Mackall <mpm@selenic.com>
parents: 2470
diff changeset
   100
            os.unlink(cleanup)
3667
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   101
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   102
def readbundle(fh):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   103
    header = fh.read(6)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   104
    if not header.startswith("HG"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   105
        raise util.Abort(_("%s: not a Mercurial bundle file") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   106
    elif not header.startswith("HG10"):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   107
        raise util.Abort(_("%s: unknown bundle version") % fname)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   108
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   109
    if header == "HG10BZ":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   110
        def generator(f):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   111
            zd = bz2.BZ2Decompressor()
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   112
            zd.decompress("BZ")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   113
            for chunk in util.filechunkiter(f, 4096):
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   114
                yield zd.decompress(chunk)
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   115
        return util.chunkbuffer(generator(fh))
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   116
    elif header == "HG10UN":
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   117
        return fh
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   118
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   119
    raise util.Abort(_("%s: unknown bundle compression type")
8500a13ec44b create a readbundle function
Matt Mackall <mpm@selenic.com>
parents: 3666
diff changeset
   120
                     % fname)