mercurial/sshrepo.py
author mason@suse.com
Thu, 01 Sep 2005 07:34:53 -0700
changeset 1183 d9e85a75dbda
parent 1103 808a9f0e7af0
child 1251 84cf8834efb5
permissions -rw-r--r--
Optimize dirstate walking This generally cuts the time for hg status/diff in half, from 2s down to 1s. The main parts I'm trying to optimize are: 1) os.walk stats every file. dirstate.changes then stats every file again. 2) os.walk yields every file and subdir to dirstate.traverse who yields every file and everything in the dirstate map. dirstate.walk then filters this mass and yields every file to the caller. There should be fewer steps in here, and fewer duplicate strings yielded. 3) dirstate.walk runs util.unique on the results from dirstate.traverse, even though it is also passing things through dirstate.seen to look for duplicates. I've turned os.walk into something hg specific that takes all the dirstate ignore and matching rules into account. The new function also takes an function arg (statmatch()) the caller supplies to help filter out files it doesn't care about. dirstate.changes uses this to update state for each file, avoiding the second stat call. dirstate.walk is changed to turn the match function it is passed into a statmatch function. The only real difference is that a statmatch function takes the stat data as a second parameter. It now calls dirstate.walkhelper, who requires a statmatch function to be passed. This fails test-walk, but right now I think this is from a sorting error fixed by this patch. Index: crew/mercurial/dirstate.py ===================================================================
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1096
ae4f1f48c569 sshrepo: adjust file comment
mpm@selenic.com
parents: 1089
diff changeset
     1
# sshrepo.py - ssh repository proxy class for mercurial
0
9117c6561b0b Add back links from file revisions to changeset revisions
mpm@selenic.com
parents:
diff changeset
     2
#
9117c6561b0b Add back links from file revisions to changeset revisions
mpm@selenic.com
parents:
diff changeset
     3
# Copyright 2005 Matt Mackall <mpm@selenic.com>
9117c6561b0b Add back links from file revisions to changeset revisions
mpm@selenic.com
parents:
diff changeset
     4
#
9117c6561b0b Add back links from file revisions to changeset revisions
mpm@selenic.com
parents:
diff changeset
     5
# This software may be used and distributed according to the terms
9117c6561b0b Add back links from file revisions to changeset revisions
mpm@selenic.com
parents:
diff changeset
     6
# of the GNU General Public License, incorporated herein by reference.
9117c6561b0b Add back links from file revisions to changeset revisions
mpm@selenic.com
parents:
diff changeset
     7
1089
142b5d5ec9cc Break apart hg.py
mpm@selenic.com
parents: 1072
diff changeset
     8
import os, re, select
1103
808a9f0e7af0 Fix sshrepo imports
mpm@selenic.com
parents: 1096
diff changeset
     9
from node import *
808a9f0e7af0 Fix sshrepo imports
mpm@selenic.com
parents: 1096
diff changeset
    10
from remoterepo import *
60
e32fdbd97839 Add hg:// protocol
mpm@selenic.com
parents: 56
diff changeset
    11
926
b765e970c9ff Add a local() method to repository classes
mpm@selenic.com
parents: 923
diff changeset
    12
class sshrepository(remoterepository):
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    13
    def __init__(self, ui, path):
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    14
        self.url = path
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    15
        self.ui = ui
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    16
1069
4337cd845a2a Allow using a ssh repository without a path.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1062
diff changeset
    17
        m = re.match(r'ssh://(([^@]+)@)?([^:/]+)(:(\d+))?(/(.*))?', path)
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    18
        if not m:
817
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    19
            raise RepoError("couldn't parse destination %s" % path)
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    20
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    21
        self.user = m.group(2)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    22
        self.host = m.group(3)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    23
        self.port = m.group(5)
1069
4337cd845a2a Allow using a ssh repository without a path.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1062
diff changeset
    24
        self.path = m.group(7) or "."
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    25
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    26
        args = self.user and ("%s@%s" % (self.user, self.host)) or self.host
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    27
        args = self.port and ("%s -p %s") % (args, self.port) or args
817
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    28
961
3e11d5038649 Add --ssh and --remotecmd to push
mpm@selenic.com
parents: 934
diff changeset
    29
        sshcmd = self.ui.config("ui", "ssh", "ssh")
3e11d5038649 Add --ssh and --remotecmd to push
mpm@selenic.com
parents: 934
diff changeset
    30
        remotecmd = self.ui.config("ui", "remotecmd", "hg")
3e11d5038649 Add --ssh and --remotecmd to push
mpm@selenic.com
parents: 934
diff changeset
    31
        cmd = "%s %s '%s -R %s serve --stdio'"
1069
4337cd845a2a Allow using a ssh repository without a path.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1062
diff changeset
    32
        cmd = cmd % (sshcmd, args, remotecmd, self.path)
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    33
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    34
        self.pipeo, self.pipei, self.pipee = os.popen3(cmd)
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    35
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    36
    def readerr(self):
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    37
        while 1:
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    38
            r,w,x = select.select([self.pipee], [], [], 0)
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    39
            if not r: break
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    40
            l = self.pipee.readline()
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    41
            if not l: break
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    42
            self.ui.status("remote: ", l)
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    43
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    44
    def __del__(self):
817
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    45
        try:
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    46
            self.pipeo.close()
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    47
            self.pipei.close()
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    48
            for l in self.pipee:
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    49
                self.ui.status("remote: ", l)
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    50
            self.pipee.close()
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    51
        except:
cf1d9a01dd92 Make ssh URL parsing more robust
mpm@selenic.com
parents: 816
diff changeset
    52
            pass
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    53
634
da5378d39269 Add a repo method to report repo device
Matt Mackall <mpm@selenic.com>
parents: 627
diff changeset
    54
    def dev(self):
da5378d39269 Add a repo method to report repo device
Matt Mackall <mpm@selenic.com>
parents: 627
diff changeset
    55
        return -1
da5378d39269 Add a repo method to report repo device
Matt Mackall <mpm@selenic.com>
parents: 627
diff changeset
    56
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    57
    def do_cmd(self, cmd, **args):
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    58
        self.ui.debug("sending %s command\n" % cmd)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    59
        self.pipeo.write("%s\n" % cmd)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    60
        for k, v in args.items():
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    61
            self.pipeo.write("%s %d\n" % (k, len(v)))
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    62
            self.pipeo.write(v)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    63
        self.pipeo.flush()
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    64
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    65
        return self.pipei
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    66
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    67
    def call(self, cmd, **args):
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    68
        r = self.do_cmd(cmd, **args)
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    69
        l = r.readline()
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    70
        self.readerr()
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    71
        try:
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    72
            l = int(l)
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    73
        except:
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    74
            raise RepoError("unexpected response '%s'" % l)
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    75
        return r.read(l)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    76
638
35f7adfefa69 Add a scheme for handling remote locking
Matt Mackall <mpm@selenic.com>
parents: 637
diff changeset
    77
    def lock(self):
35f7adfefa69 Add a scheme for handling remote locking
Matt Mackall <mpm@selenic.com>
parents: 637
diff changeset
    78
        self.call("lock")
35f7adfefa69 Add a scheme for handling remote locking
Matt Mackall <mpm@selenic.com>
parents: 637
diff changeset
    79
        return remotelock(self)
35f7adfefa69 Add a scheme for handling remote locking
Matt Mackall <mpm@selenic.com>
parents: 637
diff changeset
    80
35f7adfefa69 Add a scheme for handling remote locking
Matt Mackall <mpm@selenic.com>
parents: 637
diff changeset
    81
    def unlock(self):
35f7adfefa69 Add a scheme for handling remote locking
Matt Mackall <mpm@selenic.com>
parents: 637
diff changeset
    82
        self.call("unlock")
35f7adfefa69 Add a scheme for handling remote locking
Matt Mackall <mpm@selenic.com>
parents: 637
diff changeset
    83
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    84
    def heads(self):
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    85
        d = self.call("heads")
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    86
        try:
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    87
            return map(bin, d[:-1].split(" "))
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    88
        except:
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    89
            raise RepoError("unexpected response '%s'" % (d[:400] + "..."))
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    90
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    91
    def branches(self, nodes):
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    92
        n = " ".join(map(hex, nodes))
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    93
        d = self.call("branches", nodes=n)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    94
        try:
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    95
            br = [ tuple(map(bin, b.split(" "))) for b in d.splitlines() ]
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    96
            return br
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    97
        except:
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
    98
            raise RepoError("unexpected response '%s'" % (d[:400] + "..."))
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
    99
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   100
    def between(self, pairs):
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   101
        n = "\n".join(["-".join(map(hex, p)) for p in pairs])
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   102
        d = self.call("between", pairs=n)
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   103
        try:
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   104
            p = [ l and map(bin, l.split(" ")) or [] for l in d.splitlines() ]
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   105
            return p
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   106
        except:
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
   107
            raise RepoError("unexpected response '%s'" % (d[:400] + "..."))
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   108
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   109
    def changegroup(self, nodes):
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   110
        n = " ".join(map(hex, nodes))
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   111
        f = self.do_cmd("changegroup", roots=n)
635
85e2209d401c Protocol switch from using generators to stream-like objects.
Matt Mackall <mpm@selenic.com>
parents: 634
diff changeset
   112
        return self.pipei
624
876333a295ff Add an sshrepository class and hg serve --stdio
Matt Mackall <mpm@selenic.com>
parents: 623
diff changeset
   113
639
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   114
    def addchangegroup(self, cg):
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   115
        d = self.call("addchangegroup")
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   116
        if d:
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   117
            raise RepoError("push refused: %s", d)
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   118
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   119
        while 1:
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   120
            d = cg.read(4096)
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   121
            if not d: break
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   122
            self.pipeo.write(d)
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
   123
            self.readerr()
639
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   124
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   125
        self.pipeo.flush()
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   126
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
   127
        self.readerr()
639
31cebba881a0 Add addchangegroup to the ssh protocol
Matt Mackall <mpm@selenic.com>
parents: 638
diff changeset
   128
        l = int(self.pipei.readline())
646
342927da4f4c Show remote client output with "remote:"
Matt Mackall <mpm@selenic.com>
parents: 644
diff changeset
   129
        return self.pipei.read(l) != ""