# HG changeset patch # User Bryan O'Sullivan # Date 1191621666 25200 # Node ID 5105b119edd2280df9638db6c30898fb350b8776 # Parent e73a83af79267ea9f08a8b7fe7498dbd5f4539c5 Add osutil module, containing a listdir function. This is similar to os.listdir, only it returns a sorted list of tuples. diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py --- a/mercurial/dirstate.py +++ b/mercurial/dirstate.py @@ -10,7 +10,7 @@ of the GNU General Public License, incor from node import * from i18n import _ import struct, os, time, bisect, stat, strutil, util, re, errno, ignore -import cStringIO +import cStringIO, osutil _unknown = ('?', 0, 0, 0) _format = ">cllll" @@ -389,7 +389,7 @@ class dirstate(object): common_prefix_len += 1 normpath = util.normpath - listdir = os.listdir + listdir = osutil.listdir lstat = os.lstat bisect_left = bisect.bisect_left isdir = os.path.isdir @@ -410,8 +410,7 @@ class dirstate(object): add((normpath(s[common_prefix_len:]), 'd', lstat(s))) while work: top = work.pop() - names = listdir(top) - names.sort() + entries = listdir(top, stat=True) # nd is the top of the repository dir tree nd = normpath(top[common_prefix_len:]) if nd == '.': @@ -420,19 +419,19 @@ class dirstate(object): # do not recurse into a repo contained in this # one. use bisect to find .hg directory so speed # is good on big directory. + names = [e[0] for e in entries] hg = bisect_left(names, '.hg') if hg < len(names) and names[hg] == '.hg': if isdir(join(top, '.hg')): continue - for f in names: + for f, kind, st in entries: np = pconvert(join(nd, f)) if np in known: continue known[np] = 1 p = join(top, f) # don't trip over symlinks - st = lstat(p) - if s_isdir(st.st_mode): + if kind == stat.S_IFDIR: if not ignore(np): wadd(p) if directories: diff --git a/mercurial/osutil.c b/mercurial/osutil.c new file mode 100644 --- /dev/null +++ b/mercurial/osutil.c @@ -0,0 +1,325 @@ +/* + osutil.c - native operating system services + + Copyright 2007 Matt Mackall and others + + This software may be used and distributed according to the terms of + the GNU General Public License, incorporated herein by reference. +*/ + +#define _ATFILE_SOURCE +#define _LARGEFILE64_SOURCE +#include +#include +#include +#include +#include +#include +#include + +#include "Python.h" + +struct listdir_stat { + PyObject_HEAD + struct stat st; +}; + +#define listdir_slot(name) \ + static PyObject *listdir_stat_##name(PyObject *self, void *x) \ + { \ + return PyInt_FromLong(((struct listdir_stat *) self)->st.name); \ + } + +listdir_slot(st_dev); +listdir_slot(st_mode); +listdir_slot(st_nlink); +listdir_slot(st_size); +listdir_slot(st_mtime); +listdir_slot(st_ctime); + +static struct PyGetSetDef listdir_stat_getsets[] = { + {"st_dev", listdir_stat_st_dev, 0, 0, 0}, + {"st_mode", listdir_stat_st_mode, 0, 0, 0}, + {"st_nlink", listdir_stat_st_nlink, 0, 0, 0}, + {"st_size", listdir_stat_st_size, 0, 0, 0}, + {"st_mtime", listdir_stat_st_mtime, 0, 0, 0}, + {"st_ctime", listdir_stat_st_ctime, 0, 0, 0}, + {0, 0, 0, 0, 0} +}; + +static PyObject *listdir_stat_new(PyTypeObject *t, PyObject *a, PyObject *k) +{ + return (*t->tp_alloc)(t, 0); +} + +static void listdir_stat_dealloc(PyObject *o) +{ + (*o->ob_type->tp_free)(o); +} + +static PyTypeObject listdir_stat_type = { + PyObject_HEAD_INIT(NULL) + 0, /*ob_size*/ + "osutil.stat", /*tp_name*/ + sizeof(struct listdir_stat), /*tp_basicsize*/ + 0, /*tp_itemsize*/ + (destructor)listdir_stat_dealloc, /*tp_dealloc*/ + 0, /*tp_print*/ + 0, /*tp_getattr*/ + 0, /*tp_setattr*/ + 0, /*tp_compare*/ + 0, /*tp_repr*/ + 0, /*tp_as_number*/ + 0, /*tp_as_sequence*/ + 0, /*tp_as_mapping*/ + 0, /*tp_hash */ + 0, /*tp_call*/ + 0, /*tp_str*/ + 0, /*tp_getattro*/ + 0, /*tp_setattro*/ + 0, /*tp_as_buffer*/ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ + "stat objects", /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + 0, /* tp_methods */ + 0, /* tp_members */ + listdir_stat_getsets, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + 0, /* tp_init */ + 0, /* tp_alloc */ + listdir_stat_new, /* tp_new */ +}; + +static inline int mode_to_kind(int mode) +{ + if (S_ISREG(mode)) return S_IFREG; + if (S_ISDIR(mode)) return S_IFDIR; + if (S_ISLNK(mode)) return S_IFLNK; + if (S_ISBLK(mode)) return S_IFBLK; + if (S_ISCHR(mode)) return S_IFCHR; + if (S_ISFIFO(mode)) return S_IFIFO; + if (S_ISSOCK(mode)) return S_IFSOCK; + return mode; +} + +static PyObject *listdir(PyObject *self, PyObject *args, PyObject *kwargs) +{ + DIR *dir = NULL; + struct dirent64 *ent; + PyObject *list = NULL; + PyObject *ctor_args = NULL; + int all_kinds = 1; + char *full_path; + int path_len; + int do_stat; + char *path; + int ret; + + { + static char *kwlist[] = { "path", "stat", NULL }; + PyObject *statobj = NULL; + + if (!PyArg_ParseTupleAndKeywords(args, kwargs, "s#|O:listdir", kwlist, + &path, &path_len, &statobj)) + goto bail; + + do_stat = statobj && PyObject_IsTrue(statobj); + } + + if ((dir = opendir(path)) == NULL) { + list = PyErr_SetFromErrnoWithFilename(PyExc_OSError, path); + goto bail; + } + + if ((list = PyList_New(0)) == NULL) + goto bail; + + full_path = alloca(path_len + PATH_MAX + 2); + memcpy(full_path, path, path_len); + full_path[path_len] = '/'; + + while ((ent = readdir64(dir))) { + PyObject *name = NULL; + PyObject *py_kind = NULL; + PyObject *val = NULL; + unsigned char d_type; + int kind = -1; + + if (strcmp(ent->d_name, ".") == 0 || strcmp(ent->d_name, "..") == 0) + continue; + +#ifdef DT_REG + if (do_stat) + d_type = 0; + else + d_type = ent->d_type; +#else + d_type = 0; +#endif + + switch (d_type) { +#ifdef DT_REG + case DT_REG: kind = S_IFREG; break; + case DT_DIR: kind = S_IFDIR; break; + case DT_LNK: kind = S_IFLNK; break; + case DT_BLK: kind = S_IFBLK; break; + case DT_CHR: kind = S_IFCHR; break; + case DT_FIFO: kind = S_IFIFO; break; + case DT_SOCK: kind = S_IFSOCK; break; +#endif + default: + if (all_kinds) + all_kinds = 0; + break; + } + + name = PyString_FromString(ent->d_name); + if (kind != -1) + py_kind = PyInt_FromLong(kind); + else { + py_kind = Py_None; + Py_INCREF(Py_None); + } + + val = PyTuple_New(do_stat ? 3 : 2); + + if (name == NULL || py_kind == NULL || val == NULL) { + Py_XDECREF(name); + Py_XDECREF(py_kind); + Py_XDECREF(val); + + goto bail; + } + + PyTuple_SET_ITEM(val, 0, name); + PyTuple_SET_ITEM(val, 1, py_kind); + if (do_stat) { + PyTuple_SET_ITEM(val, 2, Py_None); + Py_INCREF(Py_None); + } + + PyList_Append(list, val); + Py_DECREF(val); + } + + PyList_Sort(list); + + if (do_stat || !all_kinds) { + ssize_t size = PyList_Size(list); + ssize_t i; +#ifdef AT_SYMLINK_NOFOLLOW + int dfd = dirfd(dir); +#endif + + for (i = 0; i < size; i++) { + PyObject *elt = PyList_GetItem(list, i); + char *name = PyString_AsString(PyTuple_GET_ITEM(elt, 0)); + PyObject *py_st = NULL; + PyObject *py_kind = PyTuple_GET_ITEM(elt, 1); + int kind; + + kind = py_kind == Py_None ? -1 : PyInt_AsLong(py_kind); + + if (kind != -1 && !do_stat) + continue; + + strcpy(full_path + path_len + 1, name); + + if (do_stat) { + struct listdir_stat *st; + + if (ctor_args == NULL) { + ctor_args = PyTuple_New(0); + if (ctor_args == NULL) + goto bail; + } + + st = (struct listdir_stat *) + PyObject_CallObject((PyObject *) &listdir_stat_type, + ctor_args); + if (st == NULL) + goto bail; +#ifdef AT_SYMLINK_NOFOLLOW + ret = fstatat(dfd, name, &st->st, AT_SYMLINK_NOFOLLOW); +#else + ret = lstat(full_path, &st->st); +#endif + if (ret == -1) { + list = PyErr_SetFromErrnoWithFilename(PyExc_OSError, + full_path); + goto bail; + } + if (kind == -1) + kind = mode_to_kind(st->st.st_mode); + py_st = (PyObject *) st; + } else { + struct stat buf; +#ifdef AT_SYMLINK_NOFOLLOW + ret = fstatat(dfd, ent->d_name, &buf, AT_SYMLINK_NOFOLLOW); +#else + ret = lstat(full_path, &buf); +#endif + if (ret == -1) { + list = PyErr_SetFromErrnoWithFilename(PyExc_OSError, + full_path); + goto bail; + } + if (kind == -1) + kind = mode_to_kind(buf.st_mode); + } + + if (py_kind == Py_None && kind != -1) { + py_kind = PyInt_FromLong(kind); + if (py_kind == NULL) + goto bail; + Py_XDECREF(Py_None); + PyTuple_SET_ITEM(elt, 1, py_kind); + } + + if (do_stat) { + if (py_st == NULL) { + py_st = Py_None; + Py_INCREF(Py_None); + } + PyTuple_SET_ITEM(elt, 2, py_st); + } + } + } + + goto done; + +bail: + Py_XDECREF(list); + +done: + Py_XDECREF(ctor_args); + if (dir) + closedir(dir); + + return list; +} + +static char osutil_doc[] = "Native operating system services."; + +static PyMethodDef methods[] = { + {"listdir", (PyCFunction) listdir, METH_VARARGS | METH_KEYWORDS, + "list a directory\n"}, + {NULL, NULL} +}; + +PyMODINIT_FUNC initosutil(void) +{ + if (PyType_Ready(&listdir_stat_type) == -1) + return; + + Py_InitModule3("osutil", methods, osutil_doc); +} diff --git a/mercurial/osutil.py b/mercurial/osutil.py new file mode 100644 --- /dev/null +++ b/mercurial/osutil.py @@ -0,0 +1,37 @@ +import os, stat + +def _mode_to_kind(mode): + if stat.S_ISREG(mode): return stat.S_IFREG + if stat.S_ISDIR(mode): return stat.S_IFDIR + if stat.S_ISLNK(mode): return stat.S_IFLNK + if stat.S_ISBLK(mode): return stat.S_IFBLK + if stat.S_ISCHR(mode): return stat.S_IFCHR + if stat.S_ISFIFO(mode): return stat.S_IFIFO + if stat.S_ISSOCK(mode): return stat.S_IFSOCK + return mode + +def listdir(path, stat=False): + '''listdir(path, stat=False) -> list_of_tuples + + Return a sorted list containing information about the entries + in the directory. + + If stat is True, each element is a 3-tuple: + + (name, type, stat object) + + Otherwise, each element is a 2-tuple: + + (name, type) + ''' + result = [] + prefix = path + os.sep + names = os.listdir(path) + names.sort() + for fn in names: + st = os.lstat(prefix + fn) + if stat: + result.append((fn, _mode_to_kind(st.st_mode), st)) + else: + result.append((fn, _mode_to_kind(st.st_mode))) + return result diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py --- a/mercurial/streamclone.py +++ b/mercurial/streamclone.py @@ -6,7 +6,7 @@ # of the GNU General Public License, incorporated herein by reference. from i18n import _ -import os, stat, util, lock +import os, osutil, stat, util, lock # if server supports streaming clone, it advertises "stream" # capability with value that is version+flags of repo it is serving. @@ -19,17 +19,14 @@ def walkrepo(root): strip_count = len(root) + len(os.sep) def walk(path, recurse): - ents = os.listdir(path) - ents.sort() - for e in ents: + for e, kind, st in osutil.listdir(path, stat=True): pe = os.path.join(path, e) - st = os.lstat(pe) - if stat.S_ISDIR(st.st_mode): + if kind == stat.S_IFDIR: if recurse: for x in walk(pe, True): yield x else: - if not stat.S_ISREG(st.st_mode) or len(e) < 2: + if kind != stat.S_IFREG or len(e) < 2: continue sfx = e[-2:] if sfx in ('.d', '.i'): diff --git a/mercurial/util.py b/mercurial/util.py --- a/mercurial/util.py +++ b/mercurial/util.py @@ -14,7 +14,7 @@ platform-specific details from the core. from i18n import _ import cStringIO, errno, getpass, popen2, re, shutil, sys, tempfile, strutil -import os, stat, threading, time, calendar, ConfigParser, locale, glob +import os, stat, threading, time, calendar, ConfigParser, locale, glob, osutil try: set = set @@ -676,7 +676,7 @@ def copyfiles(src, dst, hardlink=None): if os.path.isdir(src): os.mkdir(dst) - for name in os.listdir(src): + for name, kind in osutil.listdir(src): srcname = os.path.join(src, name) dstname = os.path.join(dst, name) copyfiles(srcname, dstname, hardlink) @@ -1060,7 +1060,8 @@ else: rcs = [os.path.join(path, 'hgrc')] rcdir = os.path.join(path, 'hgrc.d') try: - rcs.extend([os.path.join(rcdir, f) for f in os.listdir(rcdir) + rcs.extend([os.path.join(rcdir, f) + for f, kind in osutil.listdir(rcdir) if f.endswith(".rc")]) except OSError: pass @@ -1653,7 +1654,7 @@ def rcpath(): for p in os.environ['HGRCPATH'].split(os.pathsep): if not p: continue if os.path.isdir(p): - for f in os.listdir(p): + for f, kind in osutil.listdir(p): if f.endswith('.rc'): _rcpath.append(os.path.join(p, f)) else: diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -52,6 +52,19 @@ class install_package_data(install_data) mercurial.version.remember_version(version) cmdclass = {'install_data': install_package_data} +ext_modules=[ + Extension('mercurial.mpatch', ['mercurial/mpatch.c']), + Extension('mercurial.bdiff', ['mercurial/bdiff.c']), + Extension('mercurial.base85', ['mercurial/base85.c']), + Extension('mercurial.diffhelpers', ['mercurial/diffhelpers.c']) + ] + +try: + import posix + ext_modules.append(Extension('mercurial.osutil', ['mercurial/osutil.c'])) +except ImportError: + pass + setup(name='mercurial', version=mercurial.version.get_version(), author='Matt Mackall', @@ -60,10 +73,7 @@ setup(name='mercurial', description='Scalable distributed SCM', license='GNU GPL', packages=['mercurial', 'mercurial.hgweb', 'hgext', 'hgext.convert'], - ext_modules=[Extension('mercurial.mpatch', ['mercurial/mpatch.c']), - Extension('mercurial.bdiff', ['mercurial/bdiff.c']), - Extension('mercurial.base85', ['mercurial/base85.c']), - Extension('mercurial.diffhelpers', ['mercurial/diffhelpers.c'])], + ext_modules=ext_modules, data_files=[(os.path.join('mercurial', root), [os.path.join(root, file_) for file_ in files]) for root, dirs, files in os.walk('templates')],