mercurial/packagescan.py
author Vadim Gelfer <vadim.gelfer@gmail.com>
Thu, 15 Jun 2006 12:57:59 -0700
changeset 2435 ff2bac730b99
parent 2402 82cef38fea56
child 2497 976b6b2a1613
permissions -rw-r--r--
http client: support persistent connections. uses keepalive module from urlgrabber package. tested against "hg serve", cgi server, and through http proxy. used ethereal to verify that only one tcp connection used during entire "hg pull" sequence. if server supports keepalive, this makes latency of "hg pull" much lower.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1308
2073e5a71008 Cleanup of tabs and trailing spaces.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1283
diff changeset
     1
# packagescan.py - Helper module for identifing used modules.
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
     2
# Used for the py2exe distutil.
2323
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
     3
# This module must be the first mercurial module imported in setup.py
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
     4
#
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
     5
# Copyright 2005 Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
     6
#
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
     7
# This software may be used and distributed according to the terms
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
     8
# of the GNU General Public License, incorporated herein by reference.
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
     9
import glob
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    10
import os
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    11
import sys
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    12
import ihooks
2322
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    13
import types
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    14
import string
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    15
2323
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    16
# Install this module as fake demandload module
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    17
sys.modules['mercurial.demandload'] = sys.modules[__name__]
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    18
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    19
# Requiredmodules contains the modules imported by demandload.
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    20
# Please note that demandload can be invoked before the 
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    21
# mercurial.packagescan.scan method is invoked in case a mercurial
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    22
# module is imported.
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    23
requiredmodules = {} 
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    24
def demandload(scope, modules):
2322
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    25
    """ fake demandload function that collects the required modules 
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    26
        foo            import foo
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    27
        foo bar        import foo, bar
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    28
        foo.bar        import foo.bar
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    29
        foo:bar        from foo import bar
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    30
        foo:bar,quux   from foo import bar, quux
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    31
        foo.bar:quux   from foo.bar import quux"""
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    32
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    33
    for m in modules.split():
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    34
        mod = None
1841
7f12a63568ae Fix py2exe packagescan problem with new demandload
Eung-ju Park <eungju@gmail.com>
parents: 1308
diff changeset
    35
        try:
2322
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    36
            module, fromlist = m.split(':')
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    37
            fromlist = fromlist.split(',')
1841
7f12a63568ae Fix py2exe packagescan problem with new demandload
Eung-ju Park <eungju@gmail.com>
parents: 1308
diff changeset
    38
        except:
7f12a63568ae Fix py2exe packagescan problem with new demandload
Eung-ju Park <eungju@gmail.com>
parents: 1308
diff changeset
    39
            module = m
2322
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    40
            fromlist = []
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    41
        mod = __import__(module, scope, scope, fromlist)
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    42
        if fromlist == []:
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    43
            # mod is only the top package, but we need all packages
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    44
            comp = module.split('.')
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    45
            i = 1
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    46
            mn = comp[0]
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    47
            while True:
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    48
                # mn and mod.__name__ might not be the same
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    49
                scope[mn] = mod
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    50
                requiredmodules[mod.__name__] = 1
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    51
                if len(comp) == i: break
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    52
                mod = getattr(mod,comp[i]) 
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    53
                mn = string.join(comp[:i+1],'.')
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    54
                i += 1
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    55
        else:
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    56
            # mod is the last package in the component list
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    57
            requiredmodules[mod.__name__] = 1
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    58
            for f in fromlist:
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    59
                scope[f] = getattr(mod,f)
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    60
                if type(scope[f]) == types.ModuleType:
685597676a13 packagescan: handle demandload module naming changes.
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    61
                    requiredmodules[scope[f].__name__] = 1
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    62
2323
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    63
def scan(libpath,packagename):
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    64
    """ helper for finding all required modules of package <packagename> """
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    65
    # Use the package in the build directory
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    66
    libpath = os.path.abspath(libpath)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    67
    sys.path.insert(0,libpath)
2402
82cef38fea56 packagescan can't scan nested packages
Eung-Ju PARK <eungju@gmail.com>
parents: 2325
diff changeset
    68
    packdir = os.path.join(libpath,packagename.replace('.', '/'))
1308
2073e5a71008 Cleanup of tabs and trailing spaces.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1283
diff changeset
    69
    # A normal import would not find the package in
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    70
    # the build directory. ihook is used to force the import.
1308
2073e5a71008 Cleanup of tabs and trailing spaces.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1283
diff changeset
    71
    # After the package is imported the import scope for
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    72
    # the following imports is settled.
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    73
    p = importfrom(packdir)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    74
    globals()[packagename] = p
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    75
    sys.modules[packagename] = p
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    76
    # Fetch the python modules in the package
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    77
    cwd = os.getcwd()
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    78
    os.chdir(packdir)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    79
    pymodulefiles = glob.glob('*.py')
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    80
    extmodulefiles = glob.glob('*.pyd')
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    81
    os.chdir(cwd)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    82
    # Import all python modules and by that run the fake demandload
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    83
    for m in pymodulefiles:
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    84
        if m == '__init__.py': continue
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    85
        tmp = {}
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    86
        mname,ext = os.path.splitext(m)
1308
2073e5a71008 Cleanup of tabs and trailing spaces.
Thomas Arendsen Hein <thomas@intevation.de>
parents: 1283
diff changeset
    87
        fullname = packagename+'.'+mname
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    88
        __import__(fullname,tmp,tmp)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    89
        requiredmodules[fullname] = 1
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    90
    # Import all extension modules and by that run the fake demandload
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    91
    for m in extmodulefiles:
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    92
        tmp = {}
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    93
        mname,ext = os.path.splitext(m)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    94
        fullname = packagename+'.'+mname
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    95
        __import__(fullname,tmp,tmp)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
    96
        requiredmodules[fullname] = 1
2323
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    97
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    98
def getmodules():
c58a403aa830 setup.py: install packagescan before any mercurial modules is imported
Volker Kleinfeld <Volker.Kleinfeld@gmx.de>
parents: 1841
diff changeset
    99
    return requiredmodules.keys()
1283
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   100
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   101
def importfrom(filename):
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   102
    """
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   103
    import module/package from a named file and returns the module.
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   104
    It does not check on sys.modules or includes the module in the scope.
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   105
    """
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   106
    loader = ihooks.BasicModuleLoader()
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   107
    path, file = os.path.split(filename)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   108
    name, ext  = os.path.splitext(file)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   109
    m = loader.find_module_in_dir(name, path)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   110
    if not m:
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   111
        raise ImportError, name
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   112
    m = loader.load_module(name, m)
f5faab34f32e Support for the distutils extention 'py2exe' added.
Volker.Kleinfeld@gmx.de
parents:
diff changeset
   113
    return m