Mercurial > hg > mercurial-crew-with-dirclash

annotate mercurial/keepalive.py @ 3570:c141d07198b9

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Inform the user about the new URL when being redirected via http. Additionally the changed code ensures that the new URL is used, even if it doesn't end with the query string.
author Thomas Arendsen Hein <thomas@intevation.de>
date Sat, 28 Oct 2006 23:05:57 +0200
parents c4325f0a9b91
children 8520a773a141
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
1 # This library is free software; you can redistribute it and/or
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
2 # modify it under the terms of the GNU Lesser General Public
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
3 # License as published by the Free Software Foundation; either
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
4 # version 2.1 of the License, or (at your option) any later version.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
5 #
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
6 # This library is distributed in the hope that it will be useful,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
7 # but WITHOUT ANY WARRANTY; without even the implied warranty of
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
8 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
9 # Lesser General Public License for more details.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
10 #
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
11 # You should have received a copy of the GNU Lesser General Public
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
12 # License along with this library; if not, write to the
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
13 # Free Software Foundation, Inc.,
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
14 # 59 Temple Place, Suite 330,
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
15 # Boston, MA 02111-1307 USA
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
16
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
17 # This file is part of urlgrabber, a high-level cross-protocol url-grabber
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
18 # Copyright 2002-2004 Michael D. Stenner, Ryan Tomayko
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
19
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
20 """An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
21
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
22 >>> import urllib2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
23 >>> from keepalive import HTTPHandler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
24 >>> keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
25 >>> opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
26 >>> urllib2.install_opener(opener)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
27 >>>
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
28 >>> fo = urllib2.urlopen('http://www.python.org')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
29
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
30 If a connection to a given host is requested, and all of the existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
31 connections are still in use, another connection will be opened. If
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
32 the handler tries to use an existing connection but it fails in some
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
33 way, it will be closed and removed from the pool.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
34
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
35 To remove the handler, simply re-run build_opener with no arguments, and
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
36 install that opener.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
37
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
38 You can explicitly close connections by using the close_connection()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
39 method of the returned file-like object (described below) or you can
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
40 use the handler methods:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
41
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
42 close_connection(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
43 close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
44 open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
45
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
46 NOTE: using the close_connection and close_all methods of the handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
47 should be done with care when using multiple threads.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
48 * there is nothing that prevents another thread from creating new
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
49 connections immediately after connections are closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
50 * no checks are done to prevent in-use connections from being closed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
51
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
52 >>> keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
53
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
54 EXTRA ATTRIBUTES AND METHODS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
55
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
56 Upon a status of 200, the object returned has a few additional
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
57 attributes and methods, which should not be used if you want to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
58 remain consistent with the normal urllib2-returned objects:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
59
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
60 close_connection() - close the connection to the host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
61 readlines() - you know, readlines()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
62 status - the return status (ie 404)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
63 reason - english translation of status (ie 'File not found')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
64
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
65 If you want the best of both worlds, use this inside an
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
66 AttributeError-catching try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
67
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
68 >>> try: status = fo.status
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
69 >>> except AttributeError: status = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
70
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
71 Unfortunately, these are ONLY there if status == 200, so it's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
72 easy to distinguish between non-200 responses. The reason is that
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
73 urllib2 tries to do clever things with error codes 301, 302, 401,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
74 and 407, and it wraps the object upon return.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
75
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
76 For python versions earlier than 2.4, you can avoid this fancy error
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
77 handling by setting the module-level global HANDLE_ERRORS to zero.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
78 You see, prior to 2.4, it's the HTTP Handler's job to determine what
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
79 to handle specially, and what to just pass up. HANDLE_ERRORS == 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
80 means "pass everything up". In python 2.4, however, this job no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
81 longer belongs to the HTTP Handler and is now done by a NEW handler,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
82 HTTPErrorProcessor. Here's the bottom line:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
83
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
84 python version < 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
85 HANDLE_ERRORS == 1 (default) pass up 200, treat the rest as
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
86 errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
87 HANDLE_ERRORS == 0 pass everything up, error processing is
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
88 left to the calling code
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
89 python version >= 2.4
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
90 HANDLE_ERRORS == 1 pass up 200, treat the rest as errors
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
91 HANDLE_ERRORS == 0 (default) pass everything up, let the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
92 other handlers (specifically,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
93 HTTPErrorProcessor) decide what to do
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
94
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
95 In practice, setting the variable either way makes little difference
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
96 in python 2.4, so for the most consistent behavior across versions,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
97 you probably just want to use the defaults, which will give you
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
98 exceptions on errors.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
99
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
100 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
101
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
102 # $Id: keepalive.py,v 1.14 2006/04/04 21:00:32 mstenner Exp $
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
103
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
104 import urllib2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
105 import httplib
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
106 import socket
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
107 import thread
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
108
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
109 DEBUG = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
110
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
111 import sys
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
112 if sys.version_info < (2, 4): HANDLE_ERRORS = 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
113 else: HANDLE_ERRORS = 0
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
114
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
115 class ConnectionManager:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
116 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
117 The connection manager must be able to:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
118 * keep track of all existing
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
119 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
120 def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
121 self._lock = thread.allocate_lock()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
122 self._hostmap = {} # map hosts to a list of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
123 self._connmap = {} # map connections to host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
124 self._readymap = {} # map connection to ready state
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
125
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
126 def add(self, host, connection, ready):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
127 self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
128 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
129 if not self._hostmap.has_key(host): self._hostmap[host] = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
130 self._hostmap[host].append(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
131 self._connmap[connection] = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
132 self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
133 finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
134 self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
135
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
136 def remove(self, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
137 self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
138 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
139 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
140 host = self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
141 except KeyError:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
142 pass
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
143 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
144 del self._connmap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
145 del self._readymap[connection]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
146 self._hostmap[host].remove(connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
147 if not self._hostmap[host]: del self._hostmap[host]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
148 finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
149 self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
150
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
151 def set_ready(self, connection, ready):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
152 try: self._readymap[connection] = ready
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
153 except KeyError: pass
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
154
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
155 def get_ready_conn(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
156 conn = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
157 self._lock.acquire()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
158 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
159 if self._hostmap.has_key(host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
160 for c in self._hostmap[host]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
161 if self._readymap[c]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
162 self._readymap[c] = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
163 conn = c
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
164 break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
165 finally:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
166 self._lock.release()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
167 return conn
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
168
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
169 def get_all(self, host=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
170 if host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
171 return list(self._hostmap.get(host, []))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
172 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
173 return dict(self._hostmap)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
174
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
175 class HTTPHandler(urllib2.HTTPHandler):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
176 def __init__(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
177 self._cm = ConnectionManager()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
178
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
179 #### Connection Management
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
180 def open_connections(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
181 """return a list of connected hosts and the number of connections
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
182 to each. [('foo.com:80', 2), ('bar.org', 1)]"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
183 return [(host, len(li)) for (host, li) in self._cm.get_all().items()]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
184
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
185 def close_connection(self, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
186 """close connection(s) to <host>
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
187 host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
188 no error occurs if there is no connection to that host."""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
189 for h in self._cm.get_all(host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
190 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
191 h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
192
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
193 def close_all(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
194 """close all open connections"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
195 for host, conns in self._cm.get_all().items():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
196 for h in conns:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
197 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
198 h.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
199
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
200 def _request_closed(self, request, host, connection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
201 """tells us that this request is now closed and the the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
202 connection is ready for another request"""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
203 self._cm.set_ready(connection, 1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
204
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
205 def _remove_connection(self, host, connection, close=0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
206 if close: connection.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
207 self._cm.remove(connection)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
208
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
209 #### Transaction Execution
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
210 def http_open(self, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
211 return self.do_open(HTTPConnection, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
212
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
213 def do_open(self, http_class, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
214 host = req.get_host()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
215 if not host:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
216 raise urllib2.URLError('no host given')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
217
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
218 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
219 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
220 while h:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
221 r = self._reuse_connection(h, req, host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
222
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
223 # if this response is non-None, then it worked and we're
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
224 # done. Break out, skipping the else block.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
225 if r: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
226
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
227 # connection is bad - possibly closed by server
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
228 # discard it and ask for the next free connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
229 h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
230 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
231 h = self._cm.get_ready_conn(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
232 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
233 # no (working) free connections were found. Create a new one.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
234 h = http_class(host)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
235 if DEBUG: DEBUG.info("creating new connection to %s (%d)",
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
236 host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
237 self._cm.add(host, h, 0)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
238 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
239 r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
240 except (socket.error, httplib.HTTPException), err:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
241 raise urllib2.URLError(err)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
242
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
243 # if not a persistent connection, don't try to reuse it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
244 if r.will_close: self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
245
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
246 if DEBUG: DEBUG.info("STATUS: %s, %s", r.status, r.reason)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
247 r._handler = self
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
248 r._host = host
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
249 r._url = req.get_full_url()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
250 r._connection = h
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
251 r.code = r.status
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
252 r.headers = r.msg
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
253 r.msg = r.reason
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
254
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
255 if r.status == 200 or not HANDLE_ERRORS:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
256 return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
257 else:
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
258 return self.parent.error('http', req, r,
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
259 r.status, r.msg, r.headers)
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
260
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
261 def _reuse_connection(self, h, req, host):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
262 """start the transaction with a re-used connection
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
263 return a response object (r) upon success or None on failure.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
264 This DOES not close or remove bad connections in cases where
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
265 it returns. However, if an unexpected exception occurs, it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
266 will close and remove the connection before re-raising.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
267 """
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
268 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
269 self._start_transaction(h, req)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
270 r = h.getresponse()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
271 # note: just because we got something back doesn't mean it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
272 # worked. We'll check the version below, too.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
273 except (socket.error, httplib.HTTPException):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
274 r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
275 except:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
276 # adding this block just in case we've missed
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
277 # something we will still raise the exception, but
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
278 # lets try and close the connection and remove it
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
279 # first. We previously got into a nasty loop
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
280 # where an exception was uncaught, and so the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
281 # connection stayed open. On the next try, the
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
282 # same exception was raised, etc. The tradeoff is
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
283 # that it's now possible this call will raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
284 # a DIFFERENT exception
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
285 if DEBUG: DEBUG.error("unexpected exception - closing " + \
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
286 "connection to %s (%d)", host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
287 self._cm.remove(h)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
288 h.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
289 raise
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
290
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
291 if r is None or r.version == 9:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
292 # httplib falls back to assuming HTTP 0.9 if it gets a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
293 # bad header back. This is most likely to happen if
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
294 # the socket has been closed by the server since we
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
295 # last used the connection.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
296 if DEBUG: DEBUG.info("failed to re-use connection to %s (%d)",
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
297 host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
298 r = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
299 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
300 if DEBUG: DEBUG.info("re-using connection to %s (%d)", host, id(h))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
301
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
302 return r
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
303
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
304 def _start_transaction(self, h, req):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
305 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
306 if req.has_data():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
307 data = req.get_data()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
308 h.putrequest('POST', req.get_selector())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
309 if not req.headers.has_key('Content-type'):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
310 h.putheader('Content-type',
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
311 'application/x-www-form-urlencoded')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
312 if not req.headers.has_key('Content-length'):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
313 h.putheader('Content-length', '%d' % len(data))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
314 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
315 h.putrequest('GET', req.get_selector())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
316 except (socket.error, httplib.HTTPException), err:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
317 raise urllib2.URLError(err)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
318
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
319 for args in self.parent.addheaders:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
320 h.putheader(*args)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
321 for k, v in req.headers.items():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
322 h.putheader(k, v)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
323 h.endheaders()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
324 if req.has_data():
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
325 h.send(data)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
326
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
327 class HTTPResponse(httplib.HTTPResponse):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
328 # we need to subclass HTTPResponse in order to
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
329 # 1) add readline() and readlines() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
330 # 2) add close_connection() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
331 # 3) add info() and geturl() methods
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
332
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
333 # in order to add readline(), read must be modified to deal with a
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
334 # buffer. example: readline must read a buffer and then spit back
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
335 # one line at a time. The only real alternative is to read one
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
336 # BYTE at a time (ick). Once something has been read, it can't be
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
337 # put back (ok, maybe it can, but that's even uglier than this),
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
338 # so if you THEN do a normal read, you must first take stuff from
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
339 # the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
340
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
341 # the read method wraps the original to accomodate buffering,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
342 # although read() never adds to the buffer.
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
343 # Both readline and readlines have been stolen with almost no
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
344 # modification from socket.py
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
345
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
346
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
347 def __init__(self, sock, debuglevel=0, strict=0, method=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
348 if method: # the httplib in python 2.3 uses the method arg
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
349 httplib.HTTPResponse.__init__(self, sock, debuglevel, method)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
350 else: # 2.2 doesn't
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
351 httplib.HTTPResponse.__init__(self, sock, debuglevel)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
352 self.fileno = sock.fileno
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
353 self.code = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
354 self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
355 self._rbufsize = 8096
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
356 self._handler = None # inserted by the handler later
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
357 self._host = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
358 self._url = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
359 self._connection = None # (same)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
360
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
361 _raw_read = httplib.HTTPResponse.read
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
362
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
363 def close(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
364 if self.fp:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
365 self.fp.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
366 self.fp = None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
367 if self._handler:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
368 self._handler._request_closed(self, self._host,
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
369 self._connection)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
370
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
371 def close_connection(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
372 self._handler._remove_connection(self._host, self._connection, close=1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
373 self.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
374
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
375 def info(self):
2444
5eb02f9ed804 Update keepalive.py to current CVS version of urlgrabber.
Alexis S. L. Carvalho <alexis@cecm.usp.br>
parents: 2435
diff changeset
376 return self.headers
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
377
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
378 def geturl(self):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
379 return self._url
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
380
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
381 def read(self, amt=None):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
382 # the _rbuf test is only in this first if for speed. It's not
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
383 # logically necessary
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
384 if self._rbuf and not amt is None:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
385 L = len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
386 if amt > L:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
387 amt -= L
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
388 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
389 s = self._rbuf[:amt]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
390 self._rbuf = self._rbuf[amt:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
391 return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
392
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
393 s = self._rbuf + self._raw_read(amt)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
394 self._rbuf = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
395 return s
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
396
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
397 def readline(self, limit=-1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
398 data = ""
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
399 i = self._rbuf.find('\n')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
400 while i < 0 and not (0 < limit <= len(self._rbuf)):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
401 new = self._raw_read(self._rbufsize)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
402 if not new: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
403 i = new.find('\n')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
404 if i >= 0: i = i + len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
405 self._rbuf = self._rbuf + new
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
406 if i < 0: i = len(self._rbuf)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
407 else: i = i+1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
408 if 0 <= limit < len(self._rbuf): i = limit
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
409 data, self._rbuf = self._rbuf[:i], self._rbuf[i:]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
410 return data
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
411
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
412 def readlines(self, sizehint = 0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
413 total = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
414 list = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
415 while 1:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
416 line = self.readline()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
417 if not line: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
418 list.append(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
419 total += len(line)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
420 if sizehint and total >= sizehint:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
421 break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
422 return list
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
423
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
424
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
425 class HTTPConnection(httplib.HTTPConnection):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
426 # use the modified response class
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
427 response_class = HTTPResponse
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
428
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
429 #########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
430 ##### TEST FUNCTIONS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
431 #########################################################################
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
432
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
433 def error_handler(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
434 global HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
435 orig = HANDLE_ERRORS
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
436 keepalive_handler = HTTPHandler()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
437 opener = urllib2.build_opener(keepalive_handler)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
438 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
439 pos = {0: 'off', 1: 'on'}
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
440 for i in (0, 1):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
441 print " fancy error handling %s (HANDLE_ERRORS = %i)" % (pos[i], i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
442 HANDLE_ERRORS = i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
443 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
444 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
445 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
446 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
447 try: status, reason = fo.status, fo.reason
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
448 except AttributeError: status, reason = None, None
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
449 except IOError, e:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
450 print " EXCEPTION: %s" % e
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
451 raise
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
452 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
453 print " status = %s, reason = %s" % (status, reason)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
454 HANDLE_ERRORS = orig
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
455 hosts = keepalive_handler.open_connections()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
456 print "open connections:", hosts
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
457 keepalive_handler.close_all()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
458
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
459 def continuity(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
460 import md5
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
461 format = '%25s: %s'
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
462
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
463 # first fetch the file with the normal http handler
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
464 opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
465 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
466 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
467 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
468 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
469 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
470 print format % ('normal urllib', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
471
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
472 # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
473 opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
474 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
475
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
476 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
477 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
478 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
479 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
480 print format % ('keepalive read', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
481
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
482 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
483 foo = ''
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
484 while 1:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
485 f = fo.readline()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
486 if f: foo = foo + f
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
487 else: break
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
488 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
489 m = md5.new(foo)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
490 print format % ('keepalive readline', m.hexdigest())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
491
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
492 def comp(N, url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
493 print ' making %i connections to:\n %s' % (N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
494
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
495 sys.stdout.write(' first using the normal urllib handlers')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
496 # first use normal opener
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
497 opener = urllib2.build_opener()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
498 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
499 t1 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
500 print ' TIME: %.3f s' % t1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
501
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
502 sys.stdout.write(' now using the keepalive handler ')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
503 # now install the keepalive handler and try again
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
504 opener = urllib2.build_opener(HTTPHandler())
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
505 urllib2.install_opener(opener)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
506 t2 = fetch(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
507 print ' TIME: %.3f s' % t2
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
508 print ' improvement factor: %.2f' % (t1/t2, )
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
509
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
510 def fetch(N, url, delay=0):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
511 import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
512 lens = []
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
513 starttime = time.time()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
514 for i in range(N):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
515 if delay and i > 0: time.sleep(delay)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
516 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
517 foo = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
518 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
519 lens.append(len(foo))
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
520 diff = time.time() - starttime
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
521
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
522 j = 0
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
523 for i in lens[1:]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
524 j = j + 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
525 if not i == lens[0]:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
526 print "WARNING: inconsistent length on read %i: %i" % (j, i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
527
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
528 return diff
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
529
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
530 def test_timeout(url):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
531 global DEBUG
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
532 dbbackup = DEBUG
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
533 class FakeLogger:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
534 def debug(self, msg, *args): print msg % args
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
535 info = warning = error = debug
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
536 DEBUG = FakeLogger()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
537 print " fetching the file to establish a connection"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
538 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
539 data1 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
540 fo.close()
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
541
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
542 i = 20
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
543 print " waiting %i seconds for the server to close the connection" % i
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
544 while i > 0:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
545 sys.stdout.write('\r %2i' % i)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
546 sys.stdout.flush()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
547 time.sleep(1)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
548 i -= 1
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
549 sys.stderr.write('\r')
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
550
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
551 print " fetching the file a second time"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
552 fo = urllib2.urlopen(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
553 data2 = fo.read()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
554 fo.close()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
555
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
556 if data1 == data2:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
557 print ' data are identical'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
558 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
559 print ' ERROR: DATA DIFFER'
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
560
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
561 DEBUG = dbbackup
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
562
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
563
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
564 def test(url, N=10):
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
565 print "checking error hander (do this on a non-200)"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
566 try: error_handler(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
567 except IOError, e:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
568 print "exiting - exception will prevent further tests"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
569 sys.exit()
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
570 print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
571 print "performing continuity test (making sure stuff isn't corrupted)"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
572 continuity(url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
573 print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
574 print "performing speed comparison"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
575 comp(N, url)
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
576 print
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
577 print "performing dropped-connection check"
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
578 test_timeout(url)
2600
c4325f0a9b91 clean up trailing white space.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents: 2444
diff changeset
579
2435
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
580 if __name__ == '__main__':
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
581 import time
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
582 import sys
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
583 try:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
584 N = int(sys.argv[1])
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
585 url = sys.argv[2]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
586 except:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
587 print "%s <integer> <url>" % sys.argv[0]
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
588 else:
ff2bac730b99 http client: support persistent connections.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
diff changeset
589 test(url, N)