[Python-modules-commits] [python-dugong] 03/11: Import python-dugong_3.7+dfsg.orig.tar.bz2
Nikolaus Rath
nikratio-guest at moszumanska.debian.org
Sat Oct 8 03:42:21 UTC 2016
This is an automated email from the git hooks/post-receive script.
nikratio-guest pushed a commit to branch master
in repository python-dugong.
commit 4d86a28f722ed797e6780a871eeadc5b80c5686f
Author: Nikolaus Rath <Nikolaus at rath.org>
Date: Mon Jun 20 10:58:36 2016 -0700
Import python-dugong_3.7+dfsg.orig.tar.bz2
---
Changes.rst | 21 +
PKG-INFO | 40 +-
README.rst | 37 +-
dugong.egg-info/PKG-INFO | 40 +-
dugong.egg-info/SOURCES.txt | 2 +-
dugong/__init__.py | 382 +++++++++------
examples/extract_links.py | 7 +-
examples/httpcat.py | 6 +-
examples/pipeline1.py | 6 +-
setup.py | 8 +
test/conftest.py | 74 +--
test/pytest_checklogs.py | 130 ++++++
test/test_aio.py | 2 +-
test/test_dugong.py | 119 ++++-
test/test_dugong.py.orig | 1077 -------------------------------------------
test/test_examples.py | 2 +-
16 files changed, 652 insertions(+), 1301 deletions(-)
diff --git a/Changes.rst b/Changes.rst
index 7e5c70c..7005d90 100644
--- a/Changes.rst
+++ b/Changes.rst
@@ -1,5 +1,26 @@
.. currentmodule:: dugong
+Relase 3.7 (2016-06-20)
+=======================
+
+* Dugong now supports server responses that specify just ``Connection:
+ close`` instead of providing the response length or using chunked
+ encoding.
+
+* Dugong now honors the `~ssl.SSLContext.check_hostname` attribute of
+ `~ssl.SSLContext` objects.
+
+Release 3.6 (2016-04-23)
+========================
+
+* Dugong now uses semantic versioning. This means that
+ backwards-incompatible versions (i.e., versions that change the
+ existing API in some way) will be reflected in an increase of the
+ major version number, i.e. the next backwards-incompatible version
+ will have version 4.0.
+
+* Various minor bugfixes.
+
Release 3.5 (2015-01-31)
========================
diff --git a/PKG-INFO b/PKG-INFO
index c7859fc..dedc06a 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: dugong
-Version: 3.5
+Version: 3.7
Summary: A HTTP 1.1 client module supporting asynchronous IO, pipelining and `Expect: 100-continue`. Designed for RESTful protocols.
Home-page: https://bitbucket.org/nikratio/python-dugong
Author: Nikolaus Rath
@@ -48,6 +48,8 @@ Description: ==========================
Dugong requires Python 3.3 or newer.
.. _`bytes-like objects`: http://docs.python.org/3/glossary.html#term-bytes-like-object
+ .. _asyncio: http://docs.python.org/3.4/library/asyncio.html
+
Installation
============
@@ -56,12 +58,15 @@ Description: ==========================
# python3 setup.py install [--user]
- To run the self-tests, install `py.test`_ and run ::
-
- # py.test-3 test/
+ To run the self-tests, install `py.test`_ with the `pytest-catchlog`_
+ plugin and run ::
+ # python3 -m pytest test/
.. _PyPi: https://pypi.python.org/pypi/dugong/#downloads
+ .. _py.test: http://www.pytest.org/
+ .. _pytest-catchlog: https://github.com/eisensheng/pytest-catchlog
+
Getting Help
============
@@ -69,25 +74,38 @@ Description: ==========================
The documentation can be `read online`__ and is also included in the
*doc/html* directory of the dugong tarball.
- Please report any bugs on the `issue tracker`_. For discussion and
+ Please report any bugs on the `BitBucket issue tracker`_. For discussion and
questions, please subscribe to the `dugong mailing list`_.
+ .. __: http://pythonhosted.org/dugong/
+ .. _dugong mailing list: https://groups.google.com/d/forum/python-dugong
+ .. _`BitBucket issue tracker`: https://bitbucket.org/nikratio/python-dugong/issues
+
+
Development Status
==================
The Dugong API is not yet stable and may change from one release to
- the other.
+ the other. Starting with version 3.5, Dugong uses semantic
+ versioning. This means changes in the API will be reflected in an
+ increase of the major version number, i.e. the next
+ backwards-incompatible version will be 4.0. Projects designed for
+ e.g. version 3.5 of Dugong are thus recommended to declare a
+ dependency on ``dugong >= 3.5, < 4.0``.
- .. __: http://pythonhosted.org/dugong/
- .. _dugong mailing list: https://groups.google.com/d/forum/python-dugong
- .. _issue tracker: https://bitbucket.org/nikratio/python-dugong/issues
- .. _py.test: http://www.pytest.org/
- .. _asyncio: http://docs.python.org/3.4/library/asyncio.html
+ Contributing
+ ============
+
+ The LLFUSE source code is available both on GitHub_ and BitBucket_.
+
+ .. _BitBucket: https://bitbucket.org/nikratio/python-dugong/
+ .. _GitHub: https://github.com/python-dugong/main
Keywords: http
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
+Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Python Software Foundation License
Classifier: Topic :: Internet :: WWW/HTTP
diff --git a/README.rst b/README.rst
index fb8cb84..8769189 100644
--- a/README.rst
+++ b/README.rst
@@ -40,6 +40,8 @@ objects`_ or binary streams.
Dugong requires Python 3.3 or newer.
.. _`bytes-like objects`: http://docs.python.org/3/glossary.html#term-bytes-like-object
+.. _asyncio: http://docs.python.org/3.4/library/asyncio.html
+
Installation
============
@@ -48,12 +50,15 @@ As usual: download the tarball from PyPi_, extract it, and run ::
# python3 setup.py install [--user]
-To run the self-tests, install `py.test`_ and run ::
-
- # py.test-3 test/
+To run the self-tests, install `py.test`_ with the `pytest-catchlog`_
+plugin and run ::
+ # python3 -m pytest test/
.. _PyPi: https://pypi.python.org/pypi/dugong/#downloads
+.. _py.test: http://www.pytest.org/
+.. _pytest-catchlog: https://github.com/eisensheng/pytest-catchlog
+
Getting Help
============
@@ -61,18 +66,30 @@ Getting Help
The documentation can be `read online`__ and is also included in the
*doc/html* directory of the dugong tarball.
-Please report any bugs on the `issue tracker`_. For discussion and
+Please report any bugs on the `BitBucket issue tracker`_. For discussion and
questions, please subscribe to the `dugong mailing list`_.
+.. __: http://pythonhosted.org/dugong/
+.. _dugong mailing list: https://groups.google.com/d/forum/python-dugong
+.. _`BitBucket issue tracker`: https://bitbucket.org/nikratio/python-dugong/issues
+
+
Development Status
==================
The Dugong API is not yet stable and may change from one release to
-the other.
+the other. Starting with version 3.5, Dugong uses semantic
+versioning. This means changes in the API will be reflected in an
+increase of the major version number, i.e. the next
+backwards-incompatible version will be 4.0. Projects designed for
+e.g. version 3.5 of Dugong are thus recommended to declare a
+dependency on ``dugong >= 3.5, < 4.0``.
-.. __: http://pythonhosted.org/dugong/
-.. _dugong mailing list: https://groups.google.com/d/forum/python-dugong
-.. _issue tracker: https://bitbucket.org/nikratio/python-dugong/issues
-.. _py.test: http://www.pytest.org/
-.. _asyncio: http://docs.python.org/3.4/library/asyncio.html
+Contributing
+============
+
+The LLFUSE source code is available both on GitHub_ and BitBucket_.
+
+.. _BitBucket: https://bitbucket.org/nikratio/python-dugong/
+.. _GitHub: https://github.com/python-dugong/main
diff --git a/dugong.egg-info/PKG-INFO b/dugong.egg-info/PKG-INFO
index c7859fc..dedc06a 100644
--- a/dugong.egg-info/PKG-INFO
+++ b/dugong.egg-info/PKG-INFO
@@ -1,6 +1,6 @@
Metadata-Version: 1.1
Name: dugong
-Version: 3.5
+Version: 3.7
Summary: A HTTP 1.1 client module supporting asynchronous IO, pipelining and `Expect: 100-continue`. Designed for RESTful protocols.
Home-page: https://bitbucket.org/nikratio/python-dugong
Author: Nikolaus Rath
@@ -48,6 +48,8 @@ Description: ==========================
Dugong requires Python 3.3 or newer.
.. _`bytes-like objects`: http://docs.python.org/3/glossary.html#term-bytes-like-object
+ .. _asyncio: http://docs.python.org/3.4/library/asyncio.html
+
Installation
============
@@ -56,12 +58,15 @@ Description: ==========================
# python3 setup.py install [--user]
- To run the self-tests, install `py.test`_ and run ::
-
- # py.test-3 test/
+ To run the self-tests, install `py.test`_ with the `pytest-catchlog`_
+ plugin and run ::
+ # python3 -m pytest test/
.. _PyPi: https://pypi.python.org/pypi/dugong/#downloads
+ .. _py.test: http://www.pytest.org/
+ .. _pytest-catchlog: https://github.com/eisensheng/pytest-catchlog
+
Getting Help
============
@@ -69,25 +74,38 @@ Description: ==========================
The documentation can be `read online`__ and is also included in the
*doc/html* directory of the dugong tarball.
- Please report any bugs on the `issue tracker`_. For discussion and
+ Please report any bugs on the `BitBucket issue tracker`_. For discussion and
questions, please subscribe to the `dugong mailing list`_.
+ .. __: http://pythonhosted.org/dugong/
+ .. _dugong mailing list: https://groups.google.com/d/forum/python-dugong
+ .. _`BitBucket issue tracker`: https://bitbucket.org/nikratio/python-dugong/issues
+
+
Development Status
==================
The Dugong API is not yet stable and may change from one release to
- the other.
+ the other. Starting with version 3.5, Dugong uses semantic
+ versioning. This means changes in the API will be reflected in an
+ increase of the major version number, i.e. the next
+ backwards-incompatible version will be 4.0. Projects designed for
+ e.g. version 3.5 of Dugong are thus recommended to declare a
+ dependency on ``dugong >= 3.5, < 4.0``.
- .. __: http://pythonhosted.org/dugong/
- .. _dugong mailing list: https://groups.google.com/d/forum/python-dugong
- .. _issue tracker: https://bitbucket.org/nikratio/python-dugong/issues
- .. _py.test: http://www.pytest.org/
- .. _asyncio: http://docs.python.org/3.4/library/asyncio.html
+ Contributing
+ ============
+
+ The LLFUSE source code is available both on GitHub_ and BitBucket_.
+
+ .. _BitBucket: https://bitbucket.org/nikratio/python-dugong/
+ .. _GitHub: https://github.com/python-dugong/main
Keywords: http
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
+Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Python Software Foundation License
Classifier: Topic :: Internet :: WWW/HTTP
diff --git a/dugong.egg-info/SOURCES.txt b/dugong.egg-info/SOURCES.txt
index ea967dc..b85a6a4 100644
--- a/dugong.egg-info/SOURCES.txt
+++ b/dugong.egg-info/SOURCES.txt
@@ -70,9 +70,9 @@ test/ca.crt
test/ca.key
test/conftest.py
test/pytest.ini
+test/pytest_checklogs.py
test/server.crt
test/server.key
test/test_aio.py
test/test_dugong.py
-test/test_dugong.py.orig
test/test_examples.py
\ No newline at end of file
diff --git a/dugong/__init__.py b/dugong/__init__.py
index 90ec18f..92e0a52 100644
--- a/dugong/__init__.py
+++ b/dugong/__init__.py
@@ -1,7 +1,7 @@
'''
dugong.py - Python HTTP Client Module
-Copyright (C) Nikolaus Rath <Nikolaus at rath.org>
+Copyright © 2014 Nikolaus Rath <Nikolaus.org>
This module may be distributed under the terms of the Python Software Foundation
License Version 2.
@@ -33,8 +33,13 @@ try:
except ImportError:
asyncio = None
+# Enums are only available in 3.4 and newer
+try:
+ from enum import Enum
+except ImportError:
+ Enum = object
-__version__ = '3.5'
+__version__ = '3.7'
log = logging.getLogger(__name__)
@@ -50,12 +55,40 @@ MAX_LINE_SIZE = BUFFER_SIZE-1
#: this value, `InvalidResponse` will be raised.
MAX_HEADER_SIZE = BUFFER_SIZE-1
-CHUNKED_ENCODING = 'chunked_encoding'
-IDENTITY_ENCODING = 'identity_encoding'
+class Symbol:
+ '''
+ A symbol instance represents a specific state. Its value is
+ not relevant, as it should only ever be assigned to or compared
+ with other variables.
+ '''
+ __slots__ = [ 'name' ]
+
+ def __init__(self, name):
+ self.name = name
+
+ def __str__(self):
+ return self.name
+
+ def __repr__(self):
+ return '<%s>' % (self.name,)
+
+class Encodings(Enum):
+ CHUNKED = 1
+ IDENTITY = 2
+
+#: Sentinel for `HTTPConnection._out_remaining` to indicate that
+#: we're waiting for a 100-continue response from the server
+WAITING_FOR_100c = Symbol('WAITING_FOR_100c')
-#: Marker object for request body size when we're waiting
-#: for a 100-continue response from the server
-WAITING_FOR_100c = object()
+#: Sentinel for `HTTPConnection._in_remaining` to indicate that
+#: the response body cannot be read and that an exception
+#: (stored in the `HTTPConnection._encoding` attribute) should
+#: be raised.
+RESPONSE_BODY_ERROR = Symbol('RESPONSE_BODY_ERROR')
+
+#: Sentinel for `HTTPConnection._in_remaining` to indicate that
+#: we should read until EOF (i.e., no keep-alive)
+READ_UNTIL_EOF = Symbol('READ_UNTIL_EOF')
#: Sequence of ``(hostname, port)`` tuples that are used by
#: dugong to distinguish between permanent and temporary name
@@ -144,7 +177,13 @@ class HTTPResponse:
#: HTTP Response headers, a `email.message.Message` instance
self.headers = headers
- #: Length of the response body or `None`, if not known
+ #: Length of the response body or `None` if not known. This attribute
+ #: contains the actual length of the *transmitted* response. That means
+ #: that for responses where RFC 2616 mandates that no request body
+ #: be sent (e.g. in response to HEAD requests or for 1xx response
+ #: codes) this value is zero. In these cases, the length of the body that
+ #: *would* have been send can be extracted from the ``Content-Length``
+ #: response header.
self.length = length
@@ -260,7 +299,18 @@ class UnsupportedResponse(_GeneralError):
class ConnectionClosed(_GeneralError):
'''
- Raised if the server unexpectedly closed the connection.
+ Raised if the connection was unexpectedly closed.
+
+ This exception is raised also if the server declared that it will close the
+ connection (by sending a ``Connection: close`` header). Such responses can
+ still be read completely, but the next attempt to send a request or read a
+ response will raise the exception. To re-use the connection after the server
+ has closed the connection, call `HTTPConnection.reset` before further
+ requests are send.
+
+ This behavior is intentional, because the caller may have already issued
+ other requests (i.e., used pipelining). By raising an exception, the caller
+ is notified that any pending requests have been lost and need to be resend.
'''
msg = 'connection closed unexpectedly'
@@ -390,11 +440,13 @@ class HTTPConnection:
#: request headers have been sent, but request body data is still
#: pending, it is set to a ``(method, path, body_len)`` tuple. *body_len*
#: is the number of bytes that that still need to send, or
- #: WAITING_FOR_100c if we are waiting for a 100 response from the server.
+ #: `WAITING_FOR_100c` if we are waiting for a 100 response from the server.
self._out_remaining = None
#: Number of remaining bytes of the current response body (or current
- #: chunk), or `None` if the response header has not yet been read.
+ #: chunk), `None` if there is no active response or `READ_UNTIL_EOF` if
+ #: we have to read until the connection is closed (i.e., we don't know
+ #: the content-length and keep-alive is not active).
self._in_remaining = None
#: Transfer encoding of the active response (if any).
@@ -440,14 +492,24 @@ class HTTPConnection:
if self.ssl_context:
log.debug('establishing ssl layer')
- server_hostname = self.hostname if ssl.HAS_SNI else None
+ if (sys.version_info >= (3, 5) or
+ (sys.version_info >= (3, 4) and ssl.HAS_SNI)):
+ # Automatic hostname verification was added in 3.4, but only
+ # if SNI is available. In 3.5 the hostname can be checked even
+ # if there is no SNI support.
+ server_hostname = self.hostname
+ else:
+ server_hostname = None
self._sock = self.ssl_context.wrap_socket(self._sock, server_hostname=server_hostname)
- try:
- ssl.match_hostname(self._sock.getpeercert(), self.hostname)
- except:
- self.close()
- raise
+ if server_hostname is None:
+ # Manually check hostname for Python < 3.4, or if we have
+ # 3.4 without SNI.
+ try:
+ ssl.match_hostname(self._sock.getpeercert(), self.hostname)
+ except:
+ self.close()
+ raise
self._sock.setblocking(False)
self._rbuf.clear()
@@ -612,10 +674,10 @@ class HTTPConnection:
if not isinstance(buf, memoryview):
buf = memoryview(buf)
-
- fd = self._sock.fileno()
while True:
try:
+ if self._sock is None:
+ raise ConnectionClosed('connection has been closed locally')
len_ = self._sock.send(buf)
# An SSL socket has the nasty habit of returning zero
# instead of raising an exception when in non-blocking
@@ -624,12 +686,10 @@ class HTTPConnection:
raise BlockingIOError()
except (socket.timeout, ssl.SSLWantWriteError, BlockingIOError):
log.debug('yielding')
- yield PollNeeded(fd, POLLOUT)
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
+ yield PollNeeded(self._sock.fileno(), POLLOUT)
continue
except (BrokenPipeError, ConnectionResetError):
- raise ConnectionClosed('found closed when trying to write')
+ raise ConnectionClosed('connection was interrupted')
except OSError as exc:
if exc.errno == errno.EINVAL:
# Blackhole routing, according to ip(7)
@@ -664,9 +724,6 @@ class HTTPConnection:
if not self._out_remaining:
raise StateError('No active request with pending body data')
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
-
(method, path, remaining) = self._out_remaining
if remaining is WAITING_FOR_100c:
raise StateError("can't write when waiting for 100-continue")
@@ -717,9 +774,6 @@ class HTTPConnection:
log.debug('start')
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
-
if len(self._pending_requests) == 0:
raise StateError('No pending requests')
@@ -772,57 +826,78 @@ class HTTPConnection:
#
# Prepare to read body
#
- body_length = None
+ body_length = self._setup_read(method, status, header)
- tc = header['Transfer-Encoding']
- if tc:
- tc = tc.lower()
- if tc and tc == 'chunked':
- log.debug('Chunked encoding detected')
- self._encoding = CHUNKED_ENCODING
- self._in_remaining = 0
- elif tc and tc != 'identity':
- # Server must not sent anything other than identity or chunked, so
- # we raise InvalidResponse rather than UnsupportedResponse. We defer
- # raising the exception to read(), so that we can still return the
- # headers and status (and don't fail if the response body is empty).
- log.warning('Server uses invalid response encoding "%s"', tc)
- self._encoding = InvalidResponse('Cannot handle %s encoding' % tc)
- else:
- log.debug('identity encoding detected')
- self._encoding = IDENTITY_ENCODING
+ # Don't require calls to co_read() et al if there is
+ # nothing to be read.
+ if self._in_remaining is None:
+ self._pending_requests.popleft()
+
+ log.debug('done (in_remaining=%s)', self._in_remaining)
+ return HTTPResponse(method, path, status, reason, header, body_length)
+
+ def _setup_read(self, method, status, header):
+ '''Prepare for reading response body
+
+ Sets up `._encoding`, `_in_remaining` and returns Content-Length
+ (if available).
+
+ See RFC 2616, sec. 4.4 for specific rules.
+ '''
+
+ # On error, the exception is stored in _encoding and raised on
+ # the next call to co_read() et al - that way we can still
+ # return the http status and headers.
+
+ will_close = header.get('Connection', 'keep-alive').lower() == 'close'
+
+ body_length = header['Content-Length']
+ if body_length is not None:
+ try:
+ body_length = int(body_length)
+ except ValueError:
+ self._encoding = InvalidResponse('Invalid content-length: %s'
+ % body_length)
+ self._in_remaining = RESPONSE_BODY_ERROR
+ return None
- # does the body have a fixed length? (of zero)
if (status == NO_CONTENT or status == NOT_MODIFIED or
100 <= status < 200 or method == 'HEAD'):
log.debug('no content by RFC')
- body_length = 0
self._in_remaining = None
- self._pending_requests.popleft()
-
- # Chunked doesn't require content-length
- elif self._encoding is CHUNKED_ENCODING:
- pass
+ self._encoding = None
+ return 0
- # Otherwise we require a content-length. We defer raising the exception
- # to read(), so that we can still return the headers and status.
- elif ('Content-Length' not in header
- and not isinstance(self._encoding, InvalidResponse)):
- log.debug('no content length and no chunkend encoding, will raise on read')
- self._encoding = UnsupportedResponse('No content-length and no chunked encoding')
+ tc = header.get('Transfer-Encoding', 'identity').lower()
+ if tc == 'chunked':
+ log.debug('Chunked encoding detected')
+ self._encoding = Encodings.CHUNKED
self._in_remaining = 0
+ return None
- else:
- body_length = int(header['Content-Length'])
- if body_length:
- self._in_remaining = body_length
- else:
- self._in_remaining = None
- self._pending_requests.popleft()
+ elif tc != 'identity':
+ log.warning('Server uses invalid response encoding "%s"', tc)
+ self._encoding = InvalidResponse('Cannot handle %s encoding' % tc)
+ self._in_remaining = RESPONSE_BODY_ERROR
+ return None
- log.debug('done (in_remaining=%s)', self._in_remaining)
+ log.debug('identity encoding detected')
+ self._encoding = Encodings.IDENTITY
- return HTTPResponse(method, path, status, reason, header, body_length)
+ if body_length is not None:
+ log.debug('Will read response body of %d bytes', body_length)
+ self._in_remaining = body_length or None
+ return body_length
+
+ if will_close:
+ log.debug('no content-length, will read until EOF')
+ self._in_remaining = READ_UNTIL_EOF
+ return None
+
+ log.debug('no content length and no chunked encoding, will raise on read')
+ self._encoding = UnsupportedResponse('No content-length and no chunked encoding')
+ self._in_remaining = RESPONSE_BODY_ERROR
+ return None
def _co_read_status(self):
'''Read response line'''
@@ -906,21 +981,16 @@ class HTTPConnection:
log.debug('start (len=%d)', len_)
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
-
- if len_ is None:
+ if self._in_remaining is RESPONSE_BODY_ERROR:
+ raise self._encoding
+ elif len_ is None:
return (yield from self.co_readall())
-
- if len_ == 0 or self._in_remaining is None:
+ elif len_ == 0 or self._in_remaining is None:
return b''
-
- if self._encoding is IDENTITY_ENCODING:
+ elif self._encoding is Encodings.IDENTITY:
return (yield from self._co_read_id(len_))
- elif self._encoding is CHUNKED_ENCODING:
+ elif self._encoding is Encodings.CHUNKED:
return (yield from self._co_read_chunked(len_=len_))
- elif isinstance(self._encoding, Exception):
- raise self._encoding
else:
raise RuntimeError('ooops, this should not be possible')
@@ -978,18 +1048,14 @@ class HTTPConnection:
log.debug('start (buflen=%d)', len(buf))
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
-
- if len(buf) == 0 or self._in_remaining is None:
+ if self._in_remaining is RESPONSE_BODY_ERROR:
+ raise self._encoding
+ elif len(buf) == 0 or self._in_remaining is None:
return 0
-
- if self._encoding is IDENTITY_ENCODING:
+ elif self._encoding is Encodings.IDENTITY:
return (yield from self._co_readinto_id(buf))
- elif self._encoding is CHUNKED_ENCODING:
+ elif self._encoding is Encodings.CHUNKED:
return (yield from self._co_read_chunked(buf=buf))
- elif isinstance(self._encoding, Exception):
- raise self._encoding
else:
raise RuntimeError('ooops, this should not be possible')
@@ -1005,9 +1071,9 @@ class HTTPConnection:
self._pending_requests.popleft()
return b''
- sock_fd = self._sock.fileno()
rbuf = self._rbuf
- len_ = min(len_, self._in_remaining)
+ if self._in_remaining is not READ_UNTIL_EOF:
+ len_ = min(len_, self._in_remaining)
log.debug('updated len_=%d', len_)
# If buffer is empty, reset so that we start filling from
@@ -1022,17 +1088,24 @@ class HTTPConnection:
# and buffer is not full
while len(rbuf) < len_ and rbuf.e < len(rbuf.d):
got_data = self._try_fill_buffer()
- if not got_data and not rbuf:
- log.debug('buffer empty and nothing to read, yielding..')
- yield PollNeeded(sock_fd, POLLIN)
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
- elif not got_data:
- log.debug('nothing more to read')
- break
+ if got_data is None:
+ if rbuf:
+ log.debug('nothing more to read')
+ break
+ else:
+ log.debug('buffer empty and nothing to read, yielding..')
+ yield PollNeeded(self._sock.fileno(), POLLIN)
+ elif got_data == 0:
+ if self._in_remaining is READ_UNTIL_EOF:
+ log.debug('connection closed, %d bytes in buffer', len(rbuf))
+ self._in_remaining = len(rbuf)
+ break
+ else:
+ raise ConnectionClosed('server closed connection')
len_ = min(len_, len(rbuf))
- self._in_remaining -= len_
+ if self._in_remaining is not READ_UNTIL_EOF:
+ self._in_remaining -= len_
if len_ < len(rbuf):
buf = rbuf.d[rbuf.b:rbuf.b+len_]
@@ -1040,6 +1113,16 @@ class HTTPConnection:
else:
buf = rbuf.exhaust()
+ # When reading until EOF, it is possible that we read only the EOF. In
+ # this case we won't get called again, so we need to clean-up the
+ # request. This can not happen when we know the number of remaining
+ # bytes, because in this case either the check at the start of the
+ # function hits, or we can't read the remaining data and raise.
+ if len(buf) == 0:
+ assert self._in_remaining == 0
+ self._in_remaining = None
+ self._pending_requests.popleft()
+
log.debug('done (%d bytes)', len(buf))
return buf
@@ -1055,12 +1138,14 @@ class HTTPConnection:
self._pending_requests.popleft()
return 0
- sock_fd = self._sock.fileno()
rbuf = self._rbuf
if not isinstance(buf, memoryview):
buf = memoryview(buf)
- len_ = min(len(buf), self._in_remaining)
- log.debug('updated len_=%d', len_)
+ if self._in_remaining is READ_UNTIL_EOF:
+ len_ = len(buf)
+ else:
+ len_ = min(len(buf), self._in_remaining)
+ log.debug('set len_=%d', len_)
# First use read buffer contents
pos = min(len(rbuf), len_)
@@ -1068,11 +1153,12 @@ class HTTPConnection:
log.debug('using buffered data')
buf[:pos] = rbuf.d[rbuf.b:rbuf.b+pos]
rbuf.b += pos
- self._in_remaining -= pos
+ if self._in_remaining is not READ_UNTIL_EOF:
+ self._in_remaining -= pos
# If we've read enough, return immediately
if pos == len_:
- log.debug('done (got all we need, %d bytes)', pos)
+ log.debug('done (buffer filled completely)')
return pos
# Otherwise, prepare to read more from socket
@@ -1081,27 +1167,35 @@ class HTTPConnection:
while True:
log.debug('trying to read from socket')
+ if self._sock is None:
+ raise ConnectionClosed('connection has been closed locally')
try:
read = self._sock.recv_into(buf[pos:len_])
+ except (ConnectionResetError, BrokenPipeError):
+ raise ConnectionClosed('connection was interrupted')
except (socket.timeout, ssl.SSLWantReadError, BlockingIOError):
if pos:
- log.debug('done (nothing more to read, got %d bytes)', pos)
+ log.debug('done, no additional data available')
return pos
else:
log.debug('no data yet and nothing to read, yielding..')
- yield PollNeeded(sock_fd, POLLIN)
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
+ yield PollNeeded(self._sock.fileno(), POLLIN)
continue
if not read:
- raise ConnectionClosed('connection closed unexpectedly')
+ if self._in_remaining is READ_UNTIL_EOF:
+ log.debug('reached EOF')
+ self._in_remaining = 0
+ return pos
+ else:
+ raise ConnectionClosed('server closed connection')
+ log.debug('got %d bytes from socket', read)
- log.debug('got %d bytes', read)
- self._in_remaining -= read
+ if self._in_remaining is not READ_UNTIL_EOF:
+ self._in_remaining -= read
pos += read
if pos == len_:
- log.debug('done (got all we need, %d bytes)', pos)
+ log.debug('done (buffer filled completely)')
return pos
def _co_read_chunked(self, len_=None, buf=None):
@@ -1111,14 +1205,10 @@ class HTTPConnection:
a `bytes-like object`. If *buf* is not `None`, reads data into *buf*.
'''
- # TODO: In readinto mode, we always need an extra sock.recv()
- # to get the chunk trailer.. is there some way to avoid that? And
- # maybe also put the beginning of the next chunk into the read buffer right away?
-
log.debug('start (%s mode)', 'readinto' if buf else 'read')
assert (len_ is None) != (buf is None)
assert bool(len_) or bool(buf)
- assert self._in_remaining is not None
+ assert isinstance(self._in_remaining, int)
if self._in_remaining == 0:
log.debug('starting next chunk')
@@ -1167,7 +1257,6 @@ class HTTPConnection:
log.debug('reading until %s', substr)
- sock_fd = self._sock.fileno()
rbuf = self._rbuf
sub_len = len(substr)
@@ -1203,11 +1292,15 @@ class HTTPConnection:
maxsize -= len(buf)
# Refill buffer
- while not self._try_fill_buffer():
- log.debug('need more data, yielding')
- yield PollNeeded(sock_fd, POLLIN)
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
+ while True:
+ res = self._try_fill_buffer()
+ if res is None:
+ log.debug('need more data, yielding')
+ yield PollNeeded(self._sock.fileno(), POLLIN)
+ elif res == 0:
+ raise ConnectionClosed('server closed connection')
+ else:
+ break
log.debug('found substr at %d', idx)
idx += len(substr)
@@ -1226,8 +1319,11 @@ class HTTPConnection:
def _try_fill_buffer(self):
'''Try to fill up read buffer
- Returns the number of bytes read into buffer, or `None` if no
- data was available on the socket. May raise `ConnectionClosed`.
+ Returns the number of bytes read into buffer, or `None` if no data was
+ available on the socket. Return zero if the TCP connection has been
+ properly closed but the socket object still exists. On other problems
+ (e.g. if the socket object has been destroyed or the connection
+ interrupted), raise `ConnectionClosed`.
'''
log.debug('start')
@@ -1238,9 +1334,11 @@ class HTTPConnection:
rbuf.b = 0
rbuf.e = 0
- # If no capacity, return
- if rbuf.e == len(rbuf.d):
- return 0
+ # There should be free capacity
+ assert rbuf.e < len(rbuf.d)
+
+ if self._sock is None:
+ raise ConnectionClosed('connection has been closed locally')
try:
len_ = self._sock.recv_into(memoryview(rbuf.d)[rbuf.e:])
@@ -1248,10 +1346,7 @@ class HTTPConnection:
log.debug('done (nothing ready)')
return None
except (ConnectionResetError, BrokenPipeError):
- len_ = 0
-
- if not len_:
- raise ConnectionClosed('connection closed unexpectedly')
+ raise ConnectionClosed('connection was interrupted')
rbuf.e += len_
log.debug('done (got %d bytes)', len_)
@@ -1263,14 +1358,14 @@ class HTTPConnection:
rbuf = self._rbuf
if len_ > len(rbuf.d):
raise ValueError('Requested more bytes than buffer has capacity')
- sock_fd = self._sock.fileno()
while len(rbuf) < len_:
if len(rbuf.d) - rbuf.b < len_:
self._rbuf.compact()
- if not self._try_fill_buffer():
- yield PollNeeded(sock_fd, POLLIN)
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
+ res = self._try_fill_buffer()
+ if res is None:
+ yield PollNeeded(self._sock.fileno(), POLLIN)
+ elif res == 0:
+ raise ConnectionClosed('server closed connection')
def readall(self):
'''placeholder, will be replaced dynamically'''
@@ -1279,9 +1374,6 @@ class HTTPConnection:
def co_readall(self):
'''Read and return complete response body'''
- if self._sock is None:
- raise ConnectionClosed('connection has been closed locally')
-
if self._in_remaining is None:
return b''
@@ -1316,6 +1408,14 @@ class HTTPConnection:
log.debug('discarding %d bytes', len_)
log.debug('done')
+ def reset(self):
+ '''Reset HTTP connection
+
+ This method resets the status of the HTTP connection after an exception
+ has occured. Any cached data and pending responses are discarded.
+ '''
+ self.disconnect()
+
def disconnect(self):
'''Close HTTP connection'''
diff --git a/examples/extract_links.py b/examples/extract_links.py
index 44b43cd..8f51149 100755
--- a/examples/extract_links.py
+++ b/examples/extract_links.py
@@ -11,15 +11,15 @@ from urllib.parse import urlsplit, urljoin, urlunsplit
import re
import ssl
-# We are running from the dugong source directory, make sure that we use modules
-# from this directory
+# We are running from the dugong source directory, append it to module path so
+# that we can fallback on it if dugong hasn't been installed yet.
if __name__ == '__main__':
basedir = os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), '..'))
else:
basedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if (os.path.exists(os.path.join(basedir, 'setup.py')) and
os.path.exists(os.path.join(basedir, 'dugong', '__init__.py'))):
- sys.path.insert(0, basedir)
+ sys.path.append(basedir)
from dugong import HTTPConnection
@@ -98,4 +98,3 @@ def main():
if __name__ == '__main__':
main()
-
diff --git a/examples/httpcat.py b/examples/httpcat.py
index 6c995df..e517d66 100755
--- a/examples/httpcat.py
+++ b/examples/httpcat.py
@@ -9,15 +9,15 @@ from io import TextIOWrapper
import re
from urllib.parse import urlsplit
-# We are running from the dugong source directory, make sure that we use modules
-# from this directory
+# We are running from the dugong source directory, append it to module path so
+# that we can fallback on it if dugong hasn't been installed yet.
if __name__ == '__main__':
... 1554 lines suppressed ...
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/python-modules/packages/python-dugong.git
More information about the Python-modules-commits
mailing list