[Python-modules-team] Bug#797365: python-scrapy: 'import scrapy' appears to require lxml.html (python-lxml)

[Chris Dukes]

Package: python-scrapy
Version: 1.0.0-1
Severity: important

Dear Maintainer,

* What led up to the situation?
In researching #797364, tried to simplify to a minimal environment and a minimal test program.

Using Dockerfile 
```
FROM debian:sid

RUN export DEBIAN_FRONTEND=noninteractive ;\
	apt-get update &&\
	apt-get -y dist-upgrade &&\
	apt-get -y install python-scrapy --no-install-recommends

RUN python -c 'import scrapy'
```

Saw the error
Step 2 : RUN python -c 'import scrapy'
 ---> Running in 328238b88888
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/scrapy/__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "/usr/lib/python2.7/dist-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/usr/lib/python2.7/dist-packages/scrapy/http/__init__.py", line 11, in <module>
    from scrapy.http.request.form import FormRequest
  File "/usr/lib/python2.7/dist-packages/scrapy/http/request/form.py", line 9, in <module>
    import lxml.html
ImportError: No module named lxml.html
The command '/bin/sh -c python -c 'import scrapy'' returned a non-zero code: 1


I expected the test import to run cleanly.
I ran aptitude search '~Rrecommends:python-scrapy' '~Rsuggests:python-scrapy' '~Rpython-scrapy'

Looking for other ways of providing an XML parser.
I expected the output to include python-beautiful soup, but did not.



-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.1.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages python-scrapy depends on:
ii  python                2.7.9-1
ii  python-boto           2.38.0-1
ii  python-cssselect      0.9.1+git90c72b0-1
ii  python-libxml2        2.9.2+really2.9.1+dfsg1-0.1
ii  python-queuelib       1.1.1-1
ii  python-twisted-conch  1:15.2.1-1
ii  python-twisted-core   15.2.1-1
ii  python-twisted-mail   15.2.1-1
ii  python-twisted-web    15.2.1-1
ii  python-w3lib          1.11.0-1

Versions of packages python-scrapy recommends:
ii  ipython          2.3.0-2
ii  python-django    1.7.9-1
pn  python-guppy     <none>
ii  python-imaging   2.9.0-1
ii  python-lxml      3.4.4-1
ii  python-mysqldb   1.3.4-2
ii  python-pygments  2.0.1+dfsg-1.1

Versions of packages python-scrapy suggests:
ii  python-openssl  0.15.1-2

-- no debconf information