Bug#946451: spyder: work on new upstream release

[Julian Gilbey]

Dear Drew and everyone,

(This message is Bcc'd to the two ITPs for python-language-server
(#963605) and python-jsonrpc-server (#964497), but it makes sense for
the discussion to remain on one report only.  I've also cc'd Otto -
see below.)

I was frustrated that I couldn't use Spyder 3 any more because of some
fatal bug (#976966), and I really want to use Spyder 3.  So I decided
over the last few days to package the new (4.2.1) Spyder dependencies
and see where I got to.  So this is a long email giving the state of
play after my experiments.

tl;dr - I still can't run Spyder and I don't know why not.  But I have
built all of the dependencies and they are all on salsa; I haven't yet
submitted ITPs for them but they are all ready to go.  (I also haven't
checked whether anyone else has already done so, except for the
repository that already existed in the Python teams project salsa.)
Should I submit ITPs and upload what I have done so far?

Long version
------------

Here is the dependency chain for Spyder 4.2.1 (only listing packages
not yet in the archive); source packages are named first, and the
relevant binary packages listed in parentheses afterwards:

Level 1 (can be built entirely within the current sid environment)
-------

python-jsonrpc-server (python3-pyls-jsonrpc) - already has ITP #964497
syllabipy (python3-syllabipy)
pylzss (python3-lzss)
paq (python3-paq)
distance (python3-distance)
pyxdameraulevenshtein (python3-pyxdameraulevenshtein)
py-stringmatching (python3-py-stringmatching)
three-merge (python3-three-merge)
wurlitzer (python3-wurlitzer) - already has ITP #942829

Level 2 (depends on at least one Level 1 package)
-------
python-language-server (python3-pyls) - already has ITP #963605
 - depends:
     python3-pyls-jsonrpc
abydos (python3-abydos)
 - depends:
     python3-syllabipy
     python3-lzss
     python3-paq
spyder-kernels [upgrade]
 - depends:
     python3-wurlitzer

Level 3 (depends on at least one Level 2 package)
-------
pyls-black (python3-pyls-black)
 - depends: python3-pyls
pyls-spyder (python3-pyls-spyder)
 - depends: python3-pyls
textdistance (python3-textdistance)
 - depends:
     python3-abydos
     python3-distance
     python3-pylev (in NEW)
     python3-pyxdameraulevenshtein
     python3-py-stringmatching

Level 4 (depends on at least one Level 3 package)
-------
spyder [upgrade]
 - depends:
     python3-pyls-black
     python3-pyls-spyder
     python3-spyder-kernels (>= 1.10.1)
     python3-textdistance
     python3-three-merge


Notes on this chain:

* All of these packages are available on salsa as
  https://salsa.debian.org/python-team/packages/PKG

  I have included myself and Otto Kekäläinen <otto at debian.org> as
  uploaders - I included Otto just because he is already listed on the
  python-language-server packages.  Happy to hand any of these over to
  anyone else who wants them!

* python3-jedi has been upgraded to version 0.18.  This introduced
  incompatible changes that have broken at least python3-ipython,
  python3-pyls and those packages that depend on them.

* python3-parso has been upgraded to version 0.8.1.  This has had a
  similarly problematic effect.

* syllabipy is a deprecated package.  There is a very recent pull
  request on nltk to incorporate the rest of the functionality of
  syllabipy into nltk.  Once that happens, there can be a request to
  abydos to remove its dependency on syllabipy (it already depends on
  nltk) and syllabipy can be dropped completely.

* I have forked the spyder-kernels and spyder repositories on Salsa
  and pushed my work to the experimental (and upstream and
  pristine-tar) branches.


OK, so where are we now?

Well, it's not very good news, unfortunately.

I try running spyder and this is what I get:

euler:~ $ spyder 
Traceback (most recent call last):
  File "/usr/bin/spyder", line 33, in <module>
    sys.exit(load_entry_point('spyder==4.2.1', 'gui_scripts', 'spyder')())
  File "/usr/lib/python3/dist-packages/spyder/app/start.py", line 213, in main
    mainwindow.main(options, args)
  File "/usr/lib/python3/dist-packages/spyder/app/mainwindow.py", line 3624, in main
    mainwindow = create_window(app, splash, options, args)
  File "/usr/lib/python3/dist-packages/spyder/app/mainwindow.py", line 3482, in create_window
    main.setup()
  File "/usr/lib/python3/dist-packages/spyder/app/mainwindow.py", line 514, in setup
    self, icon=ima.icon('close_pane'),
  File "/usr/lib/python3/dist-packages/spyder/utils/icon_manager.py", line 421, in icon
    return qta.icon(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/qtawesome/__init__.py", line 125, in icon
    return _instance().icon(*names, **kwargs)
  File "/usr/lib/python3/dist-packages/qtawesome/iconic_font.py", line 259, in icon
    parsed_options.append(self._parse_options(specific_options,
  File "/usr/lib/python3/dist-packages/qtawesome/iconic_font.py", line 308, in _parse_options
    prefix, chars = self._get_prefix_chars(names)
  File "/usr/lib/python3/dist-packages/qtawesome/iconic_font.py", line 339, in _get_prefix_chars
    raise Exception(error)
Exception: Invalid icon name "window-close-o" in font "fa"


OK, so that isn't great.  I also tried running the test suite.  The
runtests.py script fails miserably, so instead I did the following
(with python3-pytestqt and python3-pytest-mock installed):

for i in $(find spyder -type d -name tests); do pytest-3 $i; done

(Don't even think about using xargs instead - it crashes before doing
anything useful.)  This has to be run in an interactive graphical
environment, as they test a lot of Qt stuff.

There are 40 test directories, and most of them have test failures.
Some of these might be related to the python3-jedi and python3-parso
issues, but the results are pretty depressing nevertheless: I
summarise from 32838 lines of terminal output:

spyder/app/tests
  5 passed, 90 skipped, 2 warnings
spyder/config/tests
  1 failed, 69 passed, 2 warnings (Exception: Invalid icon name
  "window-close-o" in font "fa"; this is what stopped spyder from
  starting up)
spyder/plugins/completion/fallback/tests
  failed to run tests (Exception: Invalid font prefix "mdi")
spyder/plugins/completion/kite/parsing/tests
  1 passed
spyder/plugins/completion/kite/utils/tests
  1 skipped
spyder/plugins/completion/languageserver/tests
  failed to run tests (Exception: Invalid font prefix "mdi")
spyder/plugins/completion/snippets/tests
  failed to run tests (Exception: Invalid font prefix "mdi")
spyder/plugins/console/tests
  2 warnings, 2 error
spyder/plugins/editor/extensions/tests
  2 passed, 2 warnings, 69 error
spyder/plugins/editor/panels/tests
  1 skipped, 2 warnings, 3 error
spyder/plugins/editor/tests
  1 passed, 18 error
spyder/plugins/editor/utils/tests
  25 passed
spyder/plugins/editor/widgets/tests
  failed to run tests (Exception: Invalid font prefix "mdi")
spyder/plugins/explorer/tests
  1 passed, 2 warnings
spyder/plugins/explorer/widgets/tests
  1 failed, 21 passed, 45 warnings, 30 error
spyder/plugins/findinfiles/tests
  4 passed, 16 error
spyder/plugins/help/tests
  3 skipped, 2 warnings, 2 error
spyder/plugins/history/tests
  1 passed, 2 warnings, 5 error
spyder/plugins/ipythonconsole/comms/tests
  2 passed, 2 warnings
spyder/plugins/ipythonconsole/tests
  1 passed, 6 skipped, 6 warnings, 58 error
spyder/plugins/ipythonconsole/utils/tests
  3 passed, 1 skipped, 2 warnings
spyder/plugins/ipythonconsole/widgets/tests
  4 passed, 4 warnings
spyder/plugins/onlinehelp/tests
  2 skipped, 1 error
spyder/plugins/outlineexplorer/tests
  5 failed, 1 skipped, 2 warnings
spyder/plugins/plots/widgets/tests
  30 passed, 1146 warnings
spyder/plugins/profiler/tests
  1 passed, 3 warnings, 3 error
spyder/plugins/projects/tests
  3 warnings, 21 error
spyder/plugins/projects/widgets/tests
  4 passed, then threw a fatal error (Exception: Invalid font prefix "mdi")
spyder/plugins/pylint/tests
  12 failed, 11 passed, 2 warnings
spyder/plugins/variableexplorer/tests
  1 failed, 1 passed, 2 warnings
spyder/plugins/variableexplorer/widgets/objectexplorer/tests
  9 failed, 2 warnings
spyder/plugins/variableexplorer/widgets/tests
  7 failed, 85 passed, 5 skipped, 180 warnings
spyder/plugins/workingdirectory/tests
  1 passed, 2 warnings, 3 error
spyder/preferences/tests
  1 failed, 14 passed, 31 warnings, 1 error
spyder/tests
  8 passed
spyder/utils/introspection/tests
  1 passed, 4 warnings
spyder/utils/tests
  2 failed, 61 passed, 13 skipped, 2 warnings
spyder/widgets/github/tests
  6 passed
spyder/widgets/tests
  36 passed, 5 skipped, 182 warnings, 4 error
spyder/workers/tests
  6 skipped


Most of the errors were to do with font prefixes, which I don't
understand, so any help there would be appreciated!

So what would be helpful now?  Shall I file ITPs and upload the
supporting packages to NEW?

Best wishes,

   Julian