[Piuparts-devel] exception installing parl-desktop-eu or parl-desktop-world

[Andreas Beckmann]

TL;DR: piuparts should not expect arbitrary command output to be utf-8

On 9/2/20 9:33 AM, Jonas Smedegaard wrote:
> The package debian-parl currently does not migrate to testing due to 
> piuparts failing - but I cannot figure out what is th actual failure: 
> Seems to me that it is an internal failure in piuparts itself (perhaps 
> involving very large commandline?).
> 
> Can I please ask you to have a look: 
> https://piuparts.debian.org/sid/source/d/debian-parl.html

That's the internal failure that gets recycled quickly so you
don't see the log in the web frontend:

0m12.3s DEBUG: Starting command: ['chroot', '/srv/piuparts.debian.org/tmp/tmpb2my8kx1', 'apt-get', '-y', 'install', 'parl-desktop-eu=1.9.25']

Piuparts caught exception, exiting...
------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/piuparts.debian.org/sbin/piuparts", line 3269, in <module>
    main()
  File "/srv/piuparts.debian.org/sbin/piuparts", line 3261, in main
    process_packages(regular_packages_list)
  File "/srv/piuparts.debian.org/sbin/piuparts", line 3176, in process_packages
    package_files, packages, extra_packages):
  File "/srv/piuparts.debian.org/sbin/piuparts", line 2379, in install_purge_test
    chroot.install_packages(package_files, packages, with_scripts=False)
  File "/srv/piuparts.debian.org/sbin/piuparts", line 1270, in install_packages
    self.install_packages_by_name(packages, with_scripts=with_scripts, reinstall=reinstall)
  File "/srv/piuparts.debian.org/sbin/piuparts", line 1351, in install_packages_by_name
    self.apt_get_install(to_install=packages, flags=target_flags, reinstall=reinstall)
  File "/srv/piuparts.debian.org/sbin/piuparts", line 1366, in apt_get_install
    self.run(command)
  File "/srv/piuparts.debian.org/sbin/piuparts", line 987, in run
    ignore_errors=ignore_errors, timeout=settings.max_command_runtime)
  File "/srv/piuparts.debian.org/sbin/piuparts", line 573, in run
    output += p.stdout.read(1 << 16)
  File "/usr/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 58: invalid continuation byte
------------------------------------------------------------

If I install parl-desktop-eu in a pbuilder chroot, I get

Setting up parl-desktop-eu (1.9.25) ...
Processing triggers for libc-bin (2.31-3) ...
Processing triggers for systemd (246.3-1) ...
Processing triggers for dictionaries-common (1.28.3) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
ispell-autobuildhash: Processing 'british' dict.
ispell-autobuildhash: Processing 'dutch' dict.

Word 'Abbas'' contains illegal characters

Word 'Abdelaziz'' contains illegal characters

Word 'Achilles'' contains illegal characters

Word 'Adrianus'' contains illegal characters

Word 'Aeneas'' contains illegal characters

Word 'Agnes'' contains illegal characters

Word 'Alanis'' contains illegal characters

Word 'Albertus'' contains illegal characters

Word 'Alex'' contains illegal characters

Word 'Alexandros'' contains illegal characters

Word 'Alexis'' contains illegal characters

Word 'Alfons'' contains illegal characters

Word 'Alies'' contains illegal characters

Word 'Alix'' contains illegal characters

Word 'Aloys'' contains illegal characters

Word 'Aloysius'' contains illegal characters

Word 'Alphons'' contains illegal characters

Word 'Alphonsus'' contains illegal characters

Word 'Amaryllis'' contains illegal characters

Word 'Amos'' contains illegal characters

Word 'Anas'' contains illegal characters

Word 'Ana�s'' contains illegal characters

Word 'Andreas'' contains illegal characters

Word 'Andres'' contains illegal characters

Word 'Andries'' contains illegal characters

[...]

which looks like the output contains some non-utf8 characters ...
but that does not seem to be a regression.
python choking on it seems to be the regression.

The last successful test was 
   Dec  4  2019 archive/pass/parl-desktop-eu_1.9.22.log.xz
This should have still been the python2 piuparts, so this is a regression
in piuparts switching to python3.

The likely culprit is 
https://salsa.debian.org/debian/piuparts/-/commit/669aad1f917ea82420ad01f781993707716ecfbd
I'm not sure what exactly that was supposed to fix, but assuming
that arbitrary command output is "text" in a valid encoding is
probably not a good idea.
(I'm only objecting the universal_newlines=True added in line 558
https://salsa.debian.org/debian/piuparts/-/commit/669aad1f917ea82420ad01f781993707716ecfbd#f30393d2b270e5e81c148b702a5cd6715c1fc6a7_557_558
the other uses on specific comands where output gets parsed the
utf-8 assumption is fine)


Andreas