Bug#867305: licensecheck: doesn't parse unicode files correctly

[Ximin Luo]

Package: licensecheck
Version: 3.0.29-1
Severity: normal

Dear Maintainer,

licensecheck seems to generate bad output for unicode files such as:

https://sources.debian.net/src/sagemath/7.6-2/sage/src/doc/ja/tutorial/tour_rings.rst

An example command line is:

$ licensecheck -l250 --deb-machine --merge-licenses src/doc/ja/tutorial/tour_rings.rst

I get glyphs like <U+008D>ã<U+0081> suggesting that maybe it is getting utf-8-encoded twice.

X

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'buildd-unstable'), (300, 'unstable'), (100, 'experimental'), (1, 'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8), LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages licensecheck depends on:
ii  libgetopt-long-descriptive-perl           0.100-1
ii  libmoo-perl                               2.003002-1
ii  libnamespace-clean-perl                   0.27-1
ii  libpath-iterator-rule-perl                1.009-1
ii  libpath-tiny-perl                         0.100-1
ii  libpod-constants-perl                     0.19-1
ii  libscalar-list-utils-perl                 1:1.47-1
ii  libsort-key-perl                          1.33-1+b3
ii  libstrictures-perl                        2.000003-1
ii  libstring-copyright-perl                  0.003005-1
ii  libstring-escape-perl                     2010.002-1
ii  libtry-tiny-perl                          0.28-1
ii  perl                                      5.24.1-4
ii  perl-base [libscalar-list-utils-perl]     5.24.1-4
ii  perl-modules-5.24 [libexperimental-perl]  5.24.1-4

Versions of packages licensecheck recommends:
ii  bash-completion  1:2.1-4.3

licensecheck suggests no packages.

-- no debconf information