Bug#1033406: licensecheck: scan-copyrights fails to create copyright file for texlive-extra

[Vignesh Raman]

Hi Jonas,

On 24/03/23 8:09 pm, Jonas Smedegaard wrote:
> Please try isolate the exact licensecheck command that scan-copyrights
> executes (e.g. by hacking that script to print to stderr before
> executing).
>
> When that exact command - including options and arguments - is known, we
> can move on to examine why the failure isn't deterministic.

This is the exact command.

licensecheck --encoding utf8 --copyright --machine --lines=300 
--shortname-scheme=debian,spdx --recursive 
'--ignore=^(\./)?(?s:(?>[^/]*?/po/)[^/]*)\Z|^(\./)?(?s:.*\~)\Z|^(\./)?(?s:CVS)\Z|^(\./)?(?s:DEADJOE)\Z|^(\./)?(?s:RCS)\Z|^(\./)?(?s:\..*\.swp)\Z|^(\./)?(?s:\.\#.*)\Z|^(\./)?(?s:\.arch\-ids)\Z|^(\./)?(?s:\.arch\-inventory)\Z|^(\./)?(?s:\.bzr)\Z|^(\./)?(?s:\.bzr\.backup)\Z|^(\./)?(?s:\.bzrignore)\Z|^(\./)?(?s:\.bzrtags)\Z|^(\./)?(?s:\.cvsignore)\Z|^(\./)?(?s:\.deps)\Z|^(\./)?(?s:\.git)\Z|^(\./)?(?s:\.gitignore)\Z|^(\./)?(?s:\.hg)\Z|^(\./)?(?s:\.hgtags)\Z|^(\./)?(?s:\.pc)\Z|^(\./)?(?s:\.shelf)\Z|^(\./)?(?s:\.svn)\Z|^(\./)?(?s:\{arch\})\Z|^(\./)?(?s:_MTN)\Z|^(\./)?(?s:_darcs)\Z|^(\./)?(?s:config\.guess)\Z|^(\./)?(?s:config\.sub)\Z|^(\./)?(?s:debian/(?>[^/]*?/)[^/]*\.asc)\Z|^(\./)?(?s:debian/(?>[^/]*?/)[^/]*\.gpg)\Z|^(\./)?(?s:debian/(?>[^/]*?/)[^/]*\.pgp)\Z|^(\./)?(?s:debian/apertis/[^/]*)\Z|~$|(^|/)\.\.swp$|(^|/),,.*(?:$|/.*$)|(^|/)(DEADJOE|\.cvsignore|\.arch-inventory|\.bzrignore|\.gitignore)$|(^|/)(CVS|RCS|\.pc|\.deps|\{arch\}|\.arch-ids|\.svn|\.hg|_darcs|\.git|\.shelf|_MTN|\.bzr(?:\.backup|tags)?)(?:$|/.*$)|(?:^|/)debian/((fix.scanned.)?copyright|changelog|NEWS|compat|.*yml|docs|source|patches/series)|(png|jpg|pdf|ico|bmp|jpe?g)$' 
.

> Your mentioning above that licensecheck failing is "not consistently
> reproducible" is another bug that is more serious, however: Licensecheck
> should - with same version also of its libraries - behave the same at
> each execution.
>
> So it would be helpful to me, independent of the issue of not parsing
> binary data - if you can help isolate the command which is unreliable.
>
When I run the above command the results for the problematic files are,

./texmf-dist/tex/latex/exp-testopt/exp-testopt.sty    UNKNOWN 2009 Paul 
Ebermann. | / 2008 Paul Ebermann
./texmf-dist/fonts/type1/public/baskervillef/BaskervilleF-Bold.pfb 
UNKNOWN    2016 / 2012 Rodrigo Fuenzalida (www.rfuenzalida.com|hello 
:rfuenzalida.com) / 2012 Pablo Impallari 
(www.impallari.com|impallari at gmail.com),
./texmf-dist/fonts/type1/public/baskervillef/BaskervilleF-BoldItalic.pfb 
UNKNOWN    2016 / 2012 Rodrigo Fuenzalida (www.rfuenzalida.com|hello 
:rfuenzalida.com) / 2012 Pablo Impallari 
(www.impallari.com|impallari at gmail.com),

Going with the above information, scan-copyrights should not crash. If I 
remove all other files and keep only the above in same directory 
structure, it does not crash.

Only when we run scan-copyrights with all the source files, it crashes.

> It is unclear what you are referring to as "this issue": What you
> summarise in the subject of this bugreport strictly speaking is an issue
> with libconfig-model-dpkg-perl, and above I have mentioned 2 potential
> underlying issues that might be your main concern here (as I assume you
> deliberately chose to file this bugreport against licensecheck).

Yes, I created the bug report against licensecheck because I suspected 
that parsing the binary files was causing the issue.

> If you are not really reporting a new issue here, but instead are asking
> for best practice of using licensecheck with sources that include binary
> data, then my most up-to-date notes on that is here:
> https://wiki.debian.org/CopyrightReviewTools#licensecheck

Thanks for the link.

If there are no issues with licensecheck, will report the issue in 
libconfig-model-dpkg-perl with all this information.

Regards,
Vignesh