Bug#828941: licensecheck: use binwalk to parse binary blobs?

[Jonas Smedegaard]

Quoting Andrej Shadura (2019-11-13 14:29:13)
> On Wed, 29 Jun 2016 08:24:43 +0000 (UTC) Gianfranco Costamagna
> <locutusofborg at debian.org> wrote:
> > Source: licensecheck
> > Severity: wishlist
> > Version: 3.0.0-1
> > 
> > Hi, as discussed on irc, it might be useful to use binwalk (now with 
> > a Python library/binding), to spot what is hidden/embedded into 
> > binary blobs, and then use the correct tool to search for 
> > copyrights/licenses.
> > 
> > Or, as Jonas suggested on irc, use it when in --strict mode, and the 
> > parse failed, to let the user know what was containing the blob, to 
> > better understand why licensecheck failed to parse it.
> > 
> > Pabs suggested hachoir tool
> 
> Using hachoir directly may slow down licensecheck significantly, but 
> maybe licensecheck would optionally get it involved if asked to. Jonas 
> (in a private email) suggested the slowdown issue is already quite 
> serious, so this path needs to be undertaken with caution.
> 
> However, I have this idea.
> 
> My personal issue with licensecheck is that it tries to parse binary 
> files, but parses them as text, thus dumping huge lumps of binary junk 
> into the generated copyright file:
> 
> Files: ./data/icons/hicolor/48x48/apps/com.github.maoschanz.drawing.png
>  ./help/C/figures/icon.png
> Copyright: ^@CC Attribution-ShareAlike
> http:creativecommons.org/licenses/by-sa/4.0/ÃTb^E^@^@^E<U+0091>IDATh<U+0081>í<U+0098>kl^TU^X<U+0086><U+009F>3{ë^h<U+008B>
>   n¸¤¼Ñ^E;{Ö<%¥Ì^Qz'ü<U+0083>E<U+0082>ë½<U+009D>
>   r^RtÅ|вxÒk³Ú ð¼
> License: CC-BY-SA
>  FIXME
> 
> This output is useless, wrong, and it takes a lot of time to generate 
> since binary files are typically bigger than text files.
> 
> What if we tried to detect binary files before parsing them?
> A very dumb algorithm would:
> 1) Check the first 8-16-32-whatever-sensible bytes for magic sequences
> of files that might contain copyright/license metadata, e.g. PNG, JPEG,
> SVG… (we need to keep this list short)
> 2) If something’s detected, parse that in a special way, Perl seems to
> have a lot of modules for that
> 3) If nothing found but the file looks binary (TBD how we detect this),
> use hachoir of whatever suitable if available, otherwise say UNKNOWN
> 4) Never dump binary stuff
> 
> At worst, a filter to remove non-ASCII stuff from binary-looking files
> would be very useful.

Licensecheck currently expects to be handed only sourcecode.

I agree that it makes great sense to expand to handle other file types 
as well, but *how* to handle other files depends on why you are running 
licensecheck at all.

Original purpose as authored by KDE developers was conformance with a 
narrow subset of licenses.  Extending to cover binary files would then 
probably mean a select few well-known extensions handed over to 
well-defined parsers - and everything else being either skipped or 
treated as an error, depending on more narrow use-case.

Common use nowadays for Debian packaging is to detect most possible 
copyright and licensing hints.  Extending to cover binary files would 
then probably consult libfile-libmagic-perl and/or file extensions, and 
maintain a list of more detailed parsers to hand it over to based on 
those.  Which detailed parser(s) to use and how insisting to be in 
drilling into content depends again on the more narrow use-case.

Should licensecheck detect or ignore or declare "None" for PDF content? 
PDF metadata fields? RDF resource embedded in PDF headers? metadata 
embedded in ICC profile embedded in PNG object embedded in PDF object?

I think that parsing binary data in Licensecheck should be optional, to 
limit complexity for those using it only for processing text-based 
sourcecode.

I think it should be configurable which parsers to use when, and offer 
some high-level "profiles" for common use-cases.

For use right now, I recommend to combine licensecheck with helper 
scripts part of cdbs (but *not* build-depend on or otherwise use cdbs).  
For examples of using those helper scripts to pre-parse some binary 
files and skip select other ones, while not accidentally silencing later 
introduced unknown types of files, see file debian/copyright-check in 
the source code of ghostscript (or pandoc or valentina), and the files 
/usr/lib/cdbs/license-miner and /usr/lib/cdbs/licensecheck2dep5 in 
package cdbs.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private