Bug#654474: Doesn't contain source for waf binary code

[Ian Jackson]

Michael Biebl writes ("Doesn't contain source for waf binary code"):
> as this issue affects quite a few packages, I'd like to bring this up
> for wider discussion.
> 
> The issue basically is, that the waf build system uses a python script,
> which embeds a bz2 tarball containing further python sources. Those are
> unpacked to .waf-*/ when the waf script is executed. More details can be
> found at [1].

This is quite astonishing enough, but the situation is in fact even
worse than it appears.  I investigated, and my conclusions are:

"waf" is a build system written in Python.  It is normally distributed
in the form of a script called "waf", which the waf authors intend for
upstream authors to include in their upstream distribution tarballs
etc.  The script is a self-extracting archive which, whenever it's
run, extracts the tarball out of itself into a temporary directory and
then passes control to the python code it has just extracted.

This IMO would be bad enough to reject an upstream package, in this
form, in Debian.  After all we want to be able to modify the build
system as well as the package and this approach makes it unreasonably
hard to do so.  And if the build system has some kind of bug we don't
want to have to update dozens of copies embedded in individual source
packages.

But there is more, and worse.

I compared the tarball in waf in postler 0.1.1-1.1 with the upstream
code as obtained from "git clone https://code.google.com/p/waf/".  It
turns out that the tarball embedded in the "waf" script is not the
original "waf" source distribution.  It contains a subset of the
files, and those files it does contain have been processed to remove
comments, whitespace, etc., much like a JavaScript minimisation.  Ie
the "waf" self-extracting archive is generated out of the waf.git
source code by massaging some of the files; modified versions of the
script are supposed to be generated by editing the waf.git
distribution and rerunning its build.

This means that we are distributing files derived from the waf.git
source code, but not the waf.git source code itself.  This is of
course completely unacceptable in Debian.  (It is not a violation of
the copyright on waf itself as waf has a permissive non-copyleft
licence; but will be a breach of the copyright on any GPL'd waf-using
package, because the GPL's requirements extend to the build system.)

I suggest the following fix:

 * Upstream waf should be packaged somehow.  Upstream's declared
   policy of asking packages to ship a copy of waf suggests that there
   won't be much API stability so we will need to encode the waf
   version number in the package name, and we may need to package
   multiple versions of waf.

 * All packages which currently use an included copy of waf should be
   changed to use a system-provided copy of waf instead.

 * We should treat the file "waf" in the root of affected packages as
   we do any other file which is non-DFSG-compliant but which we do
   have permission to redistribute.  Our current practice is to repack
   upstream source archives to remove these files from the Debian
   sources.  (I think this is pointless makework but changing that
   policy is out of scope for this discussion.)

 * It is possible that some upstream "source" packages contain "waf"
   scripts which were generated from modified versions of waf.git.  In
   this case we may discover that those packages cannot be built with
   publicly available versions of waf.git.  

   For those packages, we need to obtain the relevant version of
   waf.git (for example, by hassling the package upstream), or perhaps
   if it's a simple change we can create our own suitable version of
   waf.git by "decompiling" (de-minimising) the supplied "waf" script
   tarball contents (and permanently maintain the forked waf.git).

   If neither of those is possible and the issue can't be worked
   around some other way we will have to move the affected package
   from main to non-free; if the affected upstream package is
   copylefted then we will have no permission to distribute it at all
   and must drop it entirely.

I think this is a release-critical bug for all the affected packages.

Ian.