Bug#762638: metaconfig source control/distribution and Debian's DFSG

[Dominic Hargreaves]

Hi all,

Apologies for another long delay in replying; other matters have
taken up most of my recent energy. I'm replying to both
H.Merijn Brand and Andy Dougherty's most recent messages on the
subject here.

I've snipped some quoted parts for brevity, but kept what I hope
are the most pertinent bits of conversation, to help summarise.

On Sat, Oct 04, 2014 at 08:23:10PM +0200, H.Merijn Brand wrote:
> On Sat, 4 Oct 2014 17:50:51 +0100, Dominic Hargreaves <dom at earth.li>
> > On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote:

> > > If we include all the units in the source tree
> > > instead, I think end users will have an expectation of greater support.
> > > They will expect it to work, and work easily, right out of the box.
> > > Getting it to that point will require significant effort, and I really
> > > don't see the benefit.  Of course, if someone else wants to do it and
> > > support it, that's fine with me.
> > 
> > One benefit would be that we wouldn't this acknowledged bus factor
> > issue with the metaconfig wrangling.
> 
> Disagree. Areas in the perl core are picked up by people that like that
> area:
> 
> • Threading
> • 64bit/32bit
> • Regex
> • COW
> • Speed
> • Debugging
> • Toolchain
> • Configuration
> • …
> 
> It just happens that Configuration is NOT an area that appeals to
> developers. It works, and they are happy it works the way it does.
> Improving the configuration or the configuration process does NOT make
> the language itself more appealing or more interesting, and eventually,
> that is what the developers want (to achieve).
> 
> Yes, the bus-factor is (too) low, but it is not caused by
> (un)availability of the pieces

Okay, I accept this.
 
> > > > If we have to, we can make a tarball of that repository and include
> > > > it with the perl source package, but it seems like it would we should
> > > > explore the possibility of fixing this discrepancy upstream, rather
> > > > than working around it in Debian. In fact, it's likely we'd also have
> > > > to supply the patches between the current metaconfig output and what's
> > > > actually in the perl release tarball, since Configure is explicitly
> > > > allowed to be patched even though it's generated.
> > > 
> > > To be complete, you would have to include both perl's metaconfig
> > > repository and an appropriately patched 'dist' source tree, which would
> > > likely differ somewhat from Debian's existing 'dist' tree.  If the
> > > intent is to actually use it, there are probably a bunch of installation
> > > issues to work out.  If you want to check your results, you'll have to
> > > confront the reshuffling problem.
> > 
> > > Short summary:  I think it would be a lot of work for very little
> > > gain.
> > 
> > On Fri, Sep 26, 2014 at 08:34:58AM +0200, H.Merijn Brand wrote:
> > 
> > > The main problem with generating Configure from meta is not the
> > > availability of the metaunits used for perl, but the way meta/dist has
> > > to be installed to make them work the way it does.
> > 
> > Do you mean the fact that dist is being patched, or some other reason?
> 
> Just like learning a new language, the whole *process* is rather
> complicated. It is not like most GNU tools "sh autogen.sh",
> "configure", "make check", "make install", but - for people building
> perl - "Configure", "make test", "make install"
> 
> The source code for autogen is widely available, but it works fine only
> on Linux. (I'd invite you to the hell of using the auto* and (even
> worse) libtool on OS's like AIX and HP-UX.
> 
> To make the developers' life of our Perl community *less* of a hell, we
> are shipping "Configure", which is a autoconf/autogen/configure in one,
> that runs on almost every *nix like platform (VMS and Windows have
> special cased support).
> 
> For how many of the GNU tools that you ship, do you include the
> autogen.sh stuff that is only used *before* the configure is included
> in the source package? I only see the auto-stuff if I clone the source
> repo for the tool itself.

Most of the autotools stuff in Debian ships the source files and
it's generally encourages to rerun autoconf/automake to generate
configure these days, so that changes needed to support new architectures,
etc, can be reflected without any work on the source of the package.

> The fact that we patched and extended meta is NO reason for it to be
> more complex than using meta/dist from scratch. It is like installing a
> patched version of autoconf. The changes are reasonable well
> documented, and the source is widely and openly available.

Or at least it is now, thank you :)

> > > As the current maint for this process, I communicate with the
> > > maintainer of dist/meta on a rather regular basis. I give feedback of
> > > what we changed, and I look at the changes he makes to see if those
> > > will be usable for perl.
> > > 
> > > As part of the metaunits is still used from dist, part is a set of
> > > (slightly) modified units from dist, and part is the units written only
> > > for perl, keeping those in sync is a hell of a job. On simply cannot
> > > stay in sync 100%
> > 
> > Okay, so clearly from a pragmatic view we would need to ship our own version
> > of dist along with the rest of the stuff from the metaconfig repository.
> 
> Done. It is now in our public git repo.

Thanks.

> > > > It is also sometimes difficult to check or merge the generated output.
> > > > The metaconfig units are assembled in order based on dependencies,
> > > > but those dependencies don't fully constrain the order.  The result is
> > > > that the order of units can vary depending on who generates Configure.
> > > 
> > > Even if you rsync from one system to another and use exactly the same
> > > tools, the order can change based on locale setting :(
> > > 
> > > > Thus even a tiny patch to a single metaconfig unit can result in a
> > > > Configure patch with hundreds of lines that are just shuffling
> > > > parts around.
> > > > 
> > > > That's another reason why modifying the metaconfig units is not
> > > > necessarily the "preferred" form of modification of Configure.
> > > 
> > > If possible, modify the hint files instead.
> > 
> > So, one data point is whether Helmut's changes could in fact have
> > been achieved this way instead.
> 
> Yes
> 
> > > > Obviously, it's all a bit involved, and a fair amount of effort to
> > > > figure out.
> > > > 
> > > > The way it's set up now, we encourage people to simply patch Configure.
> > > > If someone wants to go the metaconfig route instead, it's a lot of
> > > > extra work, but presumably the person deliberately chooses that route
> > > > knowing it's extra work.  If we include all the units in the source tree
> > > > instead, I think end users will have an expectation of greater support.
> > > 
> > > One extra note here: the generation process decides what units to
> > > include based on the references used in the perl source tree. This
> > > means that not all units will be used. Including all units in a
> > > distribution might lead to extra confusion.
> > 
> > Could you expand on this a bit more? It's not obvious to me what
> > this means. Aside from the 'promise of support' issue, this seems
> > to be the main reason against moving the units into the main perl source,
> > which would itself be a huge step forward.
> 
> There are units that check for features that none of the perl core
> parts touch or use. The generator looks at the perl source code to see
> which defines, functions, symbols, and tests are used, and includes the
> units that provide those tests in to generated include:
> 
> Locating units...
> Extracting dependency lists from 877 units...
> Extracting filenames (C and SH files) from MANIFEST.new...
> Building a Wanted file...
>     Scanning .[chyl] .xs files for symbols...
>     Scanning .SH files for symbols...
> *** Obsolete symbols found -- see file 'Obsolete' for a list.
> Computing optimal dependency graph...
>     Building private make file...
>     Determining loadable units...
> make: Circular perllibs <- perllibs dependency dropped.
>     Updating make file...
> Determining the correct order for the units...
> make: Circular perllibs <- perllibs dependency dropped.
> Creating Configure...
> Check if HAS_GETGROUPS  (26) precedes Groups_t  (97)
> Check if BYTEORDER      (135) precedes UVSIZE   (487)
> Check if LONGSIZE       (300) precedes BYTEORDER        (135)
> Check if HAS_SETGROUPS  (66) precedes Groups_t  (97)
> Check if HAS_QUAD       (314) precedes I64TYPE  (487)
> Check if MULTIARCH      (306) precedes BYTEORDER        (136)
> Check if MULTIARCH      (136) precedes MEM_ALIGNBYTES   (131)
> Check if HAS_GETGROUPS  (26) precedes Groups_t  (97)
> Check if BYTEORDER      (137) precedes UVSIZE   (487)
> Check if LONGSIZE       (136) precedes BYTEORDER        (137)
> Check if HAS_SETGROUPS  (66) precedes Groups_t  (97)
> Check if HAS_QUAD       (314) precedes I64TYPE  (487)
> Check if MULTIARCH      (131) precedes BYTEORDER        (137)
> Check if MULTIARCH      (131) precedes MEM_ALIGNBYTES   (132)
> Done.

Thanksm this is clearer now.
 
> > > > Short summary:  I think it would be a lot of work for very little
> > > > gain.
> > > 
> > > 100% agree
> > > 
> > > I'd rather put my free time in trying to get closer to dist-4.0 than we
> > > are now (3.5+) than modifying the current process to enable
> > > distribution of something that would work for all users that are
> > > unlikely to use it anyway
> > 
> > I'm genuinely confused here. Isn't making the perl build infrastructure
> > less dependent on a small number of people a significant gain, even if
> > you don't care about the ideological importance of the 'preferred form
> > of modification' requirement from Debian?
> 
> Now *I* am confused. I do not understand the question.
> 
> I volunteered to do this work, and I have so far no reason to quit.
> I don't see the maintenance of Configure from the units as a strain or
> a burden. Only *IF* other/more people are *seriously* interested in
> sharing in this process, modifying the process itself would be worth
> considering. Only modifying the process to enable people that will not
> show help in this area anyway is a waste of time.

This is a very fair point, and I do greatly appreciate your efforts in
this area.

> > On Mon, Sep 29, 2014 at 01:33:41PM -0400, Andy Dougherty wrote:
> > > 
> > > One final follow-up:  Debian is certainly welcome to distribute the
> > > files used to build Configure.  The following two items would be needed:
> > > 
> > >     1.  The precise version of dist used.  This can be obtained here:
> > > 
> > >         http://www.cpan.org/modules/by-authors/id/H/HM/HMBRAND/perl-meta-3.5-20.tgz
> > 
> > Incidentally, this should be mentioned in the comments at the top of
> > Configure, rather than the misleading reference to the 'normal' version
> > of dist.
> 
> It is now obsolete. I still have to modify the docs to explain the new
> state. I had no time yet.

Unless I'm misunderstanding, the attached trivial patch would go a long
way towards sending people in the right direction, even if there is more
detail to be added.

> > >     2.  The custom metaconfig units used.  These are available here:
> > >      
> > >         git://perl5.git.perl.org/metaconfig.git
> > > 
> > > Rebuilding Configure would not be easy or automatic, but all the necessary
> > > files would be available.  Would that satisfy the Debian guidelines?
> > 
> > I think it would satisfy the letter of the law, if not the full spirit.
> > It's a starting point; I'd like it to not be the ending point, but
> > I guess I would need to convince you and/or H.Merijn Brand of this in
> > order to make significant progress.
> > 
> > Having to pull in extra sources is a bit of a pain for the Debian package
> > management, but it's doable. The main thing that seems to be missing is
> > a way of knowing what version of that repository is to be used (or was
> > used) for a given release of perl. That doesn't seem to be recorded
> > in either the metaconfig repository or the perl repository.
> 
> The units and the source are now (in sync) in the same repo. After the
> docs are changed, it should be clear(er) how to use the bunch

Okay, let me know if we can help with this from the Debian POV.

> > The other option open to us is to convince Debian that the metaconfig
> > units genuinely are external bits of tooling that aren't part of the
> > build process, and that Configure really is the preferred form of
> > modification. Working through Helmut's use case might help there.
> 
> That is still *my* preference!

I think I'm starting to see that we can get to a situation which is
closer to the autotools situation, which I think will help - even if
we're not (yet) in a position to rerun the generation of Configure
automatically during the perl package build process.

On Mon, Oct 06, 2014 at 08:47:20AM -0400, Andy Dougherty wrote:
> On Sat, Oct 04, 2014 at 05:50:51PM +0100, Dominic Hargreaves wrote:
> 
> > On Thu, Sep 25, 2014 at 09:07:59PM -0400, Andy Dougherty wrote:
> > > On Thu, Sep 25, 2014 at 10:16:32PM +0100, Dominic Hargreaves wrote:
> 
> > > The way it's set up now, we encourage people to simply patch Configure.
> > > If someone wants to go the metaconfig route instead, it's a lot of
> > > extra work, but presumably the person deliberately chooses that route
> > > knowing it's extra work.
> > 
> > Again, there is an ideological point here. It *shouldn't* be a lot of
> > extra work to do things in the way that upstream developers would.
> > Clearly perl isn't going to be kicked out of Debian because of this,
> > but a less important package might well be.
> 
> I think we may have confused you here such that you have it exactly
> backwards.  It *isn't* a lot of extra work for anyone to do things in
> the way the upstream developers would.  Everyone has access to the exact
> same source.  What a few upstream developers do have is *experience*
> using the package, so that they can do that work somewhat more easily.
> 
> Those few upstream developers (well, really only H.Merijn these days) have
> volunteered to help do that work for people who would rather spend their
> time fixing something else rather than learning a complete configuration
> system.  Thus if a casual hacker wants to make a simple Configure change,
> they can simply make it directly to Configure.  Thus it is *easier* for
> the casual hacker to get involved.  If they want to do it the way the
> upstream developer would, then they have to do the same hard work that
> the upstream deveveloper does, and they are certainly free to do so.
> 
> Both the dist subversion repository and the perl metaconfig git
> repository are freely available, so anyone can check out the appropriate
> versions to recreate Configure (subject to machine-dependent ordering).
> I hadn't realized that the precise versions used weren't clearly labeled
> because I don't recall anybody ever asking before.  Encouraged by this
> request, I will try to remember and document that more clearly in perl's
> documentation.  If someone else wants to do it first, the patches would
> certainly be welcome.
> 
> > Okay, so clearly from a pragmatic view we would need to ship our own version
> > of dist along with the rest of the stuff from the metaconfig repository.
> > Depressingly this violates another part of Debian policy relating to
> > embedding copies of code not intended to be embedded, but the freedom
> > to modify the code using the preferred form clearly overrides that in
> > my mind.
> 
> You could instead separately package dist-3.5.20 and depend on that,
> if you liked.  (Presumably, that package would point to the standard
> 'dist' package as the recommended starting point for new projects,
> and would not overwrite the standard 'dist' package files.)  I don't
> see how this situation is fundamentally any different from that of any
> other program built with a particular version of an external tool (such
> as bison, for example).
> 
> > > Rebuilding Configure would not be easy or automatic, but all the necessary
> > > files would be available.  Would that satisfy the Debian guidelines?
> > 
> > I think it would satisfy the letter of the law, if not the full spirit.
> 
> I'm sorry, but I really fail to see why, and suspect that there is
> a lingering misunderstanding.  Rebuilding Configure is not easy or
> automatic for *anyone*, but that's not an issue of freedom.  I want to
> be helpful and share what we have, but we can only share what we have.

And I think my answer above applies here, too.

I will propose within Debian that we package up the modified dist
package and metaconfig units, and include documentation in the perl
source package about this.

Thanks,
Dominic.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Point-to-customised-dist-package-in-Configure-commen.patch
Type: text/x-diff
Size: 1037 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/perl-maintainers/attachments/20141101/a890e3e2/attachment.patch>