sound distribution practices

[Allison Randal]

On 04/07/2012 12:34 PM, Jonathan Worthington wrote:
> Easiest way to analyze it is just git log on
> build/tools/PARROT_REVISION. Here I've done a very rough categorization
> of the last 20 entries (which takes us back to October).
> 
> Some of them are to take advantage of Parrot feature additions:
> 
>     bump PARROT_REVISION to get separate unlink and rmdir primtives
> 
>     use parrot's new Hash.update method to speed up method cache generation
> 
>     bump parrot revision to one that has the is_inf_or_nan opcode
> 
>     use parrot op to switch to the profiling runcore
> 
> So not bumping would mean Rakudo forgoing making user-facing improvements.

Okay, so these result from eager use of new features. From a
distribution perspective, it would be desirable to make these changes
with a smoother transition. On the Parrot side, there's nothing to do
here, it's just an introduction of a new feature and harmless if it's
not adopted. But on the Rakudo side, it would be possible to introduce
some levels of backward compatibility for very new Parrot features.

> Some are to get Parrot fixes:
> 
>     bump parrot revision to get MacOS build fixes
> 
>     Bump to a Parrot version with a fix for ByteBuffer segfaults, which
> caused programs using Buf to sometimes segfault.
> 
> Again, there are Rakudo user benefits to following this.

These are things that Rakudo shouldn't care about. A bug-fix or
performance enhancement in Parrot is good, but doesn't actually affect
Rakudo's ability to build or run.

> Some are due to chasing API changes in Parrot:
> 
>     Update to new version of getprop op
> 
>     Bump parrot revision to kill_props_vtables marge commit.
> 
> The getprop op changed at a PIR level. No guts poking here, just a (for
> the better) change. I should add that when I say "chasing" in reality
> Parrot folks have supplied the patches here, and it's been managed
> smoothly.

Did the actual APIs change? Different arguments required, or
substantially different behavior so that there was absolutely no way
Rakudo could run on both the old and new versions of Parrot? If so, then
Parrot could be doing a better job of supporting language development
here, by preserving old APIs.

> One is due to getting a PAST improvement:
> 
>     bump parrot revision to get directaccess support, use directaccess
> for most var lookups
> 
> We're currently in the process of redeveloping PAST so it's written in
> NQP, can do native types better, is more memory efficient and so forth,
> and that work will be in the NQP repository, which would have avoided a
> bump of this nature - but it was just one bump out of 20.

This is eager adoption of new features again. But, in this case I agree
the fix is that PAST should move to NQP and out of the Parrot packages
entirely. For one thing, having NQP depend on a library in Parrot that
depends on NQP is nightmarish for packaging. But also, in general it's
best to bundle the code that's most tightly coupled, for sanity of
maintenance.

> Which leaves the biggest group, which are a combination of a policy of
> targeting the latest Parrot in releases and wanting to have those using
> Rakudo from git testing against latest Parrot after sizable changes, so
> we can have faster feedback if there are problems, which would seem
> desirable for both Rakudo and Parrot devs.
[...]
> Granted some in this category aren't explicit about the reason why, so
> there's some error in the analysis here.

Indeed, not enough detail there for much analysis. But, in general there
is a difference between what you choose as your development process, and
what you choose as your release process. It is very sensible to always
test Rakudo against the latest versions of Parrot. But, that doesn't
mean you can stop testing against older version of Parrot. The most
important older versions to test against are the ones shipped in various
distributions.

> There seems to be a few things going on here. One is that we don't
> differentiate Parrot revision requirement updates that are "must have"
> from those where we're bumping it for development reasons rather than
> because it's a hard dependency. Another is that Rakudo's feature needs
> often drive feature additions to Parrot and drive some bug fixes too; it
> seems inevitable that Rakudo will want to quickly take advantage of them.

Yes, it seems like we could make quite a bit of headway simply by
separating the "minimum required version" of Parrot from the "suggested
version for all the latest bells and whistles". The minimum should only
be bumped when Rakudo cannot build or run on any older version.

It would also be helpful if Rakudo and NQP adopted more strategies to
keep the minimum required version as low as possible. Like "soft
launching" new features so they're simply not available when they depend
on a newer Parrot feature.

> Generally, I think things are currently more -Odevelopment than
> -Odistribution. I agree that makes things harder for those working on
> distributions, and I'm interested to make things better - I want it to
> be as easy for people to get their hands on recent Rakudo releases, and
> having it packaged for distributions does help with that. On the other
> hand, it's also an accurate reflection of the current state of both
> projects: in need of development work, and optimizing for it.

I guess that raises the question for me of whether Rakudo is ready to be
packaged. I don't have any trouble with Parrot packaging, but I do get a
regular series of unusual requests from the Rakudo packagers. If the
Rakudo project is totally focused on the use-case of downloading the
source code or checking out from git and compiling from source, maybe
it's just not ready for packaging.

I'm sure all the Rakudo packagers will agree with me that they'd really
hate to remove it from Debian. But, if packaging isn't something Rakudo
wants to support, then we shouldn't force the project into a different
mindset. To give you an idea of what Debian packaging is looking for,
take a skim through the upstream guide:

http://wiki.debian.org/UpstreamGuide

> Also just to check - are you packaging Rakudo compiler releases, or
> Rakudo distribution releases (which are the ones we more intend for
> packaging)? Mostly asking due to the timing of the email; we didn't do a
> distribution release for March (so last interested one to package was in
> Feb), and the next one is due this month.

I'm not building the packages for Rakudo, so can't answer. I've CC'd the
Rakudo packaging team.

Allison