Bug#1002496: dh-perl6 vs. dh-raku: reproducibility issues with vendor/precompiled

[Chris Lamb]

Chris Lamb wrote:

> [...]

Just some further scrappy braindumping here; this is, unfortunately, not a
complete solution yet.

Looking into this further, a lot of this revolves around this method in
src/core.c/CompUnit/Repository/Installation.pm6:

    method id() {
        return $!id if $!id;
        my $name = self.path-spec;
        $name ~= ',' ~ self.next-repo.id if self.next-repo;
        my $dist-dir = $.prefix.add('dist');
        $!id = nqp::sha1(nqp::sha1($name) ~ ($dist-dir.e ?? $dist-dir.dir !! ''))
    }

... and this one in lib/CompUnit/Repository/Staging.rakumod:

    method path-spec(CompUnit::Repository::Staging:D:) {
        self.^name ~ '#name(' ~ $!name ~ ')#' ~ $.prefix.absolute;
    }


Taken together, these basically hash together two things:

  a) All of the repository names/paths used in the precompilation process.

  b) The target directory for the precompilation.

This value is then encoded in the filename of the repo-id file and
similar. (Just to check: is this file even needed?)

Just to take these two things in turn:


## A. Repository paths

It is fine that the hash includes things like /usr/lib/perl6/site.
However, pre-compilation looks in $HOME/.raku for modules and encodes
that value into the hash of the precompiled files. The hash therefore
varies on the build user's home directory location/name.

This occurs via RepositoryRegistry.pm6:

    nqp::bindkey($custom-lib,'home',
      $next-repo := self!register-repository(
        $home-spec,
        CompUnit::Repository::Installation.new(
          :prefix($home),
          :$next-repo
        )
      )
    ) if $home-spec and nqp::not_i(nqp::existskey($unique,$home-spec));

It probably isn't a good idea that Debian package builds inherits anything from
the build user's home directory anyway, so the following should be okay:

    --- a/dh_raku_build
    +++ b/dh_raku_build
    @@ -39,6 +39,7 @@ foreach my $pkg (getpackages()) {
                      --from=. --to=debian/tmp/pre-compiled!;
         doit({
             update_env => {
    +            HOME => "/nonexistent",
                 RAKUDO_RERESOLVE_DEPENDENCIES => 0,
             }
         }, at cmd);


## B. Target directory

This directory is typically `debian/tmp/pre-compiled` in Debian (via
`dh_raku_build`). Although this is a relative path name and should
therefore not embed the full path name, the directory is converted to
its absolute version within Staging.rakumod:

    self.^name ~ '#name(' ~ $!name ~ ')#' ~ $.prefix.absolute;

Raku will thus embed the fully-qualified pathname such as
/tmp/arbitrary-build-dir/perl6-module/debian/tmp/pre-compiled).

This could be fixed via:

    --- a/lib/CompUnit/Repository/Staging.rakumod
    +++ b/lib/CompUnit/Repository/Staging.rakumod
    @@ -11,7 +11,7 @@ class CompUnit::Repository::Staging is CompUnit::Repository::Installation {
             $!name
         }
         method path-spec(CompUnit::Repository::Staging:D:) {
    -        self.^name ~ '#name(' ~ $!name ~ ')#' ~ $.prefix.absolute;
    +        self.^name ~ '#name(' ~ $!name ~ ')#' ~ ($!name ne 'vendor' ?? $.prefix.absolute !! '');
         }
         method source-file(Str $name --> IO::Path) {
             my $file = self.prefix.add($name);

This cannot be fixed by replacing $.prefix.absolute with
$.prefix.relative, otherwise the build will vary on the number of
directory levels (eg. "../../../target" vs "../../../../target"). And
it doesn't appear to be fixable using $.prefix.path either; this is also
an absolute value, so something is converting it.

(Oh, misc: this proof-of-concept patch only changes the value for
"vendor" types. This is the repository type for local/Debian builds,
leaving the global "inst" types such as /usr/lib/perl6/site, etc.
unchanged.)


Another solution might be to do something quite aggressive like this:

  --- a/src/core.c/CompUnit/Repository/Installation.pm6
  +++ b/src/core.c/CompUnit/Repository/Installation.pm6
  @@ -616,6 +616,11 @@ sub MAIN(:$name, :$auth, :$ver, *@, *%) {

       method id() {
           return $!id if $!id;
  +        if %*ENV<SOURCE_DATE_EPOCH> -> $sde {
  +            $!id = nqp::sha1($sde);
  +            return $!id;
  +        }
           my $name = self.path-spec;
           $name ~= ',' ~ self.next-repo.id if self.next-repo;
           my $dist-dir = $.prefix.add('dist');

Whilst this sidesteps some of the above issues, it strangely doesn't
fix the contents of the files, which still seem to contain the target
(and possibly the source) directory.

Anyway, again, just braindumping here so I don't forget stuff. Thanks
for maintaining Rakudo.


-- 
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby at debian.org 🍥 chris-lamb.co.uk
       `-