Re: Help with Mroonga to finally get all of MariaDB Server reproducible

[Chris Lamb]

Hey Otto and Vagrant,

Otto, thank you for your enthusiasm towards getting this package
reproducible. It's clear from re-reading the issue on the GitHub issue
for Mroonga how much you want to solve this.

To respect that work, I spent I little more time on this yesterday and
today although a considerable amount of time was spent compiling; it
takes well over 30 minutes to compile the mariadb-10.5 package locally.

(Just to quickly say though, reading over some old posts I think there
are points where you may be inadvertently muddling cause and effect.
It's very easy to do when looking at diffoscope output, but changes
to, for example, offsets are often the result of some other cause —
fixing the offset value itself is unfortunately not going to fix the
problem and can ultimately just be a distraction and timesink. You
don't say any of this explicitly, but your use of some specific
diffoscope screenshots does somewhat suggest that. And I write this
paragraph only in order to save everyone time and energy!)

> Pretty sure this is the zeroed-out build path length getting embedded
> (e.g. /build/path/1 vs. /build/path/1/2 have a different number of
> characters).

Interesting theory. So to test that, I built the version using
differing build paths of the *same* length, and there was
unfortunately still a diff between the ha_mroonga.so files. However,
the 'amount' of differences were significantly reduced compared to the
version currently on tests.reproducible-builds.org. Here is what I
got:

  https://people.debian.org/~lamby/EiquoKi5.html

On a hunch, I then repeated the build to see if it varied between
builds using the identical (same length) build paths. I got the
following difference in this case:

  https://people.debian.org/~lamby/EiquoKi5-2.html

i.e. exactly the same as before. (This is actually a good thing, as
it means that we have a deterministic difference, and not something
based on, say, a random number.)

Looking at the just-built (unstripped?) version of ha_mroonga.so, I
can find the following paths embedded in the binary:

  $ strings builddir/storage/mroonga/ha_mroonga.so
  /home/lamby/temp/cdt.20210723120540.jpheJcnR5m.repro.mariadb-10.5/build-a/mariadb-10.5-10.5.11/storage/mroonga/vendor/groonga/lib:/home/lamby/temp/cdt.20210723120540.jpheJcnR5m.repro.mariadb-10.5/build-a/mariadb-10.5-10.5.11/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/normalizers:/home/lamby/temp/cdt.20210723120540.jpheJcnR5m.repro.mariadb-10.5/build-a/mariadb-10.5-10.5.11/libservices:

This looks like something to do with GROONGA_PLUGINS_DIR, so it may be
possible to set this to a fixed path at the configure stage. The
version of ha_mroonga.so that ends up in the binary package does NOT
have these paths in the output of strings(1), although that of course
does not invalidate Vagrant's theory -- they could have been zeroed
out.

Focusing on the varying-length zeroed-out build path for a moment: I'd
dearly love diffoscope to be able to display that clearly. The current
output does not communicate that effectively, assuming that is the
problem.

That that end, Vagrant do you have a more info for exactly what tool
is doing this? Any GCC (??) documentation that you can reference? And
perhaps a tool or utility that can show precisely this? I'd like to at
least get a reliable testcase that we can incorporate into diffoscope.

> Once your package with the timestamp patches applied migrates to
> "testing" where we don't test build paths, it should be reproducible.

(Just to clarify that the timestamp patches are in the experimental
branch and not the version of the package in unstable. They therefore
will not automatically migrate.)


Best wishes,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      lamby at debian.org 🍥 chris-lamb.co.uk
       `-