Bug#1028254: hippotat build is not reproducible in compiled Rust

[Ian Jackson]

Package: hippotat
Verson: 1.1.3
User: reproducible-builds at lists.alioth.debian.org
Usertags: buildpath
X-Debbugs-Cc: reproducible-bugs at lists.alioth.debian.org

Unfortunately, the build is not reproducible.  I don't know why.
The build definitely captures the build path.

I'm filing this bug to record my investigations and what I have
found.  But I don't know what my next steps are.


diffoscope
----------

I looked at the tests.reproducible-builds.org output (for 1.1.3):
  https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/hippotat.html
There's a lot of differences in binary code there but that could just
be byte offsets.  The strings diff shows the build path being
captured
  /build/1st/hippot1.1.3/src/config.rs

That filename is in the hippotat tree so it doesn't seem like this
ought to be the fault of some dependency crate.  Many of the other
filenames in the hippotat source tree appear there too.

I grepped the source for uses of "file!", but there weren't any.
(I doubt I would have made such a mistake; this whole Rust codebase
postdates my awareness of the need for reproducibility.)

I ran "cargo expand" (using upstream dependencies and a nightly
compiler) with --lib and --bin hippotat.  The output does not contain
either (a) the build path (b) any invocztions of unexpanded ! macros.


grep for captured path
----------------------

I ran the Debian package build (manually set up sid chroot,
dpkg-buildpackage -uc -b) and grepped the target/ directory:

$ grep --exclude={\*.d,root-output,output,stderr,\*.rmeta} -lR /volatile/rawcargo target/ 
target/x86_64-unknown-linux-gnu/release/hippotatd
target/x86_64-unknown-linux-gnu/release/libhippotat.rlib
target/x86_64-unknown-linux-gnu/release/hippotat
target/x86_64-unknown-linux-gnu/release/deps/libtypenum-35b30b82f1b581e3.rlib
target/x86_64-unknown-linux-gnu/release/deps/hippotatd-816f104fd34c8e22
target/x86_64-unknown-linux-gnu/release/deps/hippotat-217cbc019ba3f5c7
target/x86_64-unknown-linux-gnu/release/deps/libhippotat-4b0cf61fee6d94b1.rlib
target/release/libhippotat_macros.so
target/release/deps/libnum_bigint-9e9ff3efb7aa6776.rlib
target/release/deps/libhippotat_macros-eb0640ebce7c05b7.so
target/release/deps/libhippotat_macros-92a972e6f986fd5e.so
$

strings on libtypenum-35b30b82f1b581e3.rlib shows consts.rs.  I got
the source for typenum (from the Debian package) and found that
consts.rs is generated by a cargo build script.  The file consts.rs
itself doesn't capture the path, nor does it invoke any ! macros.

strings on libhippotat_macros.so shows it having captured some parent
of the build directory.  But I don't think anything in there ought to
matter since this io is a proc macro crate - it doesn't end up in the
.deb.


build runes
-----------

Because hippotat's build system is not like a standard Debian Rust
package, I suspected I might have missed something that the standard
Debian Rust tooling would do.

So I used another package as a comparator.  A contributor on
#debian-rust suggested ripgrep.  So I checked and the rust-ripgrep
source package is currently reported to build reproducibly.

So I looked at the build logs.  I compared:

* The diagnostic output from the Debian cargo wrapper.  This was
  completely identical, showing that my way of invoking thigns is
  indeed using the cargo wrapper.

* One of the rustc command lines.  I chose proc_macro2 arbitrarily.
  These wouldn't be expected to be identical but they were quite
  similar.  Here's a wdiff with the changed parts nicely highlighted:

     Running `CARGO=/usr/bin/cargo CARGO_CRATE_NAME=proc_macro2

        [-CARGO_MANIFEST_DIR=/usr/share/cargo/registry/proc-macro2-1.0.47-]
        {+CARGO_MANIFEST_DIR=/<<PKGBUILDDIR>>/debian/cargo_registry/proc-macro2-1.0.47+}

        CARGO_PKG_AUTHORS='David Tolnay <dtolnay at gmail.com>:Alex
        Crichton <alex at alexcrichton.com>' [many unchanged things elided] CARGO_PKG_VERSION_PRE='' 

        [-LD_LIBRARY_PATH='/<<PKGBUILDDIR>>/target/release/deps:/usr/lib' OUT_DIR=/<<PKGBUILDDIR>>/target/release/build/proc-macro2-e08ec009b342643f/out-]
        {+LD_LIBRARY_PATH='/<<PKGBUILDDIR>>/target/debug/deps:/usr/lib' OUT_DIR=/<<PKGBUILDDIR>>/target/debug/build/proc-macro2-9cccbb8dcf1ff8a7/out+}

        rustc --crate-name proc_macro2 --edition=2018

        [-/usr/share/cargo/registry/proc-macro2-1.0.47/src/lib.rs-]
	{+/<<PKGBUILDDIR>>/debian/cargo_registry/proc-macro2-1.0.47/src/lib.rs+}

        --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no

        -C [-debug-assertions=off-]
           {+debuginfo=2+}

	--cfg 'feature="default"' --cfg 'feature="proc-macro"'

        -C [-metadata=ab9fbb9c9680acee-]
           {+metadata=aa8ae3518bc48afa+}
	-C [-extra-filename=-ab9fbb9c9680acee-]
	   {+extra-filename=-aa8ae3518bc48afa+}
	--out-dir [-/<<PKGBUILDDIR>>/target/release/deps-]
		    {+/<<PKGBUILDDIR>>/target/debug/deps+}
	-L [-dependency=/<<PKGBUILDDIR>>/target/release/deps-]
	   {+dependency=/<<PKGBUILDDIR>>/target/debug/deps+}
	  --extern [-unicode_ident=/<<PKGBUILDDIR>>/target/release/deps/libunicode_ident-9dcbf07dc1d6a76b.rmeta-]
		   {+unicode_ident=/<<PKGBUILDDIR>>/target/debug/deps/libunicode_ident-b18be38a27a886f6.rmeta+}

	--cap-lints warn --cfg use_proc_macro --cfg wrap_proc_macro`

  IDK about the release vs debug build.  The path difference is
  because I don't use the symlink local registry in
  ./debian/cargo_registry, but rather use /usr/share/cargo/registry
  directly.

I was not able to compare the cargo invocation rune, because although
my own arrangements do print it, the cargo debhelper buildsystem
plugin does it.  I reread
  /usr/share/perl5/Debian/Debhelper/Buildsystem/cargo.pm
to see if it might contain something relevant, but it didn't seem to.


Ian.

-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.