Bug#1035704: proj: reproducible-builds: timezone-dependent timestamps in .gsb/.gtx files

[Sebastiaan Couwenberg]

On 5/9/23 05:47, Sebastiaan Couwenberg wrote:
> On 5/8/23 22:43, Vagrant Cascadian wrote:
>> On 2023-05-08, Sebastiaan Couwenberg wrote:
>>> On 5/8/23 02:07, Vagrant Cascadian wrote:
>>>> The attached patch fixes this by not touching these files during the
>>>> build process.
>>>
>>>   From shar(1):
>>>
>>> "
>>>     -m, --no-timestamp
>>>            do not restore modification times.
>>>
>>>            Avoid generating 'touch' commands to restore the file
>>>            modification dates when unpacking files from the archive.
>>>
>>>            When file modification times are not preserved, project build
>>>            programs like "make" will see built files older than the 
>>> files
>>>            they get built from.  This is why, when this option is not
>>>            used, a special effort is made to restore timestamps.
>>> "
>>>
>>> That should be used when generating the archives instead of your patch
>>> to not have the mtimes touched when unpacking the archives.
>>
>> Is it actually a problem to allow dpkg to normalize the timestamps on
>> these files rather than forcefully setting them to ... a value from a
>> shar archive? It is perhaps a naive question; I really do not know.
> 
> Where does dpkg normalize the timestamps?
> 
> shar sets the timestamps when the archive is unpacked before the package 
> built starts.
> 
> Some of the files in the diffoscope-results are only installed in 
> proj-data and not used otherwise during the build.
> 
>   * BETA2007.gsb is used in test/gie/DHDN_ETRS89.gie
> 
>   * CHENYX06.gsb/CHENYX06_etrs.gsb/CHENYX06a.gsb are only installed
> 
>   * egm96_15.gtx is used in test/gie/deformation.gie,
>     test/gie/more_builtins.gie, test/gie/4D-API_cs2cs-style.gie, and
>     test/cli/testdatumfile
> 
>   * ntf_r93.gsb is used in test/gie/more_builtins.gie,
>     test/gie/4D-API_cs2cs-style.gie, and test/cli/testdatumfile
> 
>   * nzgd2kgrid0005.gsb is used in unit tests
> 
>>> But the diffoscope-results only show a few of the files from the shar
>>> archives with different mtimes, and all of them get touched when
>>> unpacking the archive just before the configure target is executed.
>>
>> Well, I too was perplexed why other files were not affected, but it is
>> consistently those .gsb and .gtx files, and the submitted patch allows
>> to consistently build reproducibly in the dozens of times I've build
>> it...
>>
>>
>>> shar actually makes the timestamps reproducible by always using the
>>> mtime recorded in the archive instead of the time the files were 
>>> unpacked.
>>>
>>> Something else is likely changing the mtime after the files are
>>> unpacked. Some of these grids are used by the testsuite for example.
>>
>> I will try to look into it further, but without really being familiar
>> with the proj codebase (or even what proj itself does)... any additional
>> very specific suggestions where to look next would definitely be
>> appreciated!  :)
> 
> CMake's configure_file() is used to copy the .gsb & .gtx files from 
> CMAKE_CURRENT_SOURCE_DIR to CMAKE_CURRENT_BINARY_DIR, that might be 
> touching the mtimes too. See: data/CMakeLists.txt

Seeing how the mtimes are off by two hours, this could be the difference 
between UTC and CEST. The latter was in effect when the archives were 
created:

  $ grep "Made on" debian/datumgrids*.shar
  debian/datumgrids-ch.shar:# Made on 2018-02-26 22:27 CET by <bas at anubis>.
  debian/datumgrids.shar:# Made on 2018-09-15 20:13 CEST by <bas at anubis>.

But why does it only affect the binary GSB & GTX files, and not also the 
binary ntv1_can.dat file or text files like README.DATUMGRID and the 
init files (the ones without extensions)?

Kind Regards,

Bas

-- 
  GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1