pipeline | WIP: Integrate atomic reprotest into salsa-ci pipeline (!168)

[Santiago Ruano Rincón]

(Replying by mail and CCing reproducible-builds at lists.alioth.debian.org
to get their opinion).

Reproducible builds folks: please see
https://salsa.debian.org/salsa-ci-team/atomic-reprotest/
and the merge request I am working on to include this idea into the
salsa CI pipeline:
https://salsa.debian.org/salsa-ci-team/pipeline/merge_requests/168


Thanks for your comments to both of you, You-Sheng Yang and Raphaël.

El 30/08/19 a las 19:08, You-Sheng Yang escribió:
> 
> 
> 
> You-Sheng Yang commented:
> 
> 
> Firstly, execution of additional jobs to debug a failing automation test is already a bad idea itself. Ideally the reprotest job in the `salsa-ci.yml` should be able to provide/collect all the information available, so that downstream developers don’t have to do anything other than reading the error messages to debug it. So if you find the saved reprotest log is not verbose enough or there are some other runtime object worth saving as artifacts to help debugging, file a MR for that. Or, you can file a MR to teach people how to manipulate the variables to run reprotest in a customized way.

Well, ideally one single reprotest job would let maintainers know what
variation produces reproducibility issues. But I am not aware of any
possible approach other than testing the different variations, and I
don't know if modifying reprotest for that would be feasible (I let
reproducible builds folks answer).

> 
> Secondly, running reprotest with specified variations again won’t give you any new information as they’re still running reprotest defined in this project. You’re wasting time and computing power by running the same thing again. Nothing gained from this MR.

I am not sure that's true. AFAIU, the --auto-build option does the same
thing:

--auto-build
    Automatically perform builds to try to determine which specific
    variations cause unreproducibility, potentially up to and including
    the ones specified by --variations and --vary. ...

I have tried modifying the salsa-ci pipeline reprotest including
--auto-build option, but this easily results in timeouts.

To give you an example of atomic results, see the current salsa ci
pipeline of konsole:

https://salsa.debian.org/qt-kde-team/kde/konsole/-/jobs/279499

and debugging it with atomic-reprotest

https://salsa.debian.org/santiago/atomic-reprotest/pipelines/67690


> 
> Thirdly, `when: on_failure` triggers jobs when *any* of previous stage has failed. So even the reprotest has passed successfully, those jobs defined in your branch will still run if, let’s say, autopkgtest job has failed. This design is problematic.

Yes, that's what I said on the MR description. And that's one of the
reasons why this is work-in-progress.

> 
> -- 
> Reply to this email directly or view it on GitLab: https://salsa.debian.org/salsa-ci-team/pipeline/merge_requests/168#note_106956
> You're receiving this email because of your account on salsa.debian.org.
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20190903/d9e41313/attachment.sig>