[Pkg-javascript-devel] Rebuilding of nodejs reverse dependencies

[Paolo Greppi]

Hi Jérémy,

Il 17/03/22 13:26, Jérémy Lal ha scritto:
> ...
> 
>     So I ended up installing a gitlab-runner with nspawn-runner and a
>     semi-active server I maintain.
>     I documented the process here:
>     https://wiki.debian.org/Salsa/Doc/CustomRunners/SystemdNspawnRunner
>     <https://wiki.debian.org/Salsa/Doc/CustomRunners/SystemdNspawnRunner>
> 
>     salsa's driving that runner to rebuild the thousands of packages now.
>     Minimal build time is 1 minute, so it should take about two days to
>     rebuild the whole.
> 
>     After that, anyone knowing gitlab API and bugs.debian.org
>     <http://bugs.debian.org> API could help to open FTBFS bugs ?
> 
> I started this:
> https://salsa.debian.org/kapouer/dance 
> <https://salsa.debian.org/kapouer/dance>
> 
> to fetch logs of failed jobs, and also to be able to retry jobs that 
> failed because of external reasons
> (runner configuration issue, for example).
> When all jobs are finished, i'll start to use "dance" to send bug reports.
> 
> Jérémy

this is all super interesting. In the past I have done quite a few mass 
rebuilds of reverse dependencies using ratt 
(https://salsa.debian.org/debian/doxygen/-/wikis/ratt_doxygen_1.9.1-1_amd64%20partial%202)

Your approach is cool because it can be easier to automate.

To look at pipeline results programmatically I would have used this 
Python library:
- https://packages.debian.org/bullseye/python3-gitlab
- 
https://python-gitlab.readthedocs.io/en/stable/gl_objects/pipelines_and_jobs.html

But I see you plan to use JS for "dance".

Do you intend to trigger the dance from within the pipeline ? You'd then 
have a recursive pipeline ... who knows if it will finish in finite time ;-)

Anyway from my experience:

1. some rdeps reproducibly fail all the time, these should be excluded

2. ratt rebuilds failed rdeps against the previous version of the parent 
package (this filters out false positives), does your job definition do 
that as well ? (too lazy to check)

3. some rdeps take much longer to build than others, I ended up creating 
lists of "fast" and "slow" rdeps; if you only include "fast" ones your 
mass rebuild can run in hours instead of days, but may still be able to 
catch regressions

4. I used to create spreadsheets to compare build times and which rdep 
failed; for max fun it would be nice to have a database (!) where to 
store (stamp, parent_package, old_version, new_version, rdep, status, 
duration)

Paolo