[Debian-salsa-ci] dumat + salsa-ci

[Santiago Ruano Rincón]

Hi!

El 18/09/23 a las 14:25, Luca Boccassi escribió:
> On Sun, 17 Sept 2023 at 20:43, Helmut Grohne <helmut at subdivi.de> wrote:
> >
> > Hi Luca and Santiago,
> >
> > I'm writing to you, because
> >  * Luca expressed interest in helping plug dumat into salsa-ci
> >  * Santiago is doing work for Freexian and has experience with salsa-ci
> >
> > dumat is https://salsa.debian.org/helmutg/dumat.
> >
> > One of the results from the /usr-merge DebConf23 BoF was the wish to add
> > dumat to the salsa CI pipeline. I put up with the groundwork for that
> > now. Can I ask you figure out a reasonable way of integrating this into
> > salsa-ci?
> >
> > Let us assume that you have a foo.changes where the Distribution field
> > is "unstable", "experimental" or a Debian codename. Then you can perform
> > the following steps.
> >
> > Download the current database. This will download about 50MB and create
> > a 1.5GB database file.
> >
> > curl -s https://subdivi.de/~helmut/dumat.sql.zst | zstd -d | sqlite3 dumat.db
> >
> > This file changes every 6h. Would it be possible to have some form of
> > caching such that this is not downloaded for every single pipeline run?

Yes, it is. Also, the images used in the jobs are built daily. A dumat
specific image can included the compressed sql file without issues.

> > It would also be possible to have salsa-ci maintain its own version of
> > this database by regularly updating it locally. The steps to create this
> > database are as follows.
> >
> > sqlite3 dumat.db < schema.sql
> > ./import_mirror.py -d dumat.db
> >
> > That latter step will download very many packages from deb.debian.org on
> > the first invocation. In later invocations, it'll do incremental
> > updates.

Could you please clarify what do you mean with locally?

Do you have any idea about the current size of packages to be
downloaded?

> > Then add the built packages to the database.
> >
> > ./import_mirror.py -d dumat.db --changes path/to/foo.changes
> >
> > This last command changes dumat.db in a way that makes it unsuitable for
> > reusing by other jobs in any way. Then, perform the analysis. This will
> > take about 30 seconds of CPU time.
> >
> > ./analyze.py -d dumat.db > dumat.yaml
> >
> > I think the most reasonable gating here is looking up all the built
> > .debs in the dumat.yaml by their package name and then binary version.
> > Alternatively, filter by source and source version though those fields
> > are optional and only present when they differ from the binary package
> > name and binary version respectively.  If there are any reported issues,
> > consider the job failed.
> >
> > There definitely is one false positive python3-notebook, because it has
> > a file conflict that is mitigated in the maintainer script, but dumat
> > fails to understand that this is correctly mitigated.  I expect more
> > false positives in future, so we need to think about that eventually.
> >
> > I do not recommend adding this to the default pipeline due to the
> > resources consumed by this job. It only makes sense for a tiny fraction
> > of the archive. Maybe we can hard code a list of potentially affected
> > source packages and skip it in all other cases?

Salsa CI cannot control what packages are going to use it. We can add
rules so this job is not run by default. A dumat test could be opt-in.

> >
> > Is this something either of you would like to look into?
> 
> This sounds like a great idea, I can add it to the TODO list. The
> execution time won't be a problem, _maybe_ the temporary disk space?
> But I'm pretty sure many packages need way more than that
> (libreoffice, kernel, browsers), so even that might be fine.

Cheers!

 -- Santiago
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/debian-salsa-ci/attachments/20230919/0e9b679c/attachment.sig>