[Debian-salsa-ci] Release team to utilize Salsa CI for Forky quality assurance?

[Sebastian Ramacher]

On 2026-01-21 18:41:49 -0800, Otto Kekäläinen wrote:
> Hi,
> 
> I was hoping Release Team members would comment on the idea of using
> current vcswatch information to accelerate or slow down
> unstable->testing migrations and thus postponed discussing Salsa CI
> details to avoid derailing the discussions from what rules to have for
> migrations.

Release Team members did comment. Please check the thread.

Cheers

> 
> Seems that discussion is over, so I will proceed to answer Adrian's
> comment about Salsa CI now.
> 
> ...
> > > It is not about the specific tests in Salsa CI, but merely the fact
> > > that some maintainers are trying to be diligent and test their
> > > packages properly _before_ upload, and some just upload and skip doing
> > > proper QA activities. As I wrote at the end of my question I was
> > > wondering if the Release Team wants to promote such a culture.
> >
> > I am not sure there is a clear distinction between "diligent" and
> > "waste of time" here.
> 
> Initially there is some work to enable Salsa CI and most importantly
> fix all issues in the package so that the CI passes, but once done,
> the amount of work needed to run Salsa CI is minimal. You simply push
> to Salsa and wait to see if you get an email that the pipeline
> regressed or not. Or you can check it via Salsa in a browser or via
> one of the command-line tools.
> 
> There are numerous examples of when Salsa CI caught issues that
> maintainers fixed before upload, and there are numerous examples of
> packages breaking Debian unstable in vain due to issues that could
> easily have been detected with Salsa CI.
> 
> To take just one example,
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071245 describes a
> version of gnupg2 was in an uninstallable state, and stopping the work
> of everyone trying to build packages that depend on gnupg2. Testing
> the final commit by the maintainer before upload would have certainly
> been diligent and not require much extra time. The amount of 'wasted
> time' for everyone else getting their work disrupted by a broken
> Debian unstable was significant.
> 
> The maintainer started using Salsa CI and that same package has not
> caused Debian-wide breakage since. Note that the maintainer was also
> doing frequent uploads to Debian unstable to get QA results. After
> adopting Salsa CI and merging
> https://salsa.debian.org/debian/gnupg2/-/merge_requests/16 the
> maintainer no longer had the need to upload to unstable just to test,
> but got the feedback immediately in Salsa for every git push.
> 
> > The proper QA happens after uploading to unstable (or experimental),
> > and whether having more than one CI in the workflow makes sense is
> > something that should IMHO be left to the maintainer.
> 
> Looking at the 25 packages you are the primary maintainer for
> (https://udd.debian.org/dmd/?email1=bunk%40debian.org&nouploader1=on&nosponsor1=on&email2=&email3=&packages=&ignpackages=&format=html#todo),
> I see none of them is using Salsa CI, and actually not even hosted in
> Salsa at all.
> 
> Thus if the Release Team took my suggestions on how to use the
> vcswatch results it would not have any effect on your packages as they
> would never be out of sync in git or have any CI results to use for
> the rules.
> 
> > > > > Are there any new quality assurance tools you would like the Salsa CI
> > > > > pipeline to run by default on all Debian packages in an effort to
> > > > > prompt maintainers to ensure those tests pass?
> > > >
> > > > AFAIK your "on all Debian packages" is factually incorrect, with
> > > > resource limits in the Salsa CI preventing the build of many packages.
> > >
> > > I am aware only of qt6-webengine not being able to build on Salsa
> > > (https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/506).
> >
> > If there is a ranking of the packages with the largest build times on
> > amd64 somewhere, I doubt qt6-webengine would make it into the TOP 100.
> >
> > webkit2gtk, LLVM, GCC, OpenJDK, acl2 - that's just random packages
> > coming into my mind with much larger build times.
> 
> I checked these and none of them have a single pipeline run yet,
> meaning they didn't even try to use Salsa CI:
> - https://salsa.debian.org/webkit-team/webkit/-/pipelines
> - https://salsa.debian.org/pkg-llvm-team/llvm-defaults/-/pipelines
> - https://salsa.debian.org/openjdk-team/openjdk/-/pipelines
> 
> This package has pipelines disabled:
> - https://salsa.debian.org/toolchain-team/gcc
> 
> This isn't hosted on Salsa or git at all:
> - https://tracker.debian.org/pkg/acl2
> 
> If these maintainers actually want to use Salsa CI, I am happy to help
> them get it running. There are large packages like the Linux kernel
> and Godot that use Salsa CI, so it is possible for those who want to.
> 
> > > I also
> > > assume the resource issues are easy to solve if enough people report
> > > them and ask them to be solved, as the limitations are in policies and
> > > not in software or pipeline itself.
> >
> > My guesstimate would be that your "easy to solve" might end up being at
> > least an order of magnitude more CPU power.
> >
> > RAM and disk space are also issues, you could not build all packages
> > on a worker that has only 50 GB disk space.
> >
> > Speed also matters here, a maintainer might not want to wait a week for
> > Salsa CI results and then another week for complete britney CI results.
> 
> Sure, there may be some package that can't run on Salsa CI, but also
> your description above seems to have a lot of exaggeration like "wait
> a week for Salsa CI results" and considering that none of your
> packages use Salsa CI I suspect you are not actually asking for help
> on how to get Salsa CI running, but just wanting to emphasize that not
> all packages use Salsa CI.
> 
> I don't think using the vcswatch results depends on all packages being
> in git or Salsa or using Salsa CI.
> 
> > >...
> > > > If something has provably regressed, this should already block testing
> > > > migration due to failing also in the britney CI.
> > >
> > > Again, it's not about the individual jobs. Any maintainer can anyway
> > > customize their salsa-ci.yml file. This is about the culture of
> > > testing changes in CI _before_ upload, and about maintaining the
> > > ability to see from the CI which commit or merge request caused the
> > > regression. Having this capability and culture most likely leads to
> > > higher quality and fewer regressions to fix at release time.
> > >...
> >
> > I do not understand this claim.
> >
> > Anything automated CI can find should already be found by the britney CI
> > immediately and block testing migration.
> 
> Britney is not checking the same things as vcswatch.
> 
> Quoting my own rules proposal here for reference:
> 
> On Sat, 3 Jan 2026 at 22:00, Otto Kekäläinen wrote:
> ..
> > The vcswatch tool is tracking the CI pipeline status on Salsa for all
> > packages that have a Vcs-Git URL pointing to Salsa. There could be for
> > example a rule that if a package is hosted on Salsa, and if the git
> > repo is up-to-date with what was uploaded and CI is passing, the
> > package could migrate faster from unstable->testing. Alternatively
> > there could be a negative rule that adds extra delay to any package
> > that is not in sync in git or has no CI or a failing CI. I guess one
> > could also justify a ruleset where packages with no CI are considered
> > neutral, but if there is a CI and it is failing, the package would get
> > extra delay or not migrate from unstable to testing at all as there is
> > something that provably regressed.
> 

-- 
Sebastian Ramacher