[Debian-ha-maintainers] Bug#994794: booth: intermittent autopkgtest failures

[Simon McVittie]

Source: booth
Version: 1.0-237-gdd88847-2
Severity: serious
Justification: https://release.debian.org/testing/rc_policy.txt 6a
X-Debbugs-Cc: debian-ci at lists.debian.org
User: debian-ci at lists.debian.org
Usertags: flaky

While looking at glib2.0 migration from unstable to testing, I noticed
that migration was blocked by the src:booth autopkgtest failing with the
proposed glib2.0 version. Looking at the logs, it seems that this autopkgtest
is "flaky" (intermittently failing): it flips between passing and failing
without changes to the list of installed packages.

For example, the test run at 2021-09-20 15:36:40 UTC in [1] failed,
but when retried with the same package-set at 2021-09-20 22:55:06 UTC, it
passed.

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are going to cause extra work for people unrelated to this specific package.

If this test cannot be made sufficiently reliable to be used for CI gating,
one option is to mark it with "Restrictions: flaky" in debian/tests/control
so that its failures are not considered to be a serious regression.

Thanks,
    smcv