[DRE-maint] Bug#935413: ruby2.5: flaky autopkgtest: base dir expected vs actual mismatch

[Paul Gevers]

Source: ruby2.5
Version: 2.5.5-3
Severity: serious
Tags: sid bullseye buster
X-Debbugs-CC: debian-ci at lists.debian.org
User: debian-ci at lists.debian.org
Usertags: flaky

Dear maintainers,

With a recent upload of ncurses the autopkgtest of ruby2.5 fails in
stable when that autopkgtest is run with the binary packages of ncurses
from stable-proposed-updates. The same happened in testing. However,
when the test was retried, the test passed. I looked into the history of
your autopkgtest and it fails once in a while with the same error every
time (as far as I checked).

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are wasting peoples time. In this case, it also wasted time of the
stable release managers due to the failure in stable. Please either fix
the test to be more robust, or use the "flaky" restriction for the
offending test.

I copied some of the output at the bottom of this report.

I'll have the migration software ignore the results of your autopkgtest
until this bug is fixed.

Paul

https://ci.debian.net/data/autopkgtest/testing/amd64/r/ruby2.5/2332519/log.gz

  1) Failure:
TestGemSpecification#test_base_dir_default
[/tmp/autopkgtest-lxc.a80kp91c/downtmp/autopkgtest_tmp/test/rubygems/test_gem_specification.rb:1947]:
--- expected
+++ actual
@@ -1 +1 @@
-"/var/lib/gems/2.5.0"
+"/tmp/test_rubygems_17185/default"

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-ruby-extras-maintainers/attachments/20190822/985b6892/attachment.sig>