[PKG-Openstack-devel] Bug#834964: testresources: FTBFS too much often (failing tests)

[Santiago Vila]

On Fri, 9 Sep 2016, Thomas Goirand wrote:

> With an up-to-date Sid, I couldn't reproduce this issue. Therefore, I'm
> closing this bug. Feel free to continue discussing it even when it has
> the closed status if you want to continue investigating.

The bug said stretch, and you tried sid, so you didn't really try to
reproduce it. Only because of this, you have already closed the bug
improperly, as you claim unreproducibility without actually testing it
the way it was reported.

If someone reports that "ls -l" segfaults, we don't close the bug
claiming "unreproducibility" after checking that "ls" alone or
"ls --color" does not segfault.

Also, the bug said "FTBFS too much often". If you only tried to build
it once, you didn't try to reproduce it either, as you can't conclude
that the probability of something is zero just by trying it once.

The evidence I showed was my own build log and the build log from
reproducible builds:

https://tests.reproducible-builds.org/debian/rb-pkg/testing/amd64/testresources.html

which also happened in testing at the time I reported this.

My autobuilder and the one from reproducible builds are completely
independent and not related at all, so this may not be a happy coincidence.

We do even use different build systems, they use pbuilder, I use sbuild.

Moreover, the error messages are exactly the same:


FAIL:
testresources.tests.test_optimising_test_suite.TestGraphStuff.testBasicSortTests
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent
call last):
  File
"/build/testresources-1.0.0/testresources/tests/test_optimising_test_suite.py",
line 527, in testBasicSortTests
    "failed with permutation %s" % (permutation,))
      File "/usr/lib/python3/dist-packages/testtools/testcase.py",
line 356, in assertIn
    self.assertThat(haystack, Contains(needle), message)
      File "/usr/lib/python3/dist-packages/testtools/testcase.py",
line 435, in assertThat
    raise mismatch_error
    testtools.matchers._impl.MismatchError: [test_three, test_one,
test_two, test_four] not in [[test_one, test_two, test_three,
test_four], [test_three, test_two, test_one, test_four]]: failed with
permutation [test_one, test_two, test_three, test_four]


And you claim that there is probably not a bug anywhere by closing the
report, despite evidence enough that there is a bug indeed?

I'm attaching three more build logs. Would you reopen the bug now, or
would you prefer that I ask the Release Managers to reopen it for you?

Thanks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testresources_1.0.0-1_amd64-20160909T091418Z.gz
Type: application/gzip
Size: 11527 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/openstack-devel/attachments/20160909/ea50da25/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testresources_1.0.0-1_amd64-20160909T091540Z.gz
Type: application/gzip
Size: 11493 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/openstack-devel/attachments/20160909/ea50da25/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testresources_1.0.0-1_amd64-20160909T091554Z.gz
Type: application/gzip
Size: 11456 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/openstack-devel/attachments/20160909/ea50da25/attachment-0005.bin>