[Python-modules-team] Bug#928042: pikepdf: flaky autopkgtest: random input results in failing test sometimes

[Paul Gevers]

Source: pikepdf
Version: 1.0.5+dfsg-2
Severity: important
X-Debbugs-CC: debian-ci at lists.debian.org
User: debian-ci at lists.debian.org
Usertags: flaky

Dear maintainers,

With a recent upload of glibc the autopkgtest of pikepdf fails in
testing when that autopkgtest is run with the binary packages of glibc
from unstable. However, when I triggered a retry, the test passed. I
looked into the history of your autopkgtest and it fails once in a while
with the same error every time (as far as I checked). The failing test
*seems* to be using some random input. I suspect there are possible
inputs that result in the failure.

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are wasting peoples time. Please either fix the test to be more robust,
or mark the test as "flaky".

I copied some of the output at the bottom of this report.

Paul

https://ci.debian.net/data/autopkgtest/testing/amd64/p/pikepdf/2299297/log.gz

=================================== FAILURES
===================================
______________________________ test_random_dates
_______________________________

    @given(
>       integers(-9999, 9999),
        integers(0, 99),
        integers(0, 99),
        integers(0, 99),
        integers(0, 99),
        integers(0, 99),
    )
    def test_random_dates(year, month, day, hour, mins, sec):

tests/test_metadata.py:248:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
/usr/lib/python3/dist-packages/hypothesis/core.py:519: in execute
    result = self.test_runner(data, run)
/usr/lib/python3/dist-packages/hypothesis/executors.py:58: in
default_new_style_executor
    return function(data)
/usr/lib/python3/dist-packages/hypothesis/core.py:517: in run
    return test(*args, **kwargs)
tests/test_metadata.py:248: in test_random_dates
    integers(-9999, 9999),
/usr/lib/python3/dist-packages/hypothesis/core.py:464: in test
    result = self.test(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _

year = 1, month = 1, day = 1, hour = 0, mins = 0, sec = 0

    @given(
        integers(-9999, 9999),
        integers(0, 99),
        integers(0, 99),
        integers(0, 99),
        integers(0, 99),
        integers(0, 99),
    )
    def test_random_dates(year, month, day, hour, mins, sec):
        date_args = year, month, day, hour, mins, sec
        xmp = '{:04d}-{:02d}-{:02d}T{:02d}:{:02d}:{:02d}'.format(*date_args)
        docinfo = '{:04d}{:02d}{:02d}{:02d}{:02d}{:02d}'.format(*date_args)

        try:
            converted = DateConverter.docinfo_from_xmp(xmp)
        except ValueError:
            pass
        else:
>           assert converted == docinfo
E           AssertionError: assert '10101000000' == '00010101000000'
E             - 10101000000
E             + 00010101000000
E             ? +++

tests/test_metadata.py:265: AssertionError
---------------------------------- Hypothesis
----------------------------------
Falsifying example: test_random_dates(year=1, month=1, day=1, hour=0,
mins=0, sec=0)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/python-modules-team/attachments/20190426/078ee185/attachment.sig>