[pkg-netfilter-team] Bug#903083: Bug#903083: nftables: autopkgtest reports lots of errors but still passes

[Paul Gevers]

Hi Arturo,

On 06-07-18 09:25, Arturo Borrero Gonzalez wrote:
> 1) keep running the tests although they may fail because old kernel
> 2) don't run tests
> 
> Yes, I prefer to have the tests running nevertheless so my choice is 1).
> 
> I'm looking forward to improve this situation in the future.
> 
> Do you have any other idea or hint?

I sure do. Since autopkgtest 5.4 (this week) you have the restriction
"flaky".

'''
flaky
    The test is expected to fail intermittently, and is not suitable for
    gating continuous integration. This indicates a bug in either the
    package under test, a dependency or the test itself, but such bugs
    can be difficult to fix, and it is often difficult to know when the
    bug has been fixed without running the test for a while. If a
    ``flaky`` test succeeds, it will be treated like any other
    successful test, but if it fails it will be treated as though it
    had been skipped.
'''

from:
https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst

Please make the test fail AND mark it flaky.

Paul

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-netfilter-team/attachments/20180706/c85f4410/attachment-0001.sig>