[pkg-netfilter-team] Bug#903083: Bug#903083: nftables: autopkgtest reports lots of errors but still passes

[Arturo Borrero Gonzalez]

On 5 July 2018 at 22:26, Paul Gevers <elbrus at debian.org> wrote:
> Source: nftables
> Version: 0.9.0-1
> Severity: normal
> User: debian-ci at lists.debian.org
> Usertags: issue
>
> Dear maintainer,
>
> I just spotted the log of the autopkgtest of nftables. It currently
> passes but the log says it fails. E.g. internaltest-shell.sh ends with
> the message copied below. It says that 20 tests FAILED.
>
> If you want your test to be useful apart from manual inspection I
> suggest you fix your autopkgtest to exit with non-zero value if errors
> occur.
>
> Paul
>
> https://ci.debian.net/data/autopkgtest/testing/amd64/n/nftables/560295/log.gz
>
>  [1A [KW: [FAILED]      ././testcases/transactions/0040set_0: expected 0 but
> got 1
> /dev/stdin:2:1-29: Error: Could not process rule: Device or resource busy
> delete chain ip filter CIn_1
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> I: [EXECUTING]  ././testcases/transactions/0041nat_restore_0
>  [1A [KI: [OK]          ././testcases/transactions/0041nat_restore_0
>
> I: results: [OK] 138 [FAILED] 20 [TOTAL] 158
>

Dear Paul Gevers,

thanks for your interest in the nftables package, is really appreciated.

The nftables userspace program is strictly bounded to a kernel version
when it comes to running the internal tests suites (there are 2, and
soon 3).
Current autopkgtest/debci (as deployed in ci.d.n) doesn't support
running our own virtual machines with a specific kernel version,
therefore some test cases may fail because the kernel.

so we have 2 options:

1) keep running the tests although they may fail because old kernel
2) don't run tests

Yes, I prefer to have the tests running nevertheless so my choice is 1).

I'm looking forward to improve this situation in the future.

Do you have any other idea or hint?