Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?

[Stuart Prescott]

Package: diffoscope
Version: 99
Severity: normal

Dear Maintainers,

diffoscope recommends apktool but apktool is not currently in buster. The
test suite of diffoscope currently passes without error because the tests
that use apktool are decorated with @tool_required('apktool').

Consequently:

* the code paths that use apktool are not being tested in buster

* these tests are not actually passing but ci.debian.net is saying that
  the test suite has passed (skipping tests to hide errors isn't the
  same as a test passing)

There is clear value in being able to skip tests for the casual user on
another platform or when not interested in diffoscope's abilities in
file formats about which one does not care. For the Debian package,
however, I think every functionality of the Debian package should be
tested in autopkgtest. The current situation does not pick up that apktool
is not installable in buster, for instance, and would not pick up
something like /usr/bin/apktool being renamed.

Perhaps @tool_required could be extended with an "all tools required"
mode such that no tests are skipped or missing tools are a test failure
(or error). An environment variable would work for setting that mode
and that mode used at package build time and in autopkgtest.

BTW other latent bugs here:

* diffoscope has an unsatisfiable recommends

* diffoscope cannot be built in buster (unsatisfiable build-deps,
  this is RC)

cheers
Stuart