Bug#905885: diffoscope: skipping tests on ci.debian.net is perhaps wrong?

[Mattia Rizzolo]

Control: allow to override @skip_unless_tools_exist during tests and fail the test if the tool is missing

On Sun, Aug 12, 2018 at 12:03:50AM +1000, Stuart Prescott wrote:
> *also* diffoscope recommends apktool but tests on ci.d.n do not fail (but 
> should, I think)

what you say here it's half true: the autopkgtest has Depends: all the
recommends that we have.  What is happening is that ci.d.n/autopkgtest
(no clue what piece of the puzzle is responsable for this) is
downloading the test dependencies from unstable if they are not
available on testing, silently.
I believe you may want to get in touch with the CI team for this, I
don't think there is much a package can do here.

> > > Perhaps @tool_required could be extended with an "all tools required"
> > > mode such that no tests are skipped or missing tools are a test failure
> > > (or error). An environment variable would work for setting that mode
> > > and that mode used at package build time and in autopkgtest.
> > 
> > (One difficulty with "just" extending @tool_required is that sometimes
> > we really do want to skip the tests as they require a newer/older
> > version of a particular tool in order that we can correctly match up
> > the output.)

That's an interesting idea, and I'm retitling this bug to do this.
Though what we would need to twak is the @skip_unless_tools_exist
decorator we use in the tests.
Also, we have comparators (and tests) for tools that Debian doesn't
ship, so if we add a flag to make @skip_unless_tools_exist fail a test
instead of skipping it, we would also need to provide a mechanism to
"override the override" and have it really skip those :)

I'm not exactly sure how we should handle the various other
@skip_unless_tool_is_* and friends though...

-- 
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/reproducible-builds/attachments/20180903/e7f0256d/attachment.sig>