[Pkg-emacsen-addons] Bug#818142: flycheck: FTBFS in testing (failing tests)

[Sean Whitton]

Hello,

Thanks again for your feedback.

On Tue, Aug 23, 2016 at 11:30:07AM +0200, Santiago Vila wrote:
> On Mon, 22 Aug 2016, Sean Whitton wrote:
> 
> > 1. flycheck tends to FTBFS because the test suite contains many tests
> > which encode expected output from ~30 command line tools.  When one of
> > those tools sees a new upstream version which tweaks its output, the
> > test breaks.
> 
> Perhaps encoding the expected output (to the letter) is not a good
> idea, then. Assuming those are standard Unix tools, they usually follow
> some standard, which tells them how the output should be. A well
> designed test should be prepared to check any output which follow this
> standard, not the exact stream of bytes produced by a certain fixed
> version.

I don't think any of them would count as "standard unix tools": they're
language-specific linters whose output does not follow any defined
standard.

I don't personally have any experience designing a test suite.  If you
can see some better way to ensure that flycheck is working as intended,
I'd encourage you to start a discussion on upstream's bug tracker[1] --
if there's a better way to test flycheck, I'd like to know what it would
look like.

> > 2. The test suite also fails under certain autobuilder setups.  In
> > particular, the reproducible builds team's automated builders fail
> > with mysterious permission errors.  I think it is something to do
> > with their tmpfs setup -- though I use tmpfs with sbuild and can't
> > reproduce the error.
> 
> We are working with computers, which are designed to do that we tell
> them to do. There are not really "mysterious errors", only errors
> which nobody has investigated yet.
> 
> In particular, a build failure is usually one of two things: Either a
> bug in the program (even if it does not always happen), or a bug in
> the building software (I use sbuild, they use pbuilder).
> 
> So, I would encourage you to contact them to see if there are any bugs
> in pbuilder (or their setup in general).

Yes, I've been in touch with them (in relation to a different package of
mine whose test suite was failing in a similar way), but we haven't got
to the bottom of it.[2]

[1] https://github.com/flycheck/flycheck/issues
[2] http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20160425/005352.html

-- 
Sean Whitton
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-emacsen-addons/attachments/20160823/384e395a/attachment.sig>