Release sprint results - team changes, auto-rm and arch status

[Robert Millan]

On 12/01/2014 09:24, Niels Thykier wrote:
> It was filed as serious and then downgraded by Julien on July 9th.
> Indeed, buildd.d.o lists no build problems at all.  So at first glance I
> would expect the tests to have been disabled/ignored.  Assuming this is
> no a simple "error-hiding" tactics, then the bug is non-RC.  If it hides
> user-visible errors, then the severity depends on the (consequences of
> the) errors hidden.
>
> I had no intention of making such an implication.  From what I gathered
> from various sources, glib and GNOME on kFreeBSD had issues and that is
> what I am following up on.

I think it'd be helpful for us to have a more precise list of the RC problems,
or the releasability criteria in general.

Having a clearly defined set of goals makes it much easier for us to allocate
our resources efficiently.

For example, if some source says that "kFreeBSD has issues", I think it would
be easiest for us to handle them if such issues were filed in the BTS. This
ensures that for each issue, we know:

- Its severity

- An accurate description of the problem

- Instructions on how to reproduce it

Lacking that, we can still try to help, but our ability is impaired. For example,
I've been trying to assess the state of GNOME in general by trying to find bugs
myself. I will report my findings soon, however this is clearly not optimal. My
quick "kick the tires" testing is much less reliable than day-to-day usage in
production done by real users.

>> In our experience as porters, we've found that we get lots of testsuite
>> failures (and not just in GNOME). However, often the tests just fail because
>> they're overzealous, or because they make wrong (unportable) assumptions
>> about the underlying APIs.
>>
> 
> The assumption here is that the test is indeed overzealous / not
> relevant to kFreeBSD and have no value as a regression test on kFreeBSD.
>  If that is indeed the case, by all means (have the maintainer) disable
> the test on kFreeBSD and close the bug.

Well, unfortunately we don't know for sure yet. Question is: how much priority
should we give to this kind of issues?

I don't mean to undermine the usefulness of testsuites. I often use them myself
(in fact, we've written some for this port, for example see the one in
kfreebsd-kernel-headers).

However, while as you pointed out, testsuites are very useful to detect
regressions, I think you'll agree that they're not that useful when they're
just more unportable code that we need to fix.

> The concern from my point of view, is if a valid test is disabled when
> it could have been ported.  The last thing I want is to have an RC bug
> appear in testing for a problem the regression test suite from upstream
> could have caught.

Yes, I perfectly understand that. I don't propose to leave testsuites
unattended. I'm just saying that when we try to stop everything when one
test fails, so far this has had counter-productive results.

(I'll ellaborate on this further below, on discussion regarding #734290,
which I think is a very good example of what I'm saying)

>> I believe #628383 would be a good example of what I'm saying. But the problem
>> is very typical. It's not uncommon for us to submit fixes for testsuite bugs
>> rather than having to fix the bugs the tests are supposed to find.
>>
> 
> I believe that is perfectly fine to fix portability issues in the tests
> themselves.

Yes. My point is that if we devote all our time and effort in fixing non-RC problems,
then we're diverting our attention from the actual RC problems which need fixing.

>> [1] If the reason it is RC is that it causes FTBFS (and serious buildd grief),
>>     I think http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734290#10 is a
>>     good solution in that regard.
>>
> 
> This particular kind of resolution would concern me.  Simply disabling
> all tests and ignoring all problems would undermine any value of the
> regression test suite (on non-Linux ports in this case) and allow RC
> bugs to migrate to testing before they are noticed.

Please note that what I'm proposing goes in the opposite direction of what
you want to avoid!

Currently the testsuite in glib2.0 is completely disabled. I haven't
tracked down the reasons why the maintainer disabled it, but it's probably
something like:

1- Some test failed, causing FTBFS. Maintainer filed RC bug.

2- We investigated and found that the test is unlikely to have found an actual
   bug. We give some pointers on how to resolve this.

3- Nobody fixes the test. Maintainer is in a hurry to migrate the package, and
   disables the testsuite on kfreebsd.

My proposal is to:

1- Re-enable the testsuite, without making failures result in FTBFS.

2- Avoid test hangs completely by running them with timeout(1). This preserves
   the exit status so we aren't losing any information.

> With my release hat on, I want as many problems stopped before they
> reach testing.  Having a build-time regression test suites from upstream
> is certainly valuable in this regard and we currently have no
> replacement for testing the actual program itself.

I think a good step in that direction would be to ensure that nobody, ever,
disables a testsuite on kfreebsd. Many packages run a testsuite without making
100% completion mandatory (I think of GCC as an example). I think it would
be reasonable to accept this in cases where they've proven to be more of a
problem than a solution?

-- 
Robert Millan