Bug#839580: request-tracker4: FTBFS in testing (failed tests)

Wed Oct 12 03:48:15 UTC 2016

Hi all--

On Tue 2016-10-11 16:44:28 -0400, gregor herrmann wrote:
> On Mon, 10 Oct 2016 23:10:30 +0300, Niko Tyni wrote:
>
>> Help untangling this mess would be very welcome, as both Dominic and I
>> are rather short on time. Tagging and copying possible candidates :)
>
> Good news: I got the package to build.
> Bad news: Only by forcing gpg1 even harder in the test suite.

I tried with your series of patches.  Thanks for doing this cleanup
work, Gregor!

> For gpg2 I guess we'd need a fake pinentry and maybe also changes to
> the secret key layout etc. -- Material for dkg maybe :)
> lib/RT/Test/GnuPG.pm looks interesting for setting the gpg config,
> lib/RT/Test.pm has import_gnupg_key.

Fake pinentry would probably be useful for the tests, but perhaps given
the number of tests in RT that hard-code passphrases and pass them to
GnuPG::Interface, it would be better to pass them in differently.

I've just done some more work on GnuPG::Interface to get it to work with
programmatically-passed passwords as well (that is, without a
fake-pinentry.pl), since it looks like tools like RT aren't going to
abandon that use pattern any time soon.

Those patches are posted here:

 https://github.com/bestpractical/gnupg-interface/pull/1

I'd like to upload them to debian as well, though they are *not*
interoperable with versions of GnuPG prior to 2.1.x, due to GnuPG's
non-portable interface :/ I've pushed a proposed upload to experimental
to the move-to-modern-gnupg branch on
https://anonscm.debian.org/git/pkg-perl/packages/libgnupg-interface-perl.git

(see also discussion around this portability issue here:
https://lists.gnupg.org/pipermail/gnupg-devel/2016-October/031800.html)

sigh...

Any thoughts on the right thing to do with libgnupg-interface-perl?

> What I don't know is if RT4 in the current state now works with
> gpg2 when the test suite passes with gpg1 ...
> Maybe forcing it to gpg1 (in etc/RT_Config.pm.in and debian/control)
> would buy some time. But in general I guess we want gpg2 both at
> buildtime and runtime.

Yes, we really ultimately do want to use the modern branch of GnuPG at
both buildtime and runtime, but even with the changes i've made to
GnuPG::Interface, i'm seeing a lot of failed tests, possibly due to
an overly-brittle test suite that attempts to match strings it should
not be matching.

It's also possible that upstream GnuPG has changed more significant
behavior across versions, though, so i'm trying to track that down :/

I've just re-run the test suite to get a summary of the additional
failures, which are included below.

> Before getting that far, I had to make some other changes to reduce
> the build failures to the gpg related ones.
>
> After the package built I also fixed some lintian errors :)

all this cleanup looked sensible to me, though i don't know how to apply
this tarball-of-patches in "the git-dpm way", which appears to be how
request-tracker4 is currently maintained.

below is the summary of the failing tests when used with GnuPG from the
modern suite.  I think i can clean up several of these by explicitly
ignoring GnuPG status lines KEY_CONSIDERED and NEWSIG and FAILURE, but
there will still be some that remain.  I'm particularly concerned about
the "Bad plan" response in t/mail/gnupg-incoming.t, which seems to
trigger a separate gpg-agent inocation.

Test Summary Report
-------------------
t/mail/gnupg-reverification.t                        (Wstat: 4096 Tests: 204 Failed: 16)
  Failed tests:  16, 28, 40, 52, 64, 76-77, 89-90, 102, 114
                126, 138, 150-151, 163
  Non-zero exit status: 16
t/web/gnupg-select-keys-on-update.t                  (Wstat: 768 Tests: 87 Failed: 3)
  Failed tests:  12, 22-23
  Non-zero exit status: 3
t/web/gnupg-select-keys-on-create.t                  (Wstat: 1280 Tests: 80 Failed: 5)
  Failed tests:  9, 18-21
  Non-zero exit status: 5
t/web/crypt-gnupg.t                                  (Wstat: 2816 Tests: 103 Failed: 11)
  Failed tests:  63, 66, 73, 75-77, 82-84, 100-101
  Non-zero exit status: 11
t/mail/gnupg-incoming.t                              (Wstat: 512 Tests: 18 Failed: 3)
  Failed tests:  15, 17-18
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 54 tests but ran 18.
t/mail/crypt-gnupg.t                                 (Wstat: 5632 Tests: 101 Failed: 22)
  Failed tests:  5-6, 8-10, 22-24, 26-30, 32-33, 44, 51-52
                54, 56-57, 101
  Non-zero exit status: 22
t/web/admin_user.t                                   (Wstat: 256 Tests: 26 Failed: 1)
  Failed test:  11
  Non-zero exit status: 1
t/crypt/no-signer-address.t                          (Wstat: 512 Tests: 6 Failed: 2)
  Failed tests:  4-5
  Non-zero exit status: 2
t/security/CVE-2012-4735-incoming-encryption-header.t (Wstat: 256 Tests: 12 Failed: 1)
  Failed test:  7
  Non-zero exit status: 1
Files=362, Tests=25577, 767 wallclock secs ( 5.20 usr  0.78 sys + 1173.26 cusr 82.37 csys = 1261.61 CPU)
Result: FAIL

Most of these failures seem to have to do with one of:

 a) assumptions that t/data/gnupg/keyrings is a normal GnuPG home
    directory (rather than doing a manual import of the keys there), or

 b) a missing "PasswordCheck" response (which 2.1.x might not provide), or

 c) too-strict tests of specific error messages or warnings.

But not all of them do.

One particularly heinous test ('first key shows up in preferences' at
t/web/crypt-gnupg.t line 407) prints 5563 lines of HTML+javascript and
then says it "doesn't match
'(?^s:75E156271DCCF02DDD4A7A8CDF651FA0632C4F50.*?EC1E81E7DC3DB42788FB0E4E9FA662C06DE22FC2)'"

Does any of you RT hackers know how to run specific tests manually
without needing to re-run the entire ~10 minute suite so i can reduce
the cycle time as i'm debugging this stuff and trying to propose fixes?

    --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 930 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-perl-maintainers/attachments/20161011/3c346974/attachment.sig>