[Pkg-e-devel] Disabling EFL's test suite

[Ross Vandegrift]

Hi all,

Most of my packaging effort goes to making the EFL test suite happy
enough to pass.  For reasons outlined below, this normally involves
finding tests to disable.  I'd like to improve the situation, but I
don't know what to do.  I'm considering disabling upstream tests
entirely.

I'd love to be talked out of this.  But I'd need some help making the
test suite mostly hands-off on upgrade.  The main challenges are:


1) Upstream assumes tests are run in a live desktop session, as a
unprivilieged user, with system & session dbus available, and live
network access.  Some of these assumptions fail in various build
environments, so many tests must be disabled.


2) Since the packages violate upstream's testing assumptions, new
releases usually include new breakage.  Tracking that down is difficult
& time consuming.


3) Failures are often non-deterministic or arch specific.  Worse,
sbuild, pbuilder, and none-of-the-above often experience different
failures.  Fixing one environment sometimes breaks another.


4) I've never seen EFL fail to run enlightenment & terminology, even if
the test suite fails.  Since that's the use-case I care about, I don't
get any value for the time I put in.


Thanks for any suggestions,
Ross