Bug#863915: unblock: webkit2gtk/2.16.3-2

[Carlos Alberto Lopez Perez]

On 02/06/17 14:47, Carlos Alberto Lopez Perez wrote:
>> Also it'd be nice to know what kind of automated testing is happening. I know
>> WebKit has an extensive test suite (including layout tests) that upstream
>> continuously runs for development series. I don't know if that's the same for
>> stable series though. Also can we enable the test suite in Debian?
>>
> On trunk (master) we have a extensive test coverage per each commit.
> We even have a bot testing that WebKitGTK+ always remains build-able
> both on Debian stable and Ubuntu LTS. Check our bots here:
> http://build.webkit.org/waterfall?category=GTK
> 
> All those bots are running Debian 9, except for the 32-bit and ARM bot
> that are running Debian 8 but they will be eventually upgraded to Debian
> 9 also.
> 
> (Also except the Debian and Ubuntu bots that are of course running
> Debian 8 and Ubuntu 16.04)
> 
> For the stable branch (that is what you are interested about I guess) we
> have a bot running all those tests here:
> 
> https://build-webkit.igalia.com/waterfall?category=GTK
> 
> This bot running the tests on the stable branch is currently running
> Debian 8 as base distro.
> We are happy to upgrade it to Debian 9 if you wish (is something we will
> eventually do in any case).
> 
> Note that those 9 failures are not really meaningful.
> Current layout test suite of WebKit has 47556 tests
> 
> If there is some more testing we can do to accommodate your needs, we
> will be happy to help.
> 
> Regards.
> 
> 

One note more:

The results of the layout tests are highly dependent on very specific
versions of the libraries we depend on. For example, different versions
of GTK+ or Cairo produce different results (this are usually small pixel
differences on the render of the webview).

So what we do, is to have the expected results of those tests specially
created for very specific versions of those libraries that we build with
JHBuild and we use it for testing.

If you try to run the layout test suite without using our internal
JHBuild you are going to get thousands of tests failures due to this
(they are not real failures, but the tooling is unable to detect that as
it can't know if a 1-pixel difference is because of a real failure or
because your version of GTK+ draws arrows/boxes/etc in a sightly
different way than the one was used to produce the expected result)

So, I don't think is flexible for Debian to run our layout test suite
unless you are willing to maintain your own list of expected results.

There are however other tests that you can run that don't depend on the
version of the libraries used like the jscore-tests or the API tests.
However, the tarballs we release are trimmed to remove all those testing
stuff from them. You would have to use the SVN repository for that,
which complicates things.


However, we are happy to keep our bot that runs all those tests for the
stable branch on Debian.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 897 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-webkit-maintainers/attachments/20170602/b398cc1f/attachment.sig>