[Pkg-privacy-maintainers] GUI tests for MAT

[Sascha Steinbiss]

Hi intrigeri,

thanks for your feedback!

> I'm not familiar with the technologies you used there, but after
> researching this topic independently, I also picked dogtail for
> another project. I'm glad we chose the same :)

:)
The current state of GUI testing on the Linux desktop is a bit sad, if you ask me. LDTP seems to be dead in the water and dogtail looks like it was only recently revived and under regular development again. Did you evaluate any other alternatives?

>> The test suite currently passes in my container (adt-virt-lxc), it couldn’t hurt to
>> try on other testbeds too. Anyway, I would be happy to receive some feedback :)
> 
> The tests pass for me in adt-virt-qemu.
> 
> I'm not merging this branch as-is into master since I have a few
> questions, and at least one hard-coded sleep time made me slightly
> uncomfortable.

Yes, I see what you mean. My solution was mostly a trade-off between time and beauty, but since these tests do not have deadlines it couldn’t hurt giving them a second look.

> So, while reading the code, I took some notes about
> bits I was not sure about. Here they are in the form of "XXX" comments
> inline:
> 
> --- a/debian/tests/check-mat
> +++ b/debian/tests/check-mat
> @@ -52,6 +52,7 @@ class TestMatGUIFunctional(unittest.TestCase):
>         self.add = self.rootapp.child(roleName="tool bar").child(name="Add")
>         self.clean = self.rootapp.child(
>             roleName="tool bar").child(name="Clean")
> +        # XXX: Why do we have to do this by hand?

Maybe we don’t but I had at least one case where window focus was lost and the test hung.

>         self.mat_window_title = 'Metadata Anonymisation Toolkit'
>         focus.application("mat-gui")
>         focus.frame(self.mat_window_title)
> @@ -82,6 +83,8 @@ class TestMatGUIFunctional(unittest.TestCase):
>         Leave MAT via menu option.
>         """
>         self.rootapp.child(name="File", roleName="menu").click()
> +        # XXX: do we have a guarantee that this is not racy,
> +        # that is, that there is a "Quit" menu item at this point?

dogtail uses an iterative backoff/retry approach to take the edge off typical timing issues.
See https://fedorapeople.org/~vhumpa/dogtail/epydoc/dogtail.tree-module.html ->

“ […] Dogtail implements a high-level searching system, for finding a node (or nodes) satisfying whatever criteria you are interested in. It does this with a 'backoff and retry' algorithm. This fixes most timing problems e.g. when a dialog is in the process of opening but hasn't yet done so.
If a search fails, it waits 'config.searchBackoffDuration' seconds, and then tries again, repeatedly. After several failed attempts (determined by config.searchWarningThreshold) it will start sending warnings about the search to the debug log. If it still can't succeed after 'config.searchCutoffCount' attempts, it raises an exception containing details of the search. […]”

>         self.rootapp.child(name="Quit", roleName="menu item").click()
> 
>     def test_start_no_warnings(self):
> @@ -117,6 +120,7 @@ class TestMatGUIFunctional(unittest.TestCase):
>         window.child("mat").click()
>         window.child(self.tmpdir).click()
>         window.child("OK").click()
> +        # XXX: can't we explicitly wait for the $something we need to wait for?

Yes I guess we can and it is safer of course.  Let me see what I can do. I guess one can check whether the list of loaded files to clean contains the expected number of items. 
I will take a look when I find the time (currently working on some reproducibility issues elsewhere and also my NM process is about to start).

Thanks anyway for your careful review! I will ping the list when I have made some progress on the points you raised.

Cheers
Sascha