[Freedombox-discuss] Heavy tests

[Jonas Smedegaard]

Quoting Sunil Mohan Adapa (2021-09-11 04:07:41)
> On 10/09/21 4:02 pm, Fioddor Superconcentrado wrote:
> > I've coded some tests that result too heavy (slow) for unit testing. 
> > But I think they still provide value in specific cases.
> > 
> > So I've thought of renaming the file as *heavy_test_*.py*. This will 
> > make default calls skip them but they can still be run explicitly 
> > whenever the developer/tester wants to extend the coverage or the 
> > reliability of the tests.
> > 
> > I'd like to know what other developers/testers think of this 
> > approach.
> > 
> > If we do this for other modules and components we might create a 
> > subfolder 'heavy_tests' or similar to collect separate them from 
> > regular *tests_*.py* files.
> > 
> 
> It might be preferable to use pytest's "marks" to mark heavy tests 
> (see WordPress' functional tests for example on how to mark). Then, we 
> can introduce a '--include-heavy' option to include these tests (see 
> /conftest.py for example on how --include-functional is done). This 
> method gives us the flexibility organize tests via modules as we 
> currently do and easily mark them as heavy to avoid them during 
> regular runs.

I recommend to reuse the mechanism invented by Perl modules but in no 
way specific to that: Check for environment variable EXTENDED_TESTING

Here is a concrete example of implementing EXTENDED_TESTING hinting in a 
Python package: 
https://salsa.debian.org/debian/python-aiortc/-/blob/debian/master/debian/patches/1001_skip_network_tests.patch

And here an example of how it can later be used: 
https://salsa.debian.org/debian/python-aiortc/-/blob/debian/master/debian/tests/control

Here is an example of proposing the mechanism to an upstream: 
https://github.com/etesync/etebase-py/issues/19

Her is the formal definition for the mechanism: 
https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: signature
URL: <http://alioth-lists.debian.net/pipermail/freedombox-discuss/attachments/20210911/d92d1ffe/attachment.sig>