[Python-modules-team] Bug#844233: python-passlib: FTBFS with probability of 0.22% due to non-deterministic testsuite

[James Clarke]

Control: retitle -1 python-passlib: FTBFS with probability of 0.46% due to non-deterministic testsuite

On Sun, Nov 13, 2016 at 04:25:30PM +0000, Chris Lamb wrote:
> Whilst working on the Reproducible Builds effort [0], we noticed
> that python-passlib's testsuite will non-determinstically FTBFS:
>
> ======================================================================
> FAIL: test_getrandstr (passlib.tests.test_utils.MiscTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/build/python-passlib-1.6.5/2nd/passlib/tests/test_utils.py", line 129, in test_getrandstr
>     self.assertEqual(sorted(set(x)), [u('a'),u('b'),u('c')])
> AssertionError: Lists differ: [u'a', u'b'] != [u'a', u'b', u'c']
>
> Second list contains 1 additional elements.
> First extra element 2:
> u'c'
>
> - [u'a', u'b']
> + [u'a', u'b', u'c']
> ?            ++++++
>
> This is because of:
>
> 125         x = f(u('abc'), 16)
> 126         y = f(u('abc'), 16)
> 127         self.assertIsInstance(x, unicode)
> 128         self.assertNotEqual(x,y)
> 129         self.assertEqual(sorted(set(x)), [u('a'),u('b'),u('c')])
>
> If the random string ``x`` doesn't contain a certain character, the
> assertion will fail. By my quick calculation this will happen with a
> probability of (2/3)^(16-1).

I believe it's actually (3C2)*(2/3)^16 - (3C1)*(1/3)^16 ~= 0.46% (the
subtracted value is negligible).

> It will also FTBFS if it generates the exact same random string, but
> that seems a little more unlikely.

P = (1/3)^16 = 2.3e-8 or 0.0000023%, which isn't quite as absurdly small
as you might think.

Regards,
James