Bug#1017402: scikit-learn: autopkgtest times out on powerful hosts

[Paul Gevers]

Source: scikit-learn
Version: 0.23.2-5
Severity: serious
User: debian-ci at lists.debian.org
Usertags: flaky timeout

Dear maintainer(s),

I looked at the results of the autopkgtest of you package because of the 
recent issues we had with the package not migrating. I had already put 
it on the ci.d.n reject list for amd64, armel and armhf a while ago 
intending to file this bug already earlier. Today I removed the block 
for amd64 to give the fresh upload a try. However, the autopkgtest fails 
when run on ci-worker13, which is our powerful amd64 worker. That worker 
has 64 cores and 256GB RAM, while the other amd64 workers have only 2 
cores and 8GB RAM. Also our armel and armhf workers are powerful 16 
resp. 160 cores and 26GB resp. 511 GB RAM. I have copied an example of a 
failing test below, but note that not all timeouts happen on the same 
location.

Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.

On top of that, when a test just hangs that's not good for our 
infrastructure.

Don't hesitate to reach out if you need help and some more information
from our infrastructure.

Paul

https://ci.debian.net/data/autopkgtest/unstable/amd64/s/scikit-learn/24826845/log.gz

Fit 150 trees in 561.863 s, (740 total leaves)
Time spent computing histograms: 152.240s
Time spent finding best splits:  75.432s
Time spent applying splits:      202.079s
Time spent predicting:           18.902s
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_early_stopping_default[HistGradientBoostingClassifier-X0-y0] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_early_stopping_default[HistGradientBoostingClassifier-X1-y1] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_early_stopping_default[HistGradientBoostingRegressor-X2-y2] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_early_stopping_default[HistGradientBoostingRegressor-X3-y3] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores0-1-0.001-False] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores1-5-0.001-False] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores2-5-0.001-False] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores3-5-0.001-False] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores4-5-0.0-False] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores5-5-0.999-False] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores6-5-4.99999-False] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores7-5-0.0-True] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores8-5-0.001-True] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_should_stop[scores9-5-5-True] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_absolute_error 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_absolute_error_sample_weight 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_asymmetric_error[0.2] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_asymmetric_error[0.5] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_asymmetric_error[0.8] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_poisson_y_positive[y0] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_poisson_y_positive[y1] 
PASSED
../../../../usr/lib/python3/dist-packages/sklearn/ensemble/_hist_gradient_boosting/tests/test_gradient_boosting.py::test_poisson 
autopkgtest [12:25:15]: ERROR: timed out on command "su -s /bin/bash 
debci -c set -e; export USER=`id -nu`; . /etc/profile >/dev/null 2>&1 || 
true;  . ~/.profile >/dev/null 2>&1 || true; 
buildtree="/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/build.TbM/src"; mkdir 
-p -m 1777 -- "/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/python3-artifacts"; 
export 
AUTOPKGTEST_ARTIFACTS="/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/python3-artifacts"; 
export ADT_ARTIFACTS="$AUTOPKGTEST_ARTIFACTS"; mkdir -p -m 755 
"/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/autopkgtest_tmp"; export 
AUTOPKGTEST_TMP="/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/autopkgtest_tmp"; 
export ADTTMP="$AUTOPKGTEST_TMP"; export DEBIAN_FRONTEND=noninteractive; 
export LANG=C.UTF-8; export DEB_BUILD_OPTIONS=parallel=64; unset 
LANGUAGE LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE   LC_MONETARY 
LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS   LC_TELEPHONE LC_MEASUREMENT 
LC_IDENTIFICATION LC_ALL;rm -f /tmp/autopkgtest_script_pid; set -C; echo 
$$ > /tmp/autopkgtest_script_pid; set +C; trap "rm -f 
/tmp/autopkgtest_script_pid" EXIT INT QUIT PIPE; cd "$buildtree"; chmod 
+x 
/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/build.TbM/src/debian/tests/python3; 
touch /tmp/autopkgtest-lxc.7u_5xb7m/downtmp/python3-stdout 
/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/python3-stderr; 
/tmp/autopkgtest-lxc.7u_5xb7m/downtmp/build.TbM/src/debian/tests/python3 
2> >(tee -a /tmp/autopkgtest-lxc.7u_5xb7m/downtmp/python3-stderr >&2) > 
 >(tee -a /tmp/autopkgtest-lxc.7u_5xb7m/downtmp/python3-stdout);" (kind: 
test)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/debian-science-maintainers/attachments/20220815/47b92b12/attachment-0001.sig>