[Debian-med-packaging] Bug#1101085: toil: FTBFS: toil.batchSystems.DeadlockException: Deadlock encountered: The workflow is service deadlocked - all 1 running jobs have been the same active services for at least 60 seconds
Santiago Vila
sanvila at debian.org
Sun Mar 23 00:55:09 GMT 2025
Package: src:toil
Version: 8.0.0-4
Severity: serious
Tags: ftbfs trixie sid
Dear maintainer:
During a rebuild of all packages in unstable, your package failed to build:
--------------------------------------------------------------------------------
[...]
debian/rules clean
dh clean --buildsystem=pybuild
dh_auto_clean -O--buildsystem=pybuild
debian/rules execute_after_dh_auto_clean
make[1]: Entering directory '/<<PKGBUILDDIR>>'
rm -rf docs/_build
find . -name "*.pyc" -delete
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
dh_autoreconf_clean -O--buildsystem=pybuild
dh_clean -O--buildsystem=pybuild
debian/rules binary
dh binary --buildsystem=pybuild
dh_update_autotools_config -O--buildsystem=pybuild
dh_autoreconf -O--buildsystem=pybuild
[... snipped ...]
[gw1] [ 47%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadGoogleRepo
toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepo
[gw1] [ 47%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepo
toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepoNTag
[gw1] [ 47%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayRepoNTag
toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayTag
[gw1] [ 48%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testBadQuayTag
toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepo
[gw1] [ 48%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepo
toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepoBadTag
[gw1] [ 48%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testBroadDockerRepoBadTag
toil/test/src/dockerCheckTest.py::DockerCheckTest::testGoogleRepo
[gw1] [ 49%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testGoogleRepo
toil/test/src/dockerCheckTest.py::DockerCheckTest::testNonexistentRepo
[gw1] [ 49%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testNonexistentRepo
toil/test/src/dockerCheckTest.py::DockerCheckTest::testOfficialUbuntuRepo
[gw1] [ 49%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testOfficialUbuntuRepo
toil/test/src/dockerCheckTest.py::DockerCheckTest::testToilQuayRepo
[gw1] [ 49%] SKIPPED toil/test/src/dockerCheckTest.py::DockerCheckTest::testToilQuayRepo
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testBetaInertia
[gw1] [ 50%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testBetaInertia
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMaxNodes
[gw1] [ 50%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMaxNodes
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMinNodes
[gw1] [ 50%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testMinNodes
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testNoLaunchingIfDeltaAlreadyMet
[gw1] [ 50%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testNoLaunchingIfDeltaAlreadyMet
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitIsSet
[gw1] [ 51%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitIsSet
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitResponse
[gw1] [ 51%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testPreemptibleDeficitResponse
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testRounding
[gw1] [ 51%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::testRounding
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_large
[gw1] [ 52%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_large
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_observed
[gw1] [ 52%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_observed
toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_small
[gw1] [ 52%] PASSED toil/test/provisioners/clusterScalerTest.py::ClusterScalerTest::test_overhead_accounting_small
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_and_cancel_workflows
[gw1] [ 52%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_and_cancel_workflows
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_https_url
[gw1] [ 53%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_https_url
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_inputs_zip
[gw1] [ 53%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_inputs_zip
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_and_inputs_zip
[gw1] [ 53%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_and_inputs_zip
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_zip
[gw1] [ 54%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_manifest_zip
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_multi_file_zip
[gw1] [ 54%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_multi_file_zip
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_no_params_zip
[gw1] [ 54%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_no_params_zip
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url
[gw1] [ 54%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url_no_attachments_fails
[gw1] [ 55%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_relative_url_no_attachments_fails
toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_single_file_zip
[gw1] [ 55%] SKIPPED toil/test/server/serverTest.py::ToilWESServerWorkflowTest::test_run_workflow_single_file_zip
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_and_cancel_workflows
[gw1] [ 55%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_and_cancel_workflows
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_https_url
[gw1] [ 55%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_https_url
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_inputs_zip
[gw1] [ 56%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_inputs_zip
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_and_inputs_zip
[gw1] [ 56%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_and_inputs_zip
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_zip
[gw1] [ 56%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_manifest_zip
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_multi_file_zip
[gw1] [ 57%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_multi_file_zip
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_no_params_zip
[gw1] [ 57%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_no_params_zip
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url
[gw1] [ 57%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url_no_attachments_fails
[gw1] [ 57%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_relative_url_no_attachments_fails
toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_single_file_zip
[gw1] [ 58%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryWorkflowTest::test_run_workflow_single_file_zip
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_and_cancel_workflows
[gw1] [ 58%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_and_cancel_workflows
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_https_url
[gw1] [ 58%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_https_url
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_inputs_zip
[gw1] [ 59%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_inputs_zip
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_and_inputs_zip
[gw1] [ 59%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_and_inputs_zip
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_zip
[gw1] [ 59%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_manifest_zip
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_multi_file_zip
[gw1] [ 59%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_multi_file_zip
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_no_params_zip
[gw1] [ 60%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_no_params_zip
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url
[gw1] [ 60%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url_no_attachments_fails
[gw1] [ 60%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_relative_url_no_attachments_fails
toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_single_file_zip
[gw1] [ 60%] SKIPPED toil/test/server/serverTest.py::ToilWESServerCeleryS3StateWorkflowTest::test_run_workflow_single_file_zip
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testAddingInitialNode
[gw1] [ 61%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testAddingInitialNode
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testHighTargetTime
[gw1] [ 61%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testHighTargetTime
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testJobTooLargeForAllNodes
[gw1] [ 61%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testJobTooLargeForAllNodes
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLongRunningJobs
[gw1] [ 62%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLongRunningJobs
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLowTargetTime
[gw1] [ 62%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testLowTargetTime
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPackingOneShape
[gw1] [ 62%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPackingOneShape
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPathologicalCase
[gw1] [ 62%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testPathologicalCase
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testSorting
[gw1] [ 63%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testSorting
toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testZeroResourceJobs
[gw1] [ 63%] PASSED toil/test/provisioners/clusterScalerTest.py::BinPackingTest::testZeroResourceJobs
toil/test/utils/utilsTest.py::UtilsTest::testGetPIDStatus
[gw1] [ 63%] PASSED toil/test/utils/utilsTest.py::UtilsTest::testGetPIDStatus
toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedCWLWF
[gw1] [ 63%] SKIPPED toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedCWLWF
toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF
[gw1] [ 64%] PASSED toil/test/utils/utilsTest.py::UtilsTest::testGetStatusFailedToilWF
toil/test/utils/utilsTest.py::UtilsTest::testGetStatusSuccessfulCWLWF
[gw1] [ 64%] SKIPPED toil/test/utils/utilsTest.py::UtilsTest::testGetStatusSuccessfulCWLWF
toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog
[gw1] [ 64%] PASSED toil/test/utils/utilsTest.py::UtilsTest::testPrintJobLog
toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute
[gw1] [ 65%] PASSED toil/test/utils/utilsTest.py::UtilsTest::testRestartAttribute
toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport
[gw1] [ 65%] PASSED toil/test/utils/utilsTest.py::UtilsTest::testUnicodeSupport
toil/test/utils/utilsTest.py::UtilsTest::test_config_functionality
[gw1] [ 65%] PASSED toil/test/utils/utilsTest.py::UtilsTest::test_config_functionality
toil/test/src/miscTests.py::MiscTests::testIDStability
[gw1] [ 65%] PASSED toil/test/src/miscTests.py::MiscTests::testIDStability
toil/test/src/miscTests.py::MiscTests::test_atomic_context_error
[gw1] [ 66%] PASSED toil/test/src/miscTests.py::MiscTests::test_atomic_context_error
toil/test/src/miscTests.py::MiscTests::test_atomic_context_ok
[gw1] [ 66%] PASSED toil/test/src/miscTests.py::MiscTests::test_atomic_context_ok
toil/test/src/miscTests.py::MiscTests::test_atomic_install
[gw1] [ 66%] PASSED toil/test/src/miscTests.py::MiscTests::test_atomic_install
toil/test/src/miscTests.py::MiscTests::test_atomic_install_dev
[gw1] [ 67%] PASSED toil/test/src/miscTests.py::MiscTests::test_atomic_install_dev
toil/test/src/miscTests.py::MiscTests::test_call_command_err
[gw1] [ 67%] PASSED toil/test/src/miscTests.py::MiscTests::test_call_command_err
toil/test/src/miscTests.py::MiscTests::test_call_command_ok
[gw1] [ 67%] PASSED toil/test/src/miscTests.py::MiscTests::test_call_command_ok
toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testAlways
[gw1] [ 67%] PASSED toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testAlways
toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testNever
[gw1] [ 68%] PASSED toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testNever
toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnErrorWithError
[gw1] [ 68%] PASSED toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnErrorWithError
toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnErrorWithNoError
[gw1] [ 68%] PASSED toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnErrorWithNoError
toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnSuccessWithError
[gw1] [ 68%] PASSED toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnSuccessWithError
toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnSuccessWithSuccess
[gw1] [ 69%] PASSED toil/test/src/retainTempDirTest.py::CleanWorkDirTest::testOnSuccessWithSuccess
toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag
[gw1] [ 69%] PASSED toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag
toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag_with_tags
[gw1] [ 69%] PASSED toil/test/lib/aws/test_utils.py::TagGenerationTest::test_build_tag_with_tags
toil/test/lib/aws/test_utils.py::TagGenerationTest::test_empty_aws_tags
[gw1] [ 70%] PASSED toil/test/lib/aws/test_utils.py::TagGenerationTest::test_empty_aws_tags
toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_emoji
[gw1] [ 70%] PASSED toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_emoji
toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_object
[gw1] [ 70%] PASSED toil/test/lib/aws/test_utils.py::TagGenerationTest::test_incorrect_json_object
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testImportReadFileCompatibility
[gw1] [ 70%] PASSED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testImportReadFileCompatibility
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode
[gw1] [ 71%] PASSED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testReadWriteFileStreamTextMode
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken
[gw1] [ 71%] PASSED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testToilIsNotBroken
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility
[gw1] [ 71%] PASSED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions
[gw1] [ 72%] PASSED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testImportReadFileCompatibility
[gw1] [ 72%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testImportReadFileCompatibility
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode
[gw1] [ 72%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testReadWriteFileStreamTextMode
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testToilIsNotBroken
[gw1] [ 72%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testToilIsNotBroken
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility
[gw1] [ 73%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteExportFileCompatibility
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions
[gw1] [ 73%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithAwsJobStore::testWriteReadGlobalFilePermissions
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testImportReadFileCompatibility
[gw1] [ 73%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testImportReadFileCompatibility
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode
[gw1] [ 73%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testReadWriteFileStreamTextMode
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken
[gw1] [ 74%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testToilIsNotBroken
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility
[gw1] [ 74%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteExportFileCompatibility
toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions
[gw1] [ 74%] SKIPPED toil/test/src/fileStoreTest.py::NonCachingFileStoreTestWithGoogleJobStore::testWriteReadGlobalFilePermissions
toil/test/src/resourceTest.py::ResourceTest::testBuiltIn
[gw1] [ 75%] PASSED toil/test/src/resourceTest.py::ResourceTest::testBuiltIn
toil/test/src/resourceTest.py::ResourceTest::testNonPyStandAlone
[gw1] [ 75%] PASSED toil/test/src/resourceTest.py::ResourceTest::testNonPyStandAlone
toil/test/src/resourceTest.py::ResourceTest::testPackage
[gw1] [ 75%] PASSED toil/test/src/resourceTest.py::ResourceTest::testPackage
toil/test/src/resourceTest.py::ResourceTest::testStandAlone
[gw1] [ 75%] PASSED toil/test/src/resourceTest.py::ResourceTest::testStandAlone
toil/test/src/resourceTest.py::ResourceTest::testStandAloneInPackage
[gw1] [ 76%] PASSED toil/test/src/resourceTest.py::ResourceTest::testStandAloneInPackage
toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testBatchSystemCleanupCanHandleWorkerDeaths
[gw1] [ 76%] PASSED toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testBatchSystemCleanupCanHandleWorkerDeaths
toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testDeferredFunctionRunsWithClassMethod
[gw1] [ 76%] PASSED toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testDeferredFunctionRunsWithClassMethod
toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testDeferredFunctionRunsWithLambda
[gw1] [ 77%] PASSED toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testDeferredFunctionRunsWithLambda
toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testDeferredFunctionRunsWithMethod
[gw1] [ 77%] PASSED toil/test/src/deferredFunctionTest.py::DeferredFunctionTest::testDeferredFunctionRunsWithMethod
toil/test/src/miscTests.py::TestPanic::test_nested_panic
[gw1] [ 77%] PASSED toil/test/src/miscTests.py::TestPanic::test_nested_panic
toil/test/src/miscTests.py::TestPanic::test_panic
[gw1] [ 77%] PASSED toil/test/src/miscTests.py::TestPanic::test_panic
toil/test/src/miscTests.py::TestPanic::test_panic_by_hand
[gw1] [ 78%] PASSED toil/test/src/miscTests.py::TestPanic::test_panic_by_hand
toil/test/src/miscTests.py::TestPanic::test_panic_with_secondary
[gw1] [ 78%] PASSED toil/test/src/miscTests.py::TestPanic::test_panic_with_secondary
toil/test/batchSystems/test_gridengine.py::GridEngineTest::test_coalesce_job_exit_codes_many_all_exist
[gw1] [ 78%] PASSED toil/test/batchSystems/test_gridengine.py::GridEngineTest::test_coalesce_job_exit_codes_many_all_exist
toil/test/batchSystems/test_gridengine.py::GridEngineTest::test_coalesce_job_exit_codes_one_exists
[gw1] [ 78%] PASSED toil/test/batchSystems/test_gridengine.py::GridEngineTest::test_coalesce_job_exit_codes_one_exists
toil/test/batchSystems/test_gridengine.py::GridEngineTest::test_coalesce_job_exit_codes_one_still_running
[gw1] [ 79%] PASSED toil/test/batchSystems/test_gridengine.py::GridEngineTest::test_coalesce_job_exit_codes_one_still_running
toil/test/lib/test_conversions.py::ConversionTest::test_convert
[gw1] [ 79%] PASSED toil/test/lib/test_conversions.py::ConversionTest::test_convert
toil/test/lib/test_conversions.py::ConversionTest::test_hms_duration_to_seconds
[gw1] [ 79%] PASSED toil/test/lib/test_conversions.py::ConversionTest::test_hms_duration_to_seconds
toil/test/lib/test_conversions.py::ConversionTest::test_human2bytes
[gw1] [ 80%] PASSED toil/test/lib/test_conversions.py::ConversionTest::test_human2bytes
toil/test/server/serverTest.py::ToilWESServerBenchTest::test_get_service_info
[gw1] [ 80%] SKIPPED toil/test/server/serverTest.py::ToilWESServerBenchTest::test_get_service_info
toil/test/server/serverTest.py::ToilWESServerBenchTest::test_health
[gw1] [ 80%] SKIPPED toil/test/server/serverTest.py::ToilWESServerBenchTest::test_health
toil/test/server/serverTest.py::ToilWESServerBenchTest::test_home
[gw1] [ 80%] SKIPPED toil/test/server/serverTest.py::ToilWESServerBenchTest::test_home
toil/test/src/busTest.py::MessageBusTest::test_cross_thread_messaging
[gw1] [ 81%] PASSED toil/test/src/busTest.py::MessageBusTest::test_cross_thread_messaging
toil/test/src/busTest.py::MessageBusTest::test_enum_ints_in_file
[gw1] [ 81%] PASSED toil/test/src/busTest.py::MessageBusTest::test_enum_ints_in_file
toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path
[gw1] [ 81%] PASSED toil/test/src/busTest.py::MessageBusTest::test_restart_without_bus_path
toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore
[gw1] [ 81%] PASSED toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testCachingFileStore
toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore
[gw1] [ 82%] PASSED toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testJobFileStore
toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore
[gw1] [ 82%] PASSED toil/test/src/jobFileStoreTest.py::JobFileStoreTest::testNonCachingFileStore
toil/test/src/jobTest.py::JobTest::testDAGConsistency
[gw1] [ 82%] PASSED toil/test/src/jobTest.py::JobTest::testDAGConsistency
toil/test/src/jobTest.py::JobTest::testDeadlockDetection
[gw1] [ 83%] PASSED toil/test/src/jobTest.py::JobTest::testDeadlockDetection
toil/test/src/jobTest.py::JobTest::testStatic2
[gw1] [ 83%] PASSED toil/test/src/jobTest.py::JobTest::testStatic2
toil/test/src/promisedRequirementTest.py::SingleMachinePromisedRequirementsTest::testPromisesWithJobStoreFileObjects
[gw1] [ 83%] PASSED toil/test/src/promisedRequirementTest.py::SingleMachinePromisedRequirementsTest::testPromisesWithJobStoreFileObjects
toil/test/src/promisedRequirementTest.py::SingleMachinePromisedRequirementsTest::testPromisesWithNonCachingFileStore
[gw1] [ 83%] PASSED toil/test/src/promisedRequirementTest.py::SingleMachinePromisedRequirementsTest::testPromisesWithNonCachingFileStore
toil/test/src/promisedRequirementTest.py::SingleMachinePromisedRequirementsTest::test_omp_threads <- toil/test/batchSystems/batchSystemTest.py
[gw1] [ 84%] PASSED toil/test/src/promisedRequirementTest.py::SingleMachinePromisedRequirementsTest::test_omp_threads <- toil/test/batchSystems/batchSystemTest.py
toil/test/src/promisedRequirementTest.py::MesosPromisedRequirementsTest::testPromisesWithJobStoreFileObjects
[gw1] [ 84%] SKIPPED toil/test/src/promisedRequirementTest.py::MesosPromisedRequirementsTest::testPromisesWithJobStoreFileObjects
toil/test/src/promisedRequirementTest.py::MesosPromisedRequirementsTest::testPromisesWithNonCachingFileStore
[gw1] [ 84%] SKIPPED toil/test/src/promisedRequirementTest.py::MesosPromisedRequirementsTest::testPromisesWithNonCachingFileStore
toil/test/src/promisedRequirementTest.py::MesosPromisedRequirementsTest::test_omp_threads <- toil/test/batchSystems/batchSystemTest.py
[gw1] [ 85%] SKIPPED toil/test/src/promisedRequirementTest.py::MesosPromisedRequirementsTest::test_omp_threads <- toil/test/batchSystems/batchSystemTest.py
toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClassFromMain
[gw1] [ 85%] PASSED toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobClassFromMain
toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction
[gw1] [ 85%] PASSED toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunction
toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunctionFromMain
[gw1] [ 85%] PASSED toil/test/src/userDefinedJobArgTypeTest.py::UserDefinedJobArgTypeTest::testJobFunctionFromMain
toil/test/utils/toilDebugTest.py::DebugJobTest::test_print_job_info
[gw1] [ 86%] PASSED toil/test/utils/toilDebugTest.py::DebugJobTest::test_print_job_info
toil/test/utils/toilDebugTest.py::DebugJobTest::test_retrieve_task_directory
[gw1] [ 86%] SKIPPED toil/test/utils/toilDebugTest.py::DebugJobTest::test_retrieve_task_directory
toil/test/utils/toilDebugTest.py::DebugJobTest::test_run_job
[gw1] [ 86%] PASSED toil/test/utils/toilDebugTest.py::DebugJobTest::test_run_job
toil/test/__init__.py::toil.test.make_tests
[gw1] [ 86%] PASSED toil/test/__init__.py::toil.test.make_tests
toil/test/__init__.py::toil.test.timeLimit
[gw1] [ 87%] PASSED toil/test/__init__.py::toil.test.timeLimit
toil/test/options/options.py::OptionsTest::test_caching_option_priority
[gw1] [ 87%] PASSED toil/test/options/options.py::OptionsTest::test_caching_option_priority
toil/test/options/options.py::OptionsTest::test_default_caching_slurm
[gw1] [ 87%] PASSED toil/test/options/options.py::OptionsTest::test_default_caching_slurm
toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store
[gw1] [ 88%] SKIPPED toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store
toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store_paths
[gw1] [ 88%] SKIPPED toil/test/server/serverTest.py::AWSStateStoreTest::test_state_store_paths
toil/test/src/importExportFileTest.py::ImportExportFileTest::test_basic_import_export
[gw1] [ 88%] PASSED toil/test/src/importExportFileTest.py::ImportExportFileTest::test_basic_import_export
toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_false
[gw1] [ 88%] PASSED toil/test/src/importExportFileTest.py::ImportExportFileTest::test_import_export_restart_false
toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescription
[gw1] [ 89%] PASSED toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescription
toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescriptionSequencing
[gw1] [ 89%] PASSED toil/test/src/jobDescriptionTest.py::JobDescriptionTest::testJobDescriptionSequencing
toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testAddChildEncapsulate
[gw1] [ 89%] PASSED toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testAddChildEncapsulate
toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation
[gw1] [ 90%] PASSED toil/test/src/jobEncapsulationTest.py::JobEncapsulationTest::testEncapsulation
toil/test/src/regularLogTest.py::RegularLogTest::testRegularLog
[gw1] [ 90%] PASSED toil/test/src/regularLogTest.py::RegularLogTest::testRegularLog
toil/test/src/regularLogTest.py::RegularLogTest::testWriteLogs
[gw1] [ 90%] PASSED toil/test/src/regularLogTest.py::RegularLogTest::testWriteLogs
toil/test/src/threadingTest.py::ThreadingTest::testGlobalMutexOrdering
[gw1] [ 90%] PASSED toil/test/src/threadingTest.py::ThreadingTest::testGlobalMutexOrdering
toil/test/src/threadingTest.py::ThreadingTest::testLastProcessStanding
[gw1] [ 91%] PASSED toil/test/src/threadingTest.py::ThreadingTest::testLastProcessStanding
toil/test/batchSystems/batch_system_plugin_test.py::BatchSystemPluginTest::test_batchsystem_plugin_installable
[gw1] [ 91%] PASSED toil/test/batchSystems/batch_system_plugin_test.py::BatchSystemPluginTest::test_batchsystem_plugin_installable
toil/test/batchSystems/test_lsf_helper.py::LSFHelperTest::test_parse_mem_and_cmd_from_output
[gw1] [ 91%] PASSED toil/test/batchSystems/test_lsf_helper.py::LSFHelperTest::test_parse_mem_and_cmd_from_output
toil/test/jobStores/jobStoreTest.py::InvalidAWSJobStoreTest::testInvalidJobStoreName
[gw1] [ 91%] SKIPPED toil/test/jobStores/jobStoreTest.py::InvalidAWSJobStoreTest::testInvalidJobStoreName
toil/test/lib/test_misc.py::UserNameAvailableTest::test_get_user_name
[gw1] [ 92%] PASSED toil/test/lib/test_misc.py::UserNameAvailableTest::test_get_user_name
toil/test/lib/test_misc.py::UserNameUnvailableTest::test_get_user_name
[gw1] [ 92%] PASSED toil/test/lib/test_misc.py::UserNameUnvailableTest::test_get_user_name
toil/test/lib/test_misc.py::UserNameVeryBrokenTest::test_get_user_name
[gw1] [ 92%] PASSED toil/test/lib/test_misc.py::UserNameVeryBrokenTest::test_get_user_name
toil/test/mesos/MesosDataStructuresTest.py::DataStructuresTest::testJobQueue
[gw1] [ 93%] PASSED toil/test/mesos/MesosDataStructuresTest.py::DataStructuresTest::testJobQueue
toil/test/provisioners/provisionerTest.py::ProvisionerTest::test_node_type_parsing
[gw1] [ 93%] PASSED toil/test/provisioners/provisionerTest.py::ProvisionerTest::test_node_type_parsing
toil/test/server/serverTest.py::ToilServerUtilsTest::test_workflow_canceling_recovery
[gw1] [ 93%] SKIPPED toil/test/server/serverTest.py::ToilServerUtilsTest::test_workflow_canceling_recovery
toil/test/server/serverTest.py::FileStateStoreTest::test_state_store
[gw1] [ 93%] SKIPPED toil/test/server/serverTest.py::FileStateStoreTest::test_state_store
toil/test/server/serverTest.py::FileStateStoreURLTest::test_state_store
[gw1] [ 94%] SKIPPED toil/test/server/serverTest.py::FileStateStoreURLTest::test_state_store
toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried
[gw1] [ 94%] PASSED toil/test/src/checkpointTest.py::CheckpointTest::testCheckpointNotRetried
toil/test/src/environmentTest.py::EnvironmentTest::test_environment
[gw1] [ 94%] PASSED toil/test/src/environmentTest.py::EnvironmentTest::test_environment
toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld
[gw1] [ 95%] PASSED toil/test/src/helloWorldTest.py::HelloWorldTest::testHelloWorld
toil/test/src/jobServiceTest.py::JobServiceTest::testServiceWithCheckpoints
[gw1] [ 95%] SKIPPED toil/test/src/jobServiceTest.py::JobServiceTest::testServiceWithCheckpoints
toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceWithCheckpoints
[gw1] [ 95%] SKIPPED toil/test/src/jobServiceTest.py::PerfectServiceTest::testServiceWithCheckpoints
toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test
[gw1] [ 95%] PASSED toil/test/src/promisesTest.py::CachedUnpicklingJobStoreTest::test
toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test
[gw1] [ 96%] PASSED toil/test/src/promisesTest.py::ChainedIndexedPromisesTest::test
toil/test/src/promisesTest.py::PathIndexingPromiseTest::test
[gw1] [ 96%] PASSED toil/test/src/promisesTest.py::PathIndexingPromiseTest::test
toil/test/src/realtimeLoggerTest.py::RealtimeLoggerTest::testRealtimeLogger
[gw1] [ 96%] PASSED toil/test/src/realtimeLoggerTest.py::RealtimeLoggerTest::testRealtimeLogger
toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining
[gw0] [ 96%] PASSED toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testSimultaneousReadsUncachedStream
toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken
[gw1] [ 97%] PASSED toil/test/src/resumabilityTest.py::ResumabilityTest::test_chaining
toil/test/src/systemTest.py::SystemTest::testAtomicityOfNonEmptyDirectoryRenames
[gw0] [ 97%] PASSED toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testToilIsNotBroken
toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility
[gw1] [ 97%] PASSED toil/test/src/systemTest.py::SystemTest::testAtomicityOfNonEmptyDirectoryRenames
toil/test/src/workerTest.py::WorkerTests::testNextChainable
[gw1] [ 98%] PASSED toil/test/src/workerTest.py::WorkerTests::testNextChainable
toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles
[gw0] [ 98%] PASSED toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteExportFileCompatibility
toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore
[gw1] [ 98%] PASSED toil/test/utils/toilDebugTest.py::testFetchJobStoreFiles
toil/test/utils/toilKillTest.py::ToilKillTest::test_cwl_toil_kill
[gw0] [ 98%] PASSED toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteLocalFileToJobStore
toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore
[gw1] [ 99%] PASSED toil/test/utils/toilKillTest.py::ToilKillTest::test_cwl_toil_kill
toil/test/utils/toilKillTest.py::ToilKillTestWithAWSJobStore::test_cwl_toil_kill
[gw1] [ 99%] SKIPPED toil/test/utils/toilKillTest.py::ToilKillTestWithAWSJobStore::test_cwl_toil_kill
[gw0] [ 99%] PASSED toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteNonLocalFileToJobStore
toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions
[gw0] [100%] PASSED toil/test/src/fileStoreTest.py::CachingFileStoreTestWithFileJobStore::testWriteReadGlobalFilePermissions
=================================== FAILURES ===================================
______________________ ToilDocumentationTest.testServices ______________________
[gw1] linux -- Python 3.13.2 /usr/bin/python3.13
Traceback (most recent call last):
File "/usr/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
yield
File "/usr/lib/python3.13/unittest/case.py", line 651, in run
self._callTestMethod(testMethod)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/usr/lib/python3.13/unittest/case.py", line 606, in _callTestMethod
if method() is not None:
~~~~~~^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/test/docs/scriptsTest.py", line 166, in testServices
self.checkExitCode("tutorial_services.py")
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/test/docs/scriptsTest.py", line 49, in checkExitCode
raise RuntimeError(stderr)
RuntimeError: [2025-03-23T00:22:27+0000] [MainThread] [I] [toil] Running Toil version 8.0.0-d2ae0ea9ab49f238670dbf6aafd20de7afdd8514 on host c7a-large-1742688931.
[2025-03-23T00:22:27+0000] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.
[2025-03-23T00:22:27+0000] [MainThread] [I] [toil.leader] Issued job 'Job' kind-Job/instance-d7r7bix5 v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-03-23T00:22:28+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run
[2025-03-23T00:22:30+0000] [MainThread] [I] [toil.leader] Issued job 'DemoService' kind-DemoService/instance-g5shw13_ v1 with job batch system ID: 2 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-03-23T00:22:34+0000] [MainThread] [I] [toil.leader] Issued job 'dbFn' kind-dbFn/instance-6_ixslsk v1 with job batch system ID: 3 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-03-23T00:23:00+0000] [MainThread] [W] [toil.leader] Potential deadlock detected! All 1 running jobs are service jobs, with no normal jobs to use them! The batch system reports: Not enough memory to run job 3
[2025-03-23T00:23:30+0000] [MainThread] [W] [toil.leader] Potentially deadlocked for 30 seconds. Waiting at most 30 more seconds for any of 1 issued non-service jobs to schedule and start. The batch system reports: Not enough memory to run job 3
[2025-03-23T00:24:00+0000] [MainThread] [E] [toil.leader] We have been deadlocked since 1742689380.054194 on these service jobs: {'kind-DemoService/instance-g5shw13_'}
[2025-03-23T00:24:01+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2025-03-23T00:24:01+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
[2025-03-23T00:24:01+0000] [MainThread] [W] [toil.fileStores.nonCachingFileStore] Detected that job ('DemoService' kind-DemoService/instance-g5shw13_ v1) prematurely terminated. Fixing the state of the job on disk.
[2025-03-23T00:24:01+0000] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/tmp/tmpodm70nnftutorial_services)
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/test/docs/scripts/tutorial_services.py", line 45, in <module>
toil.start(j)
~~~~~~~~~~^^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/common.py", line 1055, in start
return self._runMainLoop(rootJobDescription)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/common.py", line 1632, in _runMainLoop
).run()
~~~^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/leader.py", line 283, in run
self.innerLoop()
~~~~~~~~~~~~~~^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/leader.py", line 943, in innerLoop
self.checkForDeadlocks()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/leader.py", line 1044, in checkForDeadlocks
raise DeadlockException(
...<5 lines>...
)
toil.batchSystems.DeadlockException: Deadlock encountered: The workflow is service deadlocked - all 1 running jobs have been the same active services for at least 60 seconds
---------------------------- Captured stdout setup -----------------------------
[TEST] toil.test.docs.scriptsTest.ToilDocumentationTest:testServices (Mar 22 2025 17:22:27:500313 PDT)
=========================== short test summary info ============================
FAILED toil/test/docs/scriptsTest.py::ToilDocumentationTest::testServices - RuntimeError: [2025-03-23T00:22:27+0000] [MainThread] [I] [toil] Running Toil version 8.0.0-d2ae0ea9ab49f238670dbf6aafd20de7afdd8514 on host c7a-large-1742688931.
[2025-03-23T00:22:27+0000] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.
[2025-03-23T00:22:27+0000] [MainThread] [I] [toil.leader] Issued job 'Job' kind-Job/instance-d7r7bix5 v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-03-23T00:22:28+0000] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run
[2025-03-23T00:22:30+0000] [MainThread] [I] [toil.leader] Issued job 'DemoService' kind-DemoService/instance-g5shw13_ v1 with job batch system ID: 2 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-03-23T00:22:34+0000] [MainThread] [I] [toil.leader] Issued job 'dbFn' kind-dbFn/instance-6_ixslsk v1 with job batch system ID: 3 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-03-23T00:23:00+0000] [MainThread] [W] [toil.leader] Potential deadlock detected! All 1 running jobs are service jobs, with no normal jobs to use them! The batch system reports: Not enough memory to run job 3
[2025-03-23T00:23:30+0000] [MainThread] [W] [toil.leader] Potentially deadlocked for 30 seconds. Waiting at most 30 more seconds for any of 1 issued non-service jobs to schedule and start. The batch system reports: Not enough memory to run job 3
[2025-03-23T00:24:00+0000] [MainThread] [E] [toil.leader] We have been deadlocked since 1742689380.054194 on these service jobs: {'kind-DemoService/instance-g5shw13_'}
[2025-03-23T00:24:01+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2025-03-23T00:24:01+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
[2025-03-23T00:24:01+0000] [MainThread] [W] [toil.fileStores.nonCachingFileStore] Detected that job ('DemoService' kind-DemoService/instance-g5shw13_ v1) prematurely terminated. Fixing the state of the job on disk.
[2025-03-23T00:24:01+0000] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/tmp/tmpodm70nnftutorial_services)
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/test/docs/scripts/tutorial_services.py", line 45, in <module>
toil.start(j)
~~~~~~~~~~^^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/common.py", line 1055, in start
return self._runMainLoop(rootJobDescription)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/common.py", line 1632, in _runMainLoop
).run()
~~~^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/leader.py", line 283, in run
self.innerLoop()
~~~~~~~~~~~~~~^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/leader.py", line 943, in innerLoop
self.checkForDeadlocks()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build/toil/leader.py", line 1044, in checkForDeadlocks
raise DeadlockException(
...<5 lines>...
)
toil.batchSystems.DeadlockException: Deadlock encountered: The workflow is service deadlocked - all 1 running jobs have been the same active services for at least 60 seconds
============ 1 failed, 205 passed, 155 skipped in 521.29s (0:08:41) ============
E: pybuild pybuild:389: test: plugin pyproject failed with: exit code=1: cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.13_toil/build; python3.13 -m pytest -n 2 --dist loadscope -vv -W ignore -k "not (testVirtualEnv or DockstoreLookupTest or testMultiplejobs)" -m "not slow" --ignore toil/test/lib/aws/test_s3.py --ignore toil/test/provisioners/aws/awsProvisionerTest.py --ignore toil/test/wdl/wdltoil_test.py --ignore toil/test/cwl/cwlTest.py --ignore toil/test/lib/test_ec2.py --ignore toil/test/batchSystems/batchSystemTest.py --ignore toil/test/lib/aws/test_iam.py --ignore toil/test/provisioners/clusterTest.py --ignore toil/test/wdl/wdltoil_test_kubernetes.py --ignore toil/test/cactus/test_cactus_integration.py toil/test
dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p 3.13 returned exit code 13
make[1]: *** [debian/rules:37: override_dh_auto_test] Error 25
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:20: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
--------------------------------------------------------------------------------
The above is just how the build ends and not necessarily the most relevant part.
If required, the full build log is available here:
https://people.debian.org/~sanvila/build-logs/202503/
About the archive rebuild: The build was made on virtual machines from AWS,
using sbuild and a reduced chroot with only build-essential packages.
If you could not reproduce the bug please contact me privately, as I
am willing to provide ssh access to a virtual machine where the bug is
fully reproducible.
If this is really a bug in one of the build-depends, please use
reassign and add an affects on src:toil, so that this is still
visible in the BTS web page for this package.
Thanks.
More information about the Debian-med-packaging
mailing list