[Debichem-devel] Bug#980634: gromacs: FTBFS: Errors while running CTest
Lucas Nussbaum
lucas at debian.org
Wed Jan 20 20:41:19 GMT 2021
Source: gromacs
Version: 2020.4-2
Severity: serious
Justification: FTBFS on amd64
Tags: bullseye sid ftbfs
Usertags: ftbfs-20210120 ftbfs-bullseye
Hi,
During a rebuild of all packages in sid, your package failed to build
on amd64.
Relevant part (hopefully):
> make[4]: Entering directory '/<<PKGBUILDDIR>>/build/mpich'
> make[4]: Nothing to be done for 'CMakeFiles/tests.dir/build'.
> make[4]: Leaving directory '/<<PKGBUILDDIR>>/build/mpich'
> [100%] Built target tests
> make[3]: Leaving directory '/<<PKGBUILDDIR>>/build/mpich'
> /usr/bin/cmake -E cmake_progress_start /<<PKGBUILDDIR>>/build/mpich/CMakeFiles 0
> make[2]: Leaving directory '/<<PKGBUILDDIR>>/build/mpich'
> make[1]: Leaving directory '/<<PKGBUILDDIR>>/build/mpich'
> (cd build/mpich; LD_LIBRARY_PATH=/<<PKGBUILDDIR>>/build/mpich/lib \
> ctest -V || dpkg-architecture -i hurd-i386 || dpkg-architecture -i armhf )
> UpdateCTestConfiguration from :/<<PKGBUILDDIR>>/build/mpich/DartConfiguration.tcl
> Parse Config file:/<<PKGBUILDDIR>>/build/mpich/DartConfiguration.tcl
> UpdateCTestConfiguration from :/<<PKGBUILDDIR>>/build/mpich/DartConfiguration.tcl
> Parse Config file:/<<PKGBUILDDIR>>/build/mpich/DartConfiguration.tcl
> Test project /<<PKGBUILDDIR>>/build/mpich
> Constructing a list of tests
> Done constructing a list of tests
> Updating test list for fixtures
> Added 0 tests to meet fixture requirements
> Checking test dependency graph...
> Checking test dependency graph end
> test 1
> Start 1: TestUtilsUnitTests
>
> 1: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/TestUtilsUnitTests.xml"
> 1: Test timeout computed to be: 30
> 1: [1611165257.171056] [ip-172-31-13-129:15115:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 1: [1611165257.171089] [ip-172-31-13-129:15115:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 1: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 1: MPIR_Init_thread(152).......:
> 1: MPID_Init(597)..............:
> 1: MPIDI_UCX_mpi_init_hook(247):
> 1: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 1: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 1: :
> 1: system msg for write_line failure : Bad file descriptor
> 1: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 1: MPIR_Init_thread(152).......:
> 1: MPID_Init(597)..............:
> 1: MPIDI_UCX_mpi_init_hook(247):
> 1: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 1: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 1: :
> 1: system msg for write_line failure : Bad file descriptor
> 1: [ip-172-31-13-129:15115:0:15115] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 1: ==== backtrace (tid: 15115) ====
> 1: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f8398998ea4]
> 1: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f83989990af]
> 1: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f839899926a]
> 1: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f8399997140]
> 1: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f83991ca431]
> 1: 5 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x122f45) [0x562fbb939f45]
> 1: 6 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0xdf969) [0x562fbb8f6969]
> 1: 7 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x93890) [0x562fbb8aa890]
> 1: 8 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x49efd) [0x562fbb860efd]
> 1: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f83989e5d0a]
> 1: 10 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x4a88a) [0x562fbb86188a]
> 1: =================================
> 1/30 Test #1: TestUtilsUnitTests ...............***Exception: SegFault 0.10 sec
> [1611165257.171056] [ip-172-31-13-129:15115:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.171089] [ip-172-31-13-129:15115:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15115:0:15115] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15115) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f8398998ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f83989990af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f839899926a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f8399997140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f83991ca431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x122f45) [0x562fbb939f45]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0xdf969) [0x562fbb8f6969]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x93890) [0x562fbb8aa890]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x49efd) [0x562fbb860efd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f83989e5d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/testutils-test(+0x4a88a) [0x562fbb86188a]
> =================================
>
> test 2
> Start 2: TestUtilsMpiUnitTests
>
> 2: Test command: /usr/bin/mpiexec.mpich "-np" "2" "-host" "localhost" "/<<PKGBUILDDIR>>/build/mpich/bin/testutils-mpi-test" "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/TestUtilsMpiUnitTests.xml"
> 2: Test timeout computed to be: 30
> 2: [1611165257.208556] [ip-172-31-13-129:15125:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 2: [1611165257.208581] [ip-172-31-13-129:15125:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 2: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 2: MPIR_Init_thread(152).......:
> 2: MPID_Init(597)..............:
> 2: MPIDI_UCX_mpi_init_hook(247):
> 2: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 2/30 Test #2: TestUtilsMpiUnitTests ............***Failed 0.03 sec
> [1611165257.208556] [ip-172-31-13-129:15125:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.208581] [ip-172-31-13-129:15125:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
>
> test 3
> Start 3: UtilityUnitTests
>
> 3: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/utility-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/UtilityUnitTests.xml"
> 3: Test timeout computed to be: 30
> 3: [1611165257.232618] [ip-172-31-13-129:15142:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 3: [1611165257.232648] [ip-172-31-13-129:15142:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 3: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 3: MPIR_Init_thread(152).......:
> 3: MPID_Init(597)..............:
> 3: MPIDI_UCX_mpi_init_hook(247):
> 3: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 3: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 3: :
> 3: system msg for write_line failure : Bad file descriptor
> 3: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 3: MPIR_Init_thread(152).......:
> 3: MPID_Init(597)..............:
> 3: MPIDI_UCX_mpi_init_hook(247):
> 3: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 3: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 3: :
> 3: system msg for write_line failure : Bad file descriptor
> 3: [ip-172-31-13-129:15142:0:15142] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 3: ==== backtrace (tid: 15142) ====
> 3: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f497c87eea4]
> 3: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f497c87f0af]
> 3: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f497c87f26a]
> 3: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f497d87d140]
> 3: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f497d0b0431]
> 3: 5 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x223405) [0x562cc5240405]
> 3: 6 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x1df5d9) [0x562cc51fc5d9]
> 3: 7 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x199200) [0x562cc51b6200]
> 3: 8 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x7508d) [0x562cc509208d]
> 3: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f497c8cbd0a]
> 3: 10 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x75a9a) [0x562cc5092a9a]
> 3: =================================
> 3/30 Test #3: UtilityUnitTests .................***Exception: SegFault 0.02 sec
> [1611165257.232618] [ip-172-31-13-129:15142:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.232648] [ip-172-31-13-129:15142:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15142:0:15142] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15142) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f497c87eea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f497c87f0af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f497c87f26a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f497d87d140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f497d0b0431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x223405) [0x562cc5240405]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x1df5d9) [0x562cc51fc5d9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x199200) [0x562cc51b6200]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x7508d) [0x562cc509208d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f497c8cbd0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/utility-test(+0x75a9a) [0x562cc5092a9a]
> =================================
>
> test 4
> Start 4: UtilityMpiUnitTests
>
> 4: Test command: /usr/bin/mpiexec.mpich "-np" "4" "-host" "localhost" "/<<PKGBUILDDIR>>/build/mpich/bin/utility-mpi-test" "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/UtilityMpiUnitTests.xml"
> 4: Test timeout computed to be: 30
> 4: [1611165257.268625] [ip-172-31-13-129:15154:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 4: [1611165257.268653] [ip-172-31-13-129:15154:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 4: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 4: MPIR_Init_thread(152).......:
> 4: MPID_Init(597)..............:
> 4: MPIDI_UCX_mpi_init_hook(247):
> 4: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 4/30 Test #4: UtilityMpiUnitTests ..............***Failed 0.04 sec
> [1611165257.268625] [ip-172-31-13-129:15154:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.268653] [ip-172-31-13-129:15154:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
>
> test 5
> Start 5: MdlibUnitTest
>
> 5: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/MdlibUnitTest.xml"
> 5: Test timeout computed to be: 30
> 5: [1611165257.294476] [ip-172-31-13-129:15183:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 5: [1611165257.294505] [ip-172-31-13-129:15183:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 5: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 5: MPIR_Init_thread(152).......:
> 5: MPID_Init(597)..............:
> 5: MPIDI_UCX_mpi_init_hook(247):
> 5: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 5: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 5: :
> 5: system msg for write_line failure : Bad file descriptor
> 5: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 5: MPIR_Init_thread(152).......:
> 5: MPID_Init(597)..............:
> 5: MPIDI_UCX_mpi_init_hook(247):
> 5: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 5: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 5: :
> 5: system msg for write_line failure : Bad file descriptor
> 5: [ip-172-31-13-129:15183:0:15183] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 5: ==== backtrace (tid: 15183) ====
> 5: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f985e6deea4]
> 5: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f985e6df0af]
> 5: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f985e6df26a]
> 5: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f985f6e3140]
> 5: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f985ef16431]
> 5: 5 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0x1e4355) [0x558002461355]
> 5: 6 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0xfbd89) [0x558002378d89]
> 5: 7 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0xcf780) [0x55800234c780]
> 5: 8 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0x669ed) [0x5580022e39ed]
> 5: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f985e731d0a]
> 5: 10 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0x67a8a) [0x5580022e4a8a]
> 5: =================================
> 5/30 Test #5: MdlibUnitTest ....................***Exception: SegFault 0.02 sec
> [1611165257.294476] [ip-172-31-13-129:15183:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.294505] [ip-172-31-13-129:15183:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15183:0:15183] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15183) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f985e6deea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f985e6df0af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f985e6df26a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f985f6e3140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f985ef16431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0x1e4355) [0x558002461355]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0xfbd89) [0x558002378d89]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0xcf780) [0x55800234c780]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0x669ed) [0x5580022e39ed]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f985e731d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/mdlib-test(+0x67a8a) [0x5580022e4a8a]
> =================================
>
> test 6
> Start 6: AppliedForcesUnitTest
>
> 6: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/AppliedForcesUnitTest.xml"
> 6: Test timeout computed to be: 30
> 6: [1611165257.317635] [ip-172-31-13-129:15191:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 6: [1611165257.317657] [ip-172-31-13-129:15191:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 6: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 6: MPIR_Init_thread(152).......:
> 6: MPID_Init(597)..............:
> 6: MPIDI_UCX_mpi_init_hook(247):
> 6: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 6: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 6: :
> 6: system msg for write_line failure : Bad file descriptor
> 6: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 6: MPIR_Init_thread(152).......:
> 6: MPID_Init(597)..............:
> 6: MPIDI_UCX_mpi_init_hook(247):
> 6: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 6: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 6: :
> 6: system msg for write_line failure : Bad file descriptor
> 6: [ip-172-31-13-129:15191:0:15191] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 6: ==== backtrace (tid: 15191) ====
> 6: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f1556be5ea4]
> 6: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f1556be60af]
> 6: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f1556be626a]
> 6: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f1557bea140]
> 6: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f155741d431]
> 6: 5 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x12ff25) [0x55cc711fef25]
> 6: 6 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0xb7679) [0x55cc71186679]
> 6: 7 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x6a880) [0x55cc71139880]
> 6: 8 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x4cdcd) [0x55cc7111bdcd]
> 6: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f1556c38d0a]
> 6: 10 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x4d75a) [0x55cc7111c75a]
> 6: =================================
> 6/30 Test #6: AppliedForcesUnitTest ............***Exception: SegFault 0.02 sec
> [1611165257.317635] [ip-172-31-13-129:15191:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.317657] [ip-172-31-13-129:15191:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15191:0:15191] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15191) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f1556be5ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f1556be60af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f1556be626a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f1557bea140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f155741d431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x12ff25) [0x55cc711fef25]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0xb7679) [0x55cc71186679]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x6a880) [0x55cc71139880]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x4cdcd) [0x55cc7111bdcd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f1556c38d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/applied_forces-test(+0x4d75a) [0x55cc7111c75a]
> =================================
>
> test 7
> Start 7: CommandLineUnitTests
>
> 7: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/CommandLineUnitTests.xml"
> 7: Test timeout computed to be: 30
> 7: [1611165257.340262] [ip-172-31-13-129:15199:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 7: [1611165257.340283] [ip-172-31-13-129:15199:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 7: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 7: MPIR_Init_thread(152).......:
> 7: MPID_Init(597)..............:
> 7: MPIDI_UCX_mpi_init_hook(247):
> 7: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 7: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 7: :
> 7: system msg for write_line failure : Bad file descriptor
> 7: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 7: MPIR_Init_thread(152).......:
> 7: MPID_Init(597)..............:
> 7: MPIDI_UCX_mpi_init_hook(247):
> 7: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 7: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 7: :
> 7: system msg for write_line failure : Bad file descriptor
> 7: [ip-172-31-13-129:15199:0:15199] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 7: ==== backtrace (tid: 15199) ====
> 7: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2946613ea4]
> 7: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f29466140af]
> 7: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f294661426a]
> 7: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2947612140]
> 7: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2946e45431]
> 7: 5 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x154715) [0x56394f8fd715]
> 7: 6 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x10c1b9) [0x56394f8b51b9]
> 7: 7 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0xc0700) [0x56394f869700]
> 7: 8 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x50f8d) [0x56394f7f9f8d]
> 7: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2946660d0a]
> 7: 10 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x5194a) [0x56394f7fa94a]
> 7: =================================
> 7/30 Test #7: CommandLineUnitTests .............***Exception: SegFault 0.02 sec
> [1611165257.340262] [ip-172-31-13-129:15199:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.340283] [ip-172-31-13-129:15199:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15199:0:15199] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15199) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2946613ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f29466140af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f294661426a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2947612140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2946e45431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x154715) [0x56394f8fd715]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x10c1b9) [0x56394f8b51b9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0xc0700) [0x56394f869700]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x50f8d) [0x56394f7f9f8d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2946660d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/commandline-test(+0x5194a) [0x56394f7fa94a]
> =================================
>
> test 8
> Start 8: DomDecTests
>
> 8: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/DomDecTests.xml"
> 8: Test timeout computed to be: 30
> 8: [1611165257.362231] [ip-172-31-13-129:15207:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 8: [1611165257.362255] [ip-172-31-13-129:15207:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 8: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 8: MPIR_Init_thread(152).......:
> 8: MPID_Init(597)..............:
> 8: MPIDI_UCX_mpi_init_hook(247):
> 8: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 8: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 8: :
> 8: system msg for write_line failure : Bad file descriptor
> 8: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 8: MPIR_Init_thread(152).......:
> 8: MPID_Init(597)..............:
> 8: MPIDI_UCX_mpi_init_hook(247):
> 8: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 8: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 8: :
> 8: system msg for write_line failure : Bad file descriptor
> 8: [ip-172-31-13-129:15207:0:15207] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 8: ==== backtrace (tid: 15207) ====
> 8: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fab7f249ea4]
> 8: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fab7f24a0af]
> 8: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fab7f24a26a]
> 8: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fab80248140]
> 8: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fab7fa7b431]
> 8: 5 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0xc9e05) [0x5606bde2ce05]
> 8: 6 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x85559) [0x5606bdde8559]
> 8: 7 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x47d40) [0x5606bddaad40]
> 8: 8 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x3a80d) [0x5606bdd9d80d]
> 8: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fab7f296d0a]
> 8: 10 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x3b0ca) [0x5606bdd9e0ca]
> 8: =================================
> 8/30 Test #8: DomDecTests ......................***Exception: SegFault 0.02 sec
> [1611165257.362231] [ip-172-31-13-129:15207:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.362255] [ip-172-31-13-129:15207:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15207:0:15207] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15207) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fab7f249ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fab7f24a0af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fab7f24a26a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fab80248140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fab7fa7b431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0xc9e05) [0x5606bde2ce05]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x85559) [0x5606bdde8559]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x47d40) [0x5606bddaad40]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x3a80d) [0x5606bdd9d80d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fab7f296d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/domdec-test(+0x3b0ca) [0x5606bdd9e0ca]
> =================================
>
> test 9
> Start 9: EwaldUnitTests
>
> 9: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/EwaldUnitTests.xml"
> 9: Test timeout computed to be: 30
> 9: [1611165257.383961] [ip-172-31-13-129:15215:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 9: [1611165257.383983] [ip-172-31-13-129:15215:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 9: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 9: MPIR_Init_thread(152).......:
> 9: MPID_Init(597)..............:
> 9: MPIDI_UCX_mpi_init_hook(247):
> 9: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 9: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 9: :
> 9: system msg for write_line failure : Bad file descriptor
> 9: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 9: MPIR_Init_thread(152).......:
> 9: MPID_Init(597)..............:
> 9: MPIDI_UCX_mpi_init_hook(247):
> 9: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 9: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 9: :
> 9: system msg for write_line failure : Bad file descriptor
> 9: [ip-172-31-13-129:15215:0:15215] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 9: ==== backtrace (tid: 15215) ====
> 9: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f71736daea4]
> 9: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f71736db0af]
> 9: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f71736db26a]
> 9: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f7174707140]
> 9: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f7173f3a431]
> 9: 5 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0x169f55) [0x55ac066dbf55]
> 9: 6 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0xd5289) [0x55ac06647289]
> 9: 7 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0xa9d40) [0x55ac0661bd40]
> 9: 8 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0x553f0) [0x55ac065c73f0]
> 9: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f7173755d0a]
> 9: 10 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0x5604a) [0x55ac065c804a]
> 9: =================================
> 9/30 Test #9: EwaldUnitTests ...................***Exception: SegFault 0.02 sec
> [1611165257.383961] [ip-172-31-13-129:15215:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.383983] [ip-172-31-13-129:15215:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15215:0:15215] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15215) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f71736daea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f71736db0af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f71736db26a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f7174707140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f7173f3a431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0x169f55) [0x55ac066dbf55]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0xd5289) [0x55ac06647289]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0xa9d40) [0x55ac0661bd40]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0x553f0) [0x55ac065c73f0]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f7173755d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/ewald-test(+0x5604a) [0x55ac065c804a]
> =================================
>
> test 10
> Start 10: FFTUnitTests
>
> 10: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/fft-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/FFTUnitTests.xml"
> 10: Test timeout computed to be: 30
> 10: [1611165257.405339] [ip-172-31-13-129:15223:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 10: [1611165257.405363] [ip-172-31-13-129:15223:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 10: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 10: MPIR_Init_thread(152).......:
> 10: MPID_Init(597)..............:
> 10: MPIDI_UCX_mpi_init_hook(247):
> 10: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 10: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 10: :
> 10: system msg for write_line failure : Bad file descriptor
> 10: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 10: MPIR_Init_thread(152).......:
> 10: MPID_Init(597)..............:
> 10: MPIDI_UCX_mpi_init_hook(247):
> 10: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 10: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 10: :
> 10: system msg for write_line failure : Bad file descriptor
> 10: [ip-172-31-13-129:15223:0:15223] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 10: ==== backtrace (tid: 15223) ====
> 10: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f830e911ea4]
> 10: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f830e9120af]
> 10: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f830e91226a]
> 10: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f830f916140]
> 10: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f830f149431]
> 10: 5 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0xc8da5) [0x55c3c6b6ada5]
> 10: 6 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x842c9) [0x55c3c6b262c9]
> 10: 7 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x58c40) [0x55c3c6afac40]
> 10: 8 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x3cf3d) [0x55c3c6adef3d]
> 10: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f830e964d0a]
> 10: 10 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x3d87a) [0x55c3c6adf87a]
> 10: =================================
> 10/30 Test #10: FFTUnitTests .....................***Exception: SegFault 0.02 sec
> [1611165257.405339] [ip-172-31-13-129:15223:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.405363] [ip-172-31-13-129:15223:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15223:0:15223] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15223) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f830e911ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f830e9120af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f830e91226a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f830f916140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f830f149431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0xc8da5) [0x55c3c6b6ada5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x842c9) [0x55c3c6b262c9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x58c40) [0x55c3c6afac40]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x3cf3d) [0x55c3c6adef3d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f830e964d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/fft-test(+0x3d87a) [0x55c3c6adf87a]
> =================================
>
> test 11
> Start 11: GpuUtilsUnitTests
>
> 11: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/GpuUtilsUnitTests.xml"
> 11: Test timeout computed to be: 30
> 11: [1611165257.426336] [ip-172-31-13-129:15231:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 11: [1611165257.426358] [ip-172-31-13-129:15231:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 11: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 11: MPIR_Init_thread(152).......:
> 11: MPID_Init(597)..............:
> 11: MPIDI_UCX_mpi_init_hook(247):
> 11: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 11: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 11: :
> 11: system msg for write_line failure : Bad file descriptor
> 11: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 11: MPIR_Init_thread(152).......:
> 11: MPID_Init(597)..............:
> 11: MPIDI_UCX_mpi_init_hook(247):
> 11: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 11: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 11: :
> 11: system msg for write_line failure : Bad file descriptor
> 11: [ip-172-31-13-129:15231:0:15231] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 11: ==== backtrace (tid: 15231) ====
> 11: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f57b6af2ea4]
> 11: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f57b6af30af]
> 11: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f57b6af326a]
> 11: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f57b7af1140]
> 11: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f57b7324431]
> 11: 5 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x104d45) [0x55eb6ef06d45]
> 11: 6 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0xc0c29) [0x55eb6eec2c29]
> 11: 7 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x83490) [0x55eb6ee85490]
> 11: 8 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x4088d) [0x55eb6ee4288d]
> 11: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f57b6b3fd0a]
> 11: 10 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x4118a) [0x55eb6ee4318a]
> 11: =================================
> 11/30 Test #11: GpuUtilsUnitTests ................***Exception: SegFault 0.02 sec
> [1611165257.426336] [ip-172-31-13-129:15231:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.426358] [ip-172-31-13-129:15231:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15231:0:15231] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15231) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f57b6af2ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f57b6af30af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f57b6af326a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f57b7af1140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f57b7324431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x104d45) [0x55eb6ef06d45]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0xc0c29) [0x55eb6eec2c29]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x83490) [0x55eb6ee85490]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x4088d) [0x55eb6ee4288d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f57b6b3fd0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/gpu_utils-test(+0x4118a) [0x55eb6ee4318a]
> =================================
>
> test 12
> Start 12: HardwareUnitTests
>
> 12: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/HardwareUnitTests.xml"
> 12: Test timeout computed to be: 30
> 12: [1611165257.447686] [ip-172-31-13-129:15239:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 12: [1611165257.447706] [ip-172-31-13-129:15239:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 12: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 12: MPIR_Init_thread(152).......:
> 12: MPID_Init(597)..............:
> 12: MPIDI_UCX_mpi_init_hook(247):
> 12: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 12: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 12: :
> 12: system msg for write_line failure : Bad file descriptor
> 12: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 12: MPIR_Init_thread(152).......:
> 12: MPID_Init(597)..............:
> 12: MPIDI_UCX_mpi_init_hook(247):
> 12: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 12: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 12: :
> 12: system msg for write_line failure : Bad file descriptor
> 12: [ip-172-31-13-129:15239:0:15239] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 12: ==== backtrace (tid: 15239) ====
> 12: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fa6b1b64ea4]
> 12: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fa6b1b650af]
> 12: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fa6b1b6526a]
> 12: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fa6b2b91140]
> 12: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fa6b23c4431]
> 12: 5 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0xcb9f5) [0x561546dd99f5]
> 12: 6 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x81889) [0x561546d8f889]
> 12: 7 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x43f20) [0x561546d51f20]
> 12: 8 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x3958d) [0x561546d4758d]
> 12: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fa6b1bdfd0a]
> 12: 10 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x39e4a) [0x561546d47e4a]
> 12: =================================
> 12/30 Test #12: HardwareUnitTests ................***Exception: SegFault 0.02 sec
> [1611165257.447686] [ip-172-31-13-129:15239:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.447706] [ip-172-31-13-129:15239:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15239:0:15239] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15239) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fa6b1b64ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fa6b1b650af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fa6b1b6526a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fa6b2b91140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fa6b23c4431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0xcb9f5) [0x561546dd99f5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x81889) [0x561546d8f889]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x43f20) [0x561546d51f20]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x3958d) [0x561546d4758d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fa6b1bdfd0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/hardware-test(+0x39e4a) [0x561546d47e4a]
> =================================
>
> test 13
> Start 13: MathUnitTests
>
> 13: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/math-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/MathUnitTests.xml"
> 13: Test timeout computed to be: 30
> 13: [1611165257.468320] [ip-172-31-13-129:15247:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 13: [1611165257.468340] [ip-172-31-13-129:15247:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 13: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 13: MPIR_Init_thread(152).......:
> 13: MPID_Init(597)..............:
> 13: MPIDI_UCX_mpi_init_hook(247):
> 13: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 13: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 13: :
> 13: system msg for write_line failure : Bad file descriptor
> 13: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 13: MPIR_Init_thread(152).......:
> 13: MPID_Init(597)..............:
> 13: MPIDI_UCX_mpi_init_hook(247):
> 13: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 13: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 13: :
> 13: system msg for write_line failure : Bad file descriptor
> 13: [ip-172-31-13-129:15247:0:15247] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 13: ==== backtrace (tid: 15247) ====
> 13: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2ce59e7ea4]
> 13: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2ce59e80af]
> 13: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2ce59e826a]
> 13: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2ce69e6140]
> 13: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2ce6219431]
> 13: 5 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x1b6fb5) [0x55f69e70efb5]
> 13: 6 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x162f89) [0x55f69e6baf89]
> 13: 7 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x1360a0) [0x55f69e68e0a0]
> 13: 8 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x6571d) [0x55f69e5bd71d]
> 13: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2ce5a34d0a]
> 13: 10 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x6600a) [0x55f69e5be00a]
> 13: =================================
> 13/30 Test #13: MathUnitTests ....................***Exception: SegFault 0.02 sec
> [1611165257.468320] [ip-172-31-13-129:15247:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.468340] [ip-172-31-13-129:15247:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15247:0:15247] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15247) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2ce59e7ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2ce59e80af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2ce59e826a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2ce69e6140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2ce6219431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x1b6fb5) [0x55f69e70efb5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x162f89) [0x55f69e6baf89]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x1360a0) [0x55f69e68e0a0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x6571d) [0x55f69e5bd71d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2ce5a34d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/math-test(+0x6600a) [0x55f69e5be00a]
> =================================
>
> test 14
> Start 14: MdrunUtilityUnitTests
>
> 14: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/MdrunUtilityUnitTests.xml"
> 14: Test timeout computed to be: 30
> 14: [1611165257.489699] [ip-172-31-13-129:15255:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 14: [1611165257.489718] [ip-172-31-13-129:15255:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 14: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 14: MPIR_Init_thread(152).......:
> 14: MPID_Init(597)..............:
> 14: MPIDI_UCX_mpi_init_hook(247):
> 14: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 14: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 14: :
> 14: system msg for write_line failure : Bad file descriptor
> 14: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 14: MPIR_Init_thread(152).......:
> 14: MPID_Init(597)..............:
> 14: MPIDI_UCX_mpi_init_hook(247):
> 14: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 14: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 14: :
> 14: system msg for write_line failure : Bad file descriptor
> 14: [ip-172-31-13-129:15255:0:15255] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 14: ==== backtrace (tid: 15255) ====
> 14: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fe9fd1b8ea4]
> 14: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fe9fd1b90af]
> 14: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fe9fd1b926a]
> 14: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fe9fe1e5140]
> 14: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fe9fda18431]
> 14: 5 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0xdecb5) [0x557218e31cb5]
> 14: 6 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x997c9) [0x557218dec7c9]
> 14: 7 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x5bba0) [0x557218daeba0]
> 14: 8 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x3dd1d) [0x557218d90d1d]
> 14: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fe9fd233d0a]
> 14: 10 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x3e65a) [0x557218d9165a]
> 14: =================================
> 14/30 Test #14: MdrunUtilityUnitTests ............***Exception: SegFault 0.02 sec
> [1611165257.489699] [ip-172-31-13-129:15255:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.489718] [ip-172-31-13-129:15255:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15255:0:15255] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15255) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fe9fd1b8ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fe9fd1b90af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fe9fd1b926a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fe9fe1e5140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fe9fda18431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0xdecb5) [0x557218e31cb5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x997c9) [0x557218dec7c9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x5bba0) [0x557218daeba0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x3dd1d) [0x557218d90d1d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fe9fd233d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-test(+0x3e65a) [0x557218d9165a]
> =================================
>
> test 15
> Start 15: MdrunUtilityMpiUnitTests
>
> 15: Test command: /usr/bin/mpiexec.mpich "-np" "4" "-host" "localhost" "/<<PKGBUILDDIR>>/build/mpich/bin/mdrunutility-mpi-test" "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/MdrunUtilityMpiUnitTests.xml"
> 15: Test timeout computed to be: 30
> 15: [1611165257.527219] [ip-172-31-13-129:15268:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 15: [1611165257.527246] [ip-172-31-13-129:15268:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 15: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 15: MPIR_Init_thread(152).......:
> 15: MPID_Init(597)..............:
> 15: MPIDI_UCX_mpi_init_hook(247):
> 15: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 15/30 Test #15: MdrunUtilityMpiUnitTests .........***Failed 0.04 sec
> [1611165257.527219] [ip-172-31-13-129:15268:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.527246] [ip-172-31-13-129:15268:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
>
> test 16
> Start 16: MDSpanTests
>
> 16: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/MDSpanTests.xml"
> 16: Test timeout computed to be: 30
> 16: [1611165257.550247] [ip-172-31-13-129:15297:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 16: [1611165257.550274] [ip-172-31-13-129:15297:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 16: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 16: MPIR_Init_thread(152).......:
> 16: MPID_Init(597)..............:
> 16: MPIDI_UCX_mpi_init_hook(247):
> 16: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 16: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 16: :
> 16: system msg for write_line failure : Bad file descriptor
> 16: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 16: MPIR_Init_thread(152).......:
> 16: MPID_Init(597)..............:
> 16: MPIDI_UCX_mpi_init_hook(247):
> 16: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 16: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 16: :
> 16: system msg for write_line failure : Bad file descriptor
> 16: [ip-172-31-13-129:15297:0:15297] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 16: ==== backtrace (tid: 15297) ====
> 16: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fa0f85a1ea4]
> 16: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fa0f85a20af]
> 16: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fa0f85a226a]
> 16: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fa0f95a0140]
> 16: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fa0f8dd3431]
> 16: 5 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0xe16e5) [0x557ff0ab06e5]
> 16: 6 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x9deb9) [0x557ff0a6ceb9]
> 16: 7 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x60540) [0x557ff0a2f540]
> 16: 8 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x3f3bd) [0x557ff0a0e3bd]
> 16: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fa0f85eed0a]
> 16: 10 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x3fc7a) [0x557ff0a0ec7a]
> 16: =================================
> 16/30 Test #16: MDSpanTests ......................***Exception: SegFault 0.02 sec
> [1611165257.550247] [ip-172-31-13-129:15297:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.550274] [ip-172-31-13-129:15297:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15297:0:15297] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15297) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fa0f85a1ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fa0f85a20af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fa0f85a226a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fa0f95a0140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fa0f8dd3431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0xe16e5) [0x557ff0ab06e5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x9deb9) [0x557ff0a6ceb9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x60540) [0x557ff0a2f540]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x3f3bd) [0x557ff0a0e3bd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fa0f85eed0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/mdspan-test(+0x3fc7a) [0x557ff0a0ec7a]
> =================================
>
> test 17
> Start 17: OnlineHelpUnitTests
>
> 17: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/OnlineHelpUnitTests.xml"
> 17: Test timeout computed to be: 30
> 17: [1611165257.572739] [ip-172-31-13-129:15305:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 17: [1611165257.572762] [ip-172-31-13-129:15305:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 17: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 17: MPIR_Init_thread(152).......:
> 17: MPID_Init(597)..............:
> 17: MPIDI_UCX_mpi_init_hook(247):
> 17: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 17: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 17: :
> 17: system msg for write_line failure : Bad file descriptor
> 17: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 17: MPIR_Init_thread(152).......:
> 17: MPID_Init(597)..............:
> 17: MPIDI_UCX_mpi_init_hook(247):
> 17: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 17: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 17: :
> 17: system msg for write_line failure : Bad file descriptor
> 17: [ip-172-31-13-129:15305:0:15305] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 17: ==== backtrace (tid: 15305) ====
> 17: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2b66757ea4]
> 17: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2b667580af]
> 17: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2b6675826a]
> 17: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2b67756140]
> 17: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2b66f89431]
> 17: 5 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0xd3ed5) [0x56531bd4ced5]
> 17: 6 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x8b639) [0x56531bd04639]
> 17: 7 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x4d2d0) [0x56531bcc62d0]
> 17: 8 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x3ef1d) [0x56531bcb7f1d]
> 17: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2b667a4d0a]
> 17: 10 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x3f87a) [0x56531bcb887a]
> 17: =================================
> 17/30 Test #17: OnlineHelpUnitTests ..............***Exception: SegFault 0.02 sec
> [1611165257.572739] [ip-172-31-13-129:15305:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.572762] [ip-172-31-13-129:15305:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15305:0:15305] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15305) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2b66757ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2b667580af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2b6675826a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2b67756140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2b66f89431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0xd3ed5) [0x56531bd4ced5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x8b639) [0x56531bd04639]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x4d2d0) [0x56531bcc62d0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x3ef1d) [0x56531bcb7f1d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2b667a4d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/onlinehelp-test(+0x3f87a) [0x56531bcb887a]
> =================================
>
> test 18
> Start 18: OptionsUnitTests
>
> 18: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/options-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/OptionsUnitTests.xml"
> 18: Test timeout computed to be: 30
> 18: [1611165257.594631] [ip-172-31-13-129:15313:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 18: [1611165257.594654] [ip-172-31-13-129:15313:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 18: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 18: MPIR_Init_thread(152).......:
> 18: MPID_Init(597)..............:
> 18: MPIDI_UCX_mpi_init_hook(247):
> 18: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 18: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 18: :
> 18: system msg for write_line failure : Bad file descriptor
> 18: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 18: MPIR_Init_thread(152).......:
> 18: MPID_Init(597)..............:
> 18: MPIDI_UCX_mpi_init_hook(247):
> 18: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 18: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 18: :
> 18: system msg for write_line failure : Bad file descriptor
> 18: [ip-172-31-13-129:15313:0:15313] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 18: ==== backtrace (tid: 15313) ====
> 18: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2671e88ea4]
> 18: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2671e890af]
> 18: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2671e8926a]
> 18: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2672e87140]
> 18: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f26726ba431]
> 18: 5 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x176545) [0x559f7ad5e545]
> 18: 6 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x12f039) [0x559f7ad17039]
> 18: 7 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0xf7b10) [0x559f7acdfb10]
> 18: 8 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x7262d) [0x559f7ac5a62d]
> 18: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2671ed5d0a]
> 18: 10 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x7306a) [0x559f7ac5b06a]
> 18: =================================
> 18/30 Test #18: OptionsUnitTests .................***Exception: SegFault 0.02 sec
> [1611165257.594631] [ip-172-31-13-129:15313:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.594654] [ip-172-31-13-129:15313:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15313:0:15313] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15313) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2671e88ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2671e890af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2671e8926a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2672e87140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f26726ba431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x176545) [0x559f7ad5e545]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x12f039) [0x559f7ad17039]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0xf7b10) [0x559f7acdfb10]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x7262d) [0x559f7ac5a62d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2671ed5d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/options-test(+0x7306a) [0x559f7ac5b06a]
> =================================
>
> test 19
> Start 19: PbcutilUnitTest
>
> 19: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/PbcutilUnitTest.xml"
> 19: Test timeout computed to be: 30
> 19: [1611165257.617187] [ip-172-31-13-129:15321:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 19: [1611165257.617210] [ip-172-31-13-129:15321:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 19: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 19: MPIR_Init_thread(152).......:
> 19: MPID_Init(597)..............:
> 19: MPIDI_UCX_mpi_init_hook(247):
> 19: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 19: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 19: :
> 19: system msg for write_line failure : Bad file descriptor
> 19: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 19: MPIR_Init_thread(152).......:
> 19: MPID_Init(597)..............:
> 19: MPIDI_UCX_mpi_init_hook(247):
> 19: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 19: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 19: :
> 19: system msg for write_line failure : Bad file descriptor
> 19: [ip-172-31-13-129:15321:0:15321] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 19: ==== backtrace (tid: 15321) ====
> 19: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fc196b58ea4]
> 19: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fc196b590af]
> 19: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fc196b5926a]
> 19: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fc197b5d140]
> 19: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fc197390431]
> 19: 5 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0xc36d5) [0x5639909856d5]
> 19: 6 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x78f79) [0x56399093af79]
> 19: 7 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x4d8f0) [0x56399090f8f0]
> 19: 8 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x39ccd) [0x5639908fbccd]
> 19: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fc196babd0a]
> 19: 10 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x3a58a) [0x5639908fc58a]
> 19: =================================
> 19/30 Test #19: PbcutilUnitTest ..................***Exception: SegFault 0.02 sec
> [1611165257.617187] [ip-172-31-13-129:15321:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.617210] [ip-172-31-13-129:15321:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15321:0:15321] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15321) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fc196b58ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fc196b590af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fc196b5926a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fc197b5d140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fc197390431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0xc36d5) [0x5639909856d5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x78f79) [0x56399093af79]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x4d8f0) [0x56399090f8f0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x39ccd) [0x5639908fbccd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fc196babd0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/pbcutil-test(+0x3a58a) [0x5639908fc58a]
> =================================
>
> test 20
> Start 20: RandomUnitTests
>
> 20: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/random-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/RandomUnitTests.xml"
> 20: Test timeout computed to be: 30
> 20: [1611165257.643414] [ip-172-31-13-129:15329:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 20: [1611165257.643438] [ip-172-31-13-129:15329:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 20: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 20: MPIR_Init_thread(152).......:
> 20: MPID_Init(597)..............:
> 20: MPIDI_UCX_mpi_init_hook(247):
> 20: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 20: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 20: :
> 20: system msg for write_line failure : Bad file descriptor
> 20: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 20: MPIR_Init_thread(152).......:
> 20: MPID_Init(597)..............:
> 20: MPIDI_UCX_mpi_init_hook(247):
> 20: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 20: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 20: :
> 20: system msg for write_line failure : Bad file descriptor
> 20: [ip-172-31-13-129:15329:0:15329] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 20: ==== backtrace (tid: 15329) ====
> 20: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2c5af51ea4]
> 20: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2c5af520af]
> 20: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2c5af5226a]
> 20: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2c5bf50140]
> 20: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2c5b783431]
> 20: 5 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0xf7e95) [0x55e38ef17e95]
> 20: 6 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0xb3e99) [0x55e38eed3e99]
> 20: 7 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0x88ab0) [0x55e38eea8ab0]
> 20: 8 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0x4625d) [0x55e38ee6625d]
> 20: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2c5af9ed0a]
> 20: 10 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0x46e9a) [0x55e38ee66e9a]
> 20: =================================
> 20/30 Test #20: RandomUnitTests ..................***Exception: SegFault 0.03 sec
> [1611165257.643414] [ip-172-31-13-129:15329:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.643438] [ip-172-31-13-129:15329:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15329:0:15329] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15329) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2c5af51ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f2c5af520af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f2c5af5226a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f2c5bf50140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2c5b783431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0xf7e95) [0x55e38ef17e95]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0xb3e99) [0x55e38eed3e99]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0x88ab0) [0x55e38eea8ab0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0x4625d) [0x55e38ee6625d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2c5af9ed0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/random-test(+0x46e9a) [0x55e38ee66e9a]
> =================================
>
> test 21
> Start 21: RestraintTests
>
> 21: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/RestraintTests.xml"
> 21: Test timeout computed to be: 30
> 21: [1611165257.665150] [ip-172-31-13-129:15337:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 21: [1611165257.665175] [ip-172-31-13-129:15337:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 21: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 21: MPIR_Init_thread(152).......:
> 21: MPID_Init(597)..............:
> 21: MPIDI_UCX_mpi_init_hook(247):
> 21: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 21: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 21: :
> 21: system msg for write_line failure : Bad file descriptor
> 21: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 21: MPIR_Init_thread(152).......:
> 21: MPID_Init(597)..............:
> 21: MPIDI_UCX_mpi_init_hook(247):
> 21: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 21: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 21: :
> 21: system msg for write_line failure : Bad file descriptor
> 21: [ip-172-31-13-129:15337:0:15337] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 21: ==== backtrace (tid: 15337) ====
> 21: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f226c74bea4]
> 21: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f226c74c0af]
> 21: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f226c74c26a]
> 21: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f226d74a140]
> 21: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f226cf7d431]
> 21: 5 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0xbe6a5) [0x55913c0be6a5]
> 21: 6 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x782a9) [0x55913c0782a9]
> 21: 7 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x3a9c0) [0x55913c03a9c0]
> 21: 8 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x3875d) [0x55913c03875d]
> 21: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f226c798d0a]
> 21: 10 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x3904a) [0x55913c03904a]
> 21: =================================
> 21/30 Test #21: RestraintTests ...................***Exception: SegFault 0.02 sec
> [1611165257.665150] [ip-172-31-13-129:15337:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.665175] [ip-172-31-13-129:15337:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15337:0:15337] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15337) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f226c74bea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f226c74c0af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f226c74c26a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f226d74a140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f226cf7d431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0xbe6a5) [0x55913c0be6a5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x782a9) [0x55913c0782a9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x3a9c0) [0x55913c03a9c0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x3875d) [0x55913c03875d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f226c798d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/restraintpotential-test(+0x3904a) [0x55913c03904a]
> =================================
>
> test 22
> Start 22: TableUnitTests
>
> 22: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/table-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/TableUnitTests.xml"
> 22: Test timeout computed to be: 30
> 22: [1611165257.686156] [ip-172-31-13-129:15345:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 22: [1611165257.686178] [ip-172-31-13-129:15345:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 22: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 22: MPIR_Init_thread(152).......:
> 22: MPID_Init(597)..............:
> 22: MPIDI_UCX_mpi_init_hook(247):
> 22: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 22: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 22: :
> 22: system msg for write_line failure : Bad file descriptor
> 22: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 22: MPIR_Init_thread(152).......:
> 22: MPID_Init(597)..............:
> 22: MPIDI_UCX_mpi_init_hook(247):
> 22: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 22: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 22: :
> 22: system msg for write_line failure : Bad file descriptor
> 22: [ip-172-31-13-129:15345:0:15345] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 22: ==== backtrace (tid: 15345) ====
> 22: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7efdeac86ea4]
> 22: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7efdeac870af]
> 22: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7efdeac8726a]
> 22: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7efdebc85140]
> 22: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7efdeb4b8431]
> 22: 5 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0xf6a15) [0x55b0b7bc5a15]
> 22: 6 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0xac999) [0x55b0b7b7b999]
> 22: 7 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0x6fd70) [0x55b0b7b3ed70]
> 22: 8 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0x421bd) [0x55b0b7b111bd]
> 22: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7efdeacd3d0a]
> 22: 10 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0x42a7a) [0x55b0b7b11a7a]
> 22: =================================
> 22/30 Test #22: TableUnitTests ...................***Exception: SegFault 0.02 sec
> [1611165257.686156] [ip-172-31-13-129:15345:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.686178] [ip-172-31-13-129:15345:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15345:0:15345] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15345) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7efdeac86ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7efdeac870af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7efdeac8726a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7efdebc85140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7efdeb4b8431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0xf6a15) [0x55b0b7bc5a15]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0xac999) [0x55b0b7b7b999]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0x6fd70) [0x55b0b7b3ed70]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0x421bd) [0x55b0b7b111bd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7efdeacd3d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/table-test(+0x42a7a) [0x55b0b7b11a7a]
> =================================
>
> test 23
> Start 23: TaskAssignmentUnitTests
>
> 23: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/TaskAssignmentUnitTests.xml"
> 23: Test timeout computed to be: 30
> 23: [1611165257.706854] [ip-172-31-13-129:15353:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 23: [1611165257.706874] [ip-172-31-13-129:15353:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 23: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 23: MPIR_Init_thread(152).......:
> 23: MPID_Init(597)..............:
> 23: MPIDI_UCX_mpi_init_hook(247):
> 23: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 23: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 23: :
> 23: system msg for write_line failure : Bad file descriptor
> 23: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 23: MPIR_Init_thread(152).......:
> 23: MPID_Init(597)..............:
> 23: MPIDI_UCX_mpi_init_hook(247):
> 23: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 23: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 23: :
> 23: system msg for write_line failure : Bad file descriptor
> 23: [ip-172-31-13-129:15353:0:15353] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 23: ==== backtrace (tid: 15353) ====
> 23: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fef56631ea4]
> 23: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fef566320af]
> 23: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fef5663226a]
> 23: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fef57630140]
> 23: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fef56e63431]
> 23: 5 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0xcfe65) [0x56364ea4de65]
> 23: 6 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x8a449) [0x56364ea08449]
> 23: 7 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x4cdb0) [0x56364e9cadb0]
> 23: 8 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x3b3dd) [0x56364e9b93dd]
> 23: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fef5667ed0a]
> 23: 10 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x3bc9a) [0x56364e9b9c9a]
> 23: =================================
> 23/30 Test #23: TaskAssignmentUnitTests ..........***Exception: SegFault 0.02 sec
> [1611165257.706854] [ip-172-31-13-129:15353:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.706874] [ip-172-31-13-129:15353:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15353:0:15353] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15353) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fef56631ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fef566320af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fef5663226a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fef57630140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fef56e63431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0xcfe65) [0x56364ea4de65]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x8a449) [0x56364ea08449]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x4cdb0) [0x56364e9cadb0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x3b3dd) [0x56364e9b93dd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fef5667ed0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/taskassignment-test(+0x3bc9a) [0x56364e9b9c9a]
> =================================
>
> test 24
> Start 24: TopologyTest
>
> 24: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/topology-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/TopologyTest.xml"
> 24: Test timeout computed to be: 30
> 24: [1611165257.727790] [ip-172-31-13-129:15361:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 24: [1611165257.727813] [ip-172-31-13-129:15361:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 24: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 24: MPIR_Init_thread(152).......:
> 24: MPID_Init(597)..............:
> 24: MPIDI_UCX_mpi_init_hook(247):
> 24: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 24: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 24: :
> 24: system msg for write_line failure : Bad file descriptor
> 24: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 24: MPIR_Init_thread(152).......:
> 24: MPID_Init(597)..............:
> 24: MPIDI_UCX_mpi_init_hook(247):
> 24: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 24: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 24: :
> 24: system msg for write_line failure : Bad file descriptor
> 24: [ip-172-31-13-129:15361:0:15361] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 24: ==== backtrace (tid: 15361) ====
> 24: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f0c0484bea4]
> 24: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f0c0484c0af]
> 24: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f0c0484c26a]
> 24: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f0c0584a140]
> 24: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f0c0507d431]
> 24: 5 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0xe0955) [0x55bdcea51955]
> 24: 6 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x8d959) [0x55bdce9fe959]
> 24: 7 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x622d0) [0x55bdce9d32d0]
> 24: 8 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x3eaed) [0x55bdce9afaed]
> 24: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f0c04898d0a]
> 24: 10 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x3f3fa) [0x55bdce9b03fa]
> 24: =================================
> 24/30 Test #24: TopologyTest .....................***Exception: SegFault 0.02 sec
> [1611165257.727790] [ip-172-31-13-129:15361:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.727813] [ip-172-31-13-129:15361:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15361:0:15361] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15361) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f0c0484bea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f0c0484c0af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f0c0484c26a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f0c0584a140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f0c0507d431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0xe0955) [0x55bdcea51955]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x8d959) [0x55bdce9fe959]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x622d0) [0x55bdce9d32d0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x3eaed) [0x55bdce9afaed]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f0c04898d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/topology-test(+0x3f3fa) [0x55bdce9b03fa]
> =================================
>
> test 25
> Start 25: PullTest
>
> 25: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/pull-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/PullTest.xml"
> 25: Test timeout computed to be: 30
> 25: [1611165257.748422] [ip-172-31-13-129:15369:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 25: [1611165257.748447] [ip-172-31-13-129:15369:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 25: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 25: MPIR_Init_thread(152).......:
> 25: MPID_Init(597)..............:
> 25: MPIDI_UCX_mpi_init_hook(247):
> 25: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 25: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 25: :
> 25: system msg for write_line failure : Bad file descriptor
> 25: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 25: MPIR_Init_thread(152).......:
> 25: MPID_Init(597)..............:
> 25: MPIDI_UCX_mpi_init_hook(247):
> 25: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 25: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 25: :
> 25: system msg for write_line failure : Bad file descriptor
> 25: [ip-172-31-13-129:15369:0:15369] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 25: ==== backtrace (tid: 15369) ====
> 25: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f6c5163cea4]
> 25: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f6c5163d0af]
> 25: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f6c5163d26a]
> 25: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f6c52641140]
> 25: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f6c51e74431]
> 25: 5 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0xe5b35) [0x556d7e1afb35]
> 25: 6 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x80eb9) [0x556d7e14aeb9]
> 25: 7 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x44170) [0x556d7e10e170]
> 25: 8 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x3f80d) [0x556d7e10980d]
> 25: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f6c5168fd0a]
> 25: 10 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x4019a) [0x556d7e10a19a]
> 25: =================================
> 25/30 Test #25: PullTest .........................***Exception: SegFault 0.02 sec
> [1611165257.748422] [ip-172-31-13-129:15369:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.748447] [ip-172-31-13-129:15369:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15369:0:15369] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15369) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f6c5163cea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f6c5163d0af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f6c5163d26a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f6c52641140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f6c51e74431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0xe5b35) [0x556d7e1afb35]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x80eb9) [0x556d7e14aeb9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x44170) [0x556d7e10e170]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x3f80d) [0x556d7e10980d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f6c5168fd0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/pull-test(+0x4019a) [0x556d7e10a19a]
> =================================
>
> test 26
> Start 26: AwhTest
>
> 26: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/awh-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/AwhTest.xml"
> 26: Test timeout computed to be: 30
> 26: [1611165257.769432] [ip-172-31-13-129:15377:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 26: [1611165257.769458] [ip-172-31-13-129:15377:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 26: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 26: MPIR_Init_thread(152).......:
> 26: MPID_Init(597)..............:
> 26: MPIDI_UCX_mpi_init_hook(247):
> 26: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 26: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 26: :
> 26: system msg for write_line failure : Bad file descriptor
> 26: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 26: MPIR_Init_thread(152).......:
> 26: MPID_Init(597)..............:
> 26: MPIDI_UCX_mpi_init_hook(247):
> 26: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 26: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 26: :
> 26: system msg for write_line failure : Bad file descriptor
> 26: [ip-172-31-13-129:15377:0:15377] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 26: ==== backtrace (tid: 15377) ====
> 26: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f1b33140ea4]
> 26: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f1b331410af]
> 26: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f1b3314126a]
> 26: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f1b3413f140]
> 26: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f1b33972431]
> 26: 5 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0xfe585) [0x562f455a7585]
> 26: 6 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x97229) [0x562f45540229]
> 26: 7 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x6b980) [0x562f45514980]
> 26: 8 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x3e81d) [0x562f454e781d]
> 26: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f1b3318dd0a]
> 26: 10 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x3f30a) [0x562f454e830a]
> 26: =================================
> 26/30 Test #26: AwhTest ..........................***Exception: SegFault 0.02 sec
> [1611165257.769432] [ip-172-31-13-129:15377:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.769458] [ip-172-31-13-129:15377:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15377:0:15377] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15377) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f1b33140ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f1b331410af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f1b3314126a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f1b3413f140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f1b33972431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0xfe585) [0x562f455a7585]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x97229) [0x562f45540229]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x6b980) [0x562f45514980]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x3e81d) [0x562f454e781d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f1b3318dd0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/awh-test(+0x3f30a) [0x562f454e830a]
> =================================
>
> test 27
> Start 27: SimdUnitTests
>
> 27: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/simd-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/SimdUnitTests.xml"
> 27: Test timeout computed to be: 30
> 27: [1611165257.791495] [ip-172-31-13-129:15385:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 27: [1611165257.791521] [ip-172-31-13-129:15385:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 27: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 27: MPIR_Init_thread(152).......:
> 27: MPID_Init(597)..............:
> 27: MPIDI_UCX_mpi_init_hook(247):
> 27: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 27: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 27: :
> 27: system msg for write_line failure : Bad file descriptor
> 27: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 27: MPIR_Init_thread(152).......:
> 27: MPID_Init(597)..............:
> 27: MPIDI_UCX_mpi_init_hook(247):
> 27: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 27: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 27: :
> 27: system msg for write_line failure : Bad file descriptor
> 27: [ip-172-31-13-129:15385:0:15385] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 27: ==== backtrace (tid: 15385) ====
> 27: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f36d1530ea4]
> 27: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f36d15310af]
> 27: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f36d153126a]
> 27: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f36d252f140]
> 27: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f36d1d62431]
> 27: 5 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x16c385) [0x5636ac4fa385]
> 27: 6 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x128ee9) [0x5636ac4b6ee9]
> 27: 7 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0xfd640) [0x5636ac48b640]
> 27: 8 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x749bd) [0x5636ac4029bd]
> 27: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f36d157dd0a]
> 27: 10 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x7527a) [0x5636ac40327a]
> 27: =================================
> 27/30 Test #27: SimdUnitTests ....................***Exception: SegFault 0.02 sec
> [1611165257.791495] [ip-172-31-13-129:15385:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.791521] [ip-172-31-13-129:15385:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15385:0:15385] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15385) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f36d1530ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f36d15310af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f36d153126a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f36d252f140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f36d1d62431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x16c385) [0x5636ac4fa385]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x128ee9) [0x5636ac4b6ee9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0xfd640) [0x5636ac48b640]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x749bd) [0x5636ac4029bd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f36d157dd0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/simd-test(+0x7527a) [0x5636ac40327a]
> =================================
>
> test 28
> Start 28: CompatibilityHelpersTests
>
> 28: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/compat-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/CompatibilityHelpersTests.xml"
> 28: Test timeout computed to be: 30
> 28: [1611165257.813687] [ip-172-31-13-129:15393:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 28: [1611165257.813713] [ip-172-31-13-129:15393:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 28: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 28: MPIR_Init_thread(152).......:
> 28: MPID_Init(597)..............:
> 28: MPIDI_UCX_mpi_init_hook(247):
> 28: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 28: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 28: :
> 28: system msg for write_line failure : Bad file descriptor
> 28: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 28: MPIR_Init_thread(152).......:
> 28: MPID_Init(597)..............:
> 28: MPIDI_UCX_mpi_init_hook(247):
> 28: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 28: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 28: :
> 28: system msg for write_line failure : Bad file descriptor
> 28: [ip-172-31-13-129:15393:0:15393] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 28: ==== backtrace (tid: 15393) ====
> 28: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fecb88a3ea4]
> 28: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fecb88a40af]
> 28: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fecb88a426a]
> 28: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fecb98a2140]
> 28: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fecb90d5431]
> 28: 5 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x1178f5) [0x55ea7d7f98f5]
> 28: 6 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0xd40c9) [0x55ea7d7b60c9]
> 28: 7 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x968d0) [0x55ea7d7788d0]
> 28: 8 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x5cc1d) [0x55ea7d73ec1d]
> 28: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fecb88f0d0a]
> 28: 10 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x5d4da) [0x55ea7d73f4da]
> 28: =================================
> 28/30 Test #28: CompatibilityHelpersTests ........***Exception: SegFault 0.02 sec
> [1611165257.813687] [ip-172-31-13-129:15393:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.813713] [ip-172-31-13-129:15393:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15393:0:15393] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15393) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fecb88a3ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fecb88a40af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fecb88a426a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fecb98a2140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fecb90d5431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x1178f5) [0x55ea7d7f98f5]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0xd40c9) [0x55ea7d7b60c9]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x968d0) [0x55ea7d7788d0]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x5cc1d) [0x55ea7d73ec1d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fecb88f0d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/compat-test(+0x5d4da) [0x55ea7d73f4da]
> =================================
>
> test 29
> Start 29: FileIOTests
>
> 29: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/FileIOTests.xml"
> 29: Test timeout computed to be: 30
> 29: [1611165257.835741] [ip-172-31-13-129:15401:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 29: [1611165257.835763] [ip-172-31-13-129:15401:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 29: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 29: MPIR_Init_thread(152).......:
> 29: MPID_Init(597)..............:
> 29: MPIDI_UCX_mpi_init_hook(247):
> 29: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 29: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 29: :
> 29: system msg for write_line failure : Bad file descriptor
> 29: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 29: MPIR_Init_thread(152).......:
> 29: MPID_Init(597)..............:
> 29: MPIDI_UCX_mpi_init_hook(247):
> 29: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 29: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 29: :
> 29: system msg for write_line failure : Bad file descriptor
> 29: [ip-172-31-13-129:15401:0:15401] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 29: ==== backtrace (tid: 15401) ====
> 29: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fbfa4fa2ea4]
> 29: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fbfa4fa30af]
> 29: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fbfa4fa326a]
> 29: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fbfa5fa7140]
> 29: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fbfa57da431]
> 29: 5 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x179385) [0x55c7e60b8385]
> 29: 6 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0xb9999) [0x55c7e5ff8999]
> 29: 7 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x87900) [0x55c7e5fc6900]
> 29: 8 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x524cd) [0x55c7e5f914cd]
> 29: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fbfa4ff5d0a]
> 29: 10 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x531ca) [0x55c7e5f921ca]
> 29: =================================
> 29/30 Test #29: FileIOTests ......................***Exception: SegFault 0.02 sec
> [1611165257.835741] [ip-172-31-13-129:15401:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.835763] [ip-172-31-13-129:15401:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15401:0:15401] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15401) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7fbfa4fa2ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7fbfa4fa30af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7fbfa4fa326a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7fbfa5fa7140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7fbfa57da431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x179385) [0x55c7e60b8385]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0xb9999) [0x55c7e5ff8999]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x87900) [0x55c7e5fc6900]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x524cd) [0x55c7e5f914cd]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7fbfa4ff5d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/fileio-test(+0x531ca) [0x55c7e5f921ca]
> =================================
>
> test 30
> Start 30: SelectionUnitTests
>
> 30: Test command: /<<PKGBUILDDIR>>/build/mpich/bin/selection-test "--gtest_output=xml:/<<PKGBUILDDIR>>/build/mpich/Testing/Temporary/SelectionUnitTests.xml"
> 30: Test timeout computed to be: 30
> 30: [1611165257.857831] [ip-172-31-13-129:15409:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> 30: [1611165257.857854] [ip-172-31-13-129:15409:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> 30: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 30: MPIR_Init_thread(152).......:
> 30: MPID_Init(597)..............:
> 30: MPIDI_UCX_mpi_init_hook(247):
> 30: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 30: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 30: :
> 30: system msg for write_line failure : Bad file descriptor
> 30: Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> 30: MPIR_Init_thread(152).......:
> 30: MPID_Init(597)..............:
> 30: MPIDI_UCX_mpi_init_hook(247):
> 30: init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> 30: [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> 30: :
> 30: system msg for write_line failure : Bad file descriptor
> 30: [ip-172-31-13-129:15409:0:15409] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> 30: ==== backtrace (tid: 15409) ====
> 30: 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f28302c5ea4]
> 30: 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f28302c60af]
> 30: 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f28302c626a]
> 30: 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f28312ca140]
> 30: 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2830afd431]
> 30: 5 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0x21a835) [0x55af792a7835]
> 30: 6 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0x15b179) [0x55af791e8179]
> 30: 7 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0x12f060) [0x55af791bc060]
> 30: 8 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0xa075d) [0x55af7912d75d]
> 30: 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2830318d0a]
> 30: 10 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0xa145a) [0x55af7912e45a]
> 30: =================================
> 30/30 Test #30: SelectionUnitTests ...............***Exception: SegFault 0.02 sec
> [1611165257.857831] [ip-172-31-13-129:15409:0] rdmacm_cm.c:638 UCX ERROR rdma_create_event_channel failed: No such device
> [1611165257.857854] [ip-172-31-13-129:15409:0] ucp_worker.c:1432 UCX ERROR failed to open CM on component rdmacm with status Input/output error
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> Abort(1615247) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> MPIR_Init_thread(152).......:
> MPID_Init(597)..............:
> MPIDI_UCX_mpi_init_hook(247):
> init_worker(71).............: ucx function returned with failed status(ucx_init.c 71 init_worker Input/output error)
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1615247
> :
> system msg for write_line failure : Bad file descriptor
> [ip-172-31-13-129:15409:0:15409] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
> ==== backtrace (tid: 15409) ====
> 0 /usr/lib/x86_64-linux-gnu/libucs.so.0(ucs_handle_error+0x2a4) [0x7f28302c5ea4]
> 1 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x220af) [0x7f28302c60af]
> 2 /usr/lib/x86_64-linux-gnu/libucs.so.0(+0x2226a) [0x7f28302c626a]
> 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f28312ca140]
> 4 /usr/lib/x86_64-linux-gnu/libmpich.so.12(MPIR_Err_return_comm+0xa1) [0x7f2830afd431]
> 5 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0x21a835) [0x55af792a7835]
> 6 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0x15b179) [0x55af791e8179]
> 7 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0x12f060) [0x55af791bc060]
> 8 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0xa075d) [0x55af7912d75d]
> 9 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7f2830318d0a]
> 10 /<<PKGBUILDDIR>>/build/mpich/bin/selection-test(+0xa145a) [0x55af7912e45a]
> =================================
>
>
> 0% tests passed, 30 tests failed out of 30
>
> Label Time Summary:
> GTest = 0.77 sec*proc (30 tests)
> MpiTest = 0.11 sec*proc (3 tests)
> UnitTest = 0.77 sec*proc (30 tests)
>
> Total Test time (real) = 0.81 sec
>
> The following tests FAILED:
> 1 - TestUtilsUnitTests (SEGFAULT)
> 2 - TestUtilsMpiUnitTests (Failed)
> 3 - UtilityUnitTests (SEGFAULT)
> 4 - UtilityMpiUnitTests (Failed)
> 5 - MdlibUnitTest (SEGFAULT)
> 6 - AppliedForcesUnitTest (SEGFAULT)
> 7 - CommandLineUnitTests (SEGFAULT)
> 8 - DomDecTests (SEGFAULT)
> 9 - EwaldUnitTests (SEGFAULT)
> 10 - FFTUnitTests (SEGFAULT)
> 11 - GpuUtilsUnitTests (SEGFAULT)
> 12 - HardwareUnitTests (SEGFAULT)
> 13 - MathUnitTests (SEGFAULT)
> 14 - MdrunUtilityUnitTests (SEGFAULT)
> 15 - MdrunUtilityMpiUnitTests (Failed)
> 16 - MDSpanTests (SEGFAULT)
> 17 - OnlineHelpUnitTests (SEGFAULT)
> 18 - OptionsUnitTests (SEGFAULT)
> 19 - PbcutilUnitTest (SEGFAULT)
> 20 - RandomUnitTests (SEGFAULT)
> 21 - RestraintTests (SEGFAULT)
> 22 - TableUnitTests (SEGFAULT)
> 23 - TaskAssignmentUnitTests (SEGFAULT)
> 24 - TopologyTest (SEGFAULT)
> 25 - PullTest (SEGFAULT)
> 26 - AwhTest (SEGFAULT)
> 27 - SimdUnitTests (SEGFAULT)
> 28 - CompatibilityHelpersTests (SEGFAULT)
> 29 - FileIOTests (SEGFAULT)
> 30 - SelectionUnitTests (SEGFAULT)
> Errors while running CTest
> make: *** [debian/rules:158: build-mpich] Error 1
The full build log is available from:
http://qa-logs.debian.net/2021/01/20/gromacs_2020.4-2_unstable.log
A list of current common problems and possible solutions is available at
http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!
If you reassign this bug to another package, please marking it as 'affects'-ing
this package. See https://www.debian.org/Bugs/server-control#affects
If you fail to reproduce this, please provide a build log and diff it with me
so that we can identify if something relevant changed in the meantime.
About the archive rebuild: The rebuild was done on EC2 VM instances from
Amazon Web Services, using a clean, minimal and up-to-date chroot. Every
failed build was retried once to eliminate random failures.
More information about the Debichem-devel
mailing list