Bug#907589: asterisk crashes when using PJSIP while processing registrations
Joachim Foerster
JOFT at gmx.de
Wed Aug 29 21:09:48 BST 2018
Package: asterisk
Version: 1:13.14.1~dfsg-2+deb9u3
Severity: important
Tags: upstream
Dear Maintainer,
I'm using Asterisk with its PJSIP backend. Every few hours Asterisk segfaults
in PJSIP library code. According to backtraces of coredumps the segfaults
seem to be related to SIP registration handling. I cannot say where the root
cause is, so I'm reporting this against asterisk and not the PJSIP library.
To work around this problem I'm currently using a self-built version of
upstream Asterisk (built-in PJSIP). From this experience I can say, that
upstream version 13.15.0 does NOT have the described problem (not a single
segfault over months). However I would really like to use standard Debian
stable packages, without self-built stuff.
Details:
Over the course of roughly 24h hours I recently got 13 segfaults. 6 of these
segfaults occured in a function called tx_data_destroy() in libpjsip:
#0 tx_data_destroy (tdata=<optimized out>) at ../src/pjsip/sip_transport.c:485
485 pjsip_endpt_release_pool( tdata->mgr->endpt, tdata->pool );
(gdb) bt
#0 tx_data_destroy (tdata=<optimized out>) at ../src/pjsip/sip_transport.c:485
#1 0x00007f686cb59cc8 in pjsip_tx_data_dec_ref (tdata=0x7f6814005748) at ../src/pjsip/sip_transport.c:501
#2 0x00007f67b22b5740 in registration_response_destroy (obj=0x7f685c000dc0) at res_pjsip_outbound_registration.c:741
#3 0x000055ac1cbe7f39 in internal_ao2_ref (user_data=user_data at entry=0x7f685c000dc0, delta=delta at entry=-1,
file=file at entry=0x55ac1cd4e066 "astobj2.c", line=line at entry=518,
func=func at entry=0x55ac1cd4e158 <__FUNCTION__.9326> "__ao2_ref") at astobj2.c:451
#4 0x000055ac1cbe8528 in __ao2_ref (user_data=user_data at entry=0x7f685c000dc0, delta=delta at entry=-1) at astobj2.c:518
#5 0x00007f67b22b6ffa in handle_registration_response (data=0x7f685c000dc0) at res_pjsip_outbound_registration.c:825
#6 0x000055ac1cd290e8 in ast_taskprocessor_execute (tps=tps at entry=0x55ac1e968ff0) at taskprocessor.c:965
#7 0x000055ac1cd310a0 in execute_tasks (data=0x55ac1e968ff0) at threadpool.c:1322
#8 0x000055ac1cd290e8 in ast_taskprocessor_execute (tps=0x55ac1e39b2c0) at taskprocessor.c:965
#9 0x000055ac1cd30a74 in threadpool_execute (pool=0x55ac1e39ae80) at threadpool.c:351
#10 worker_active (worker=0x7f67e0001a30) at threadpool.c:1105
#11 worker_start (arg=arg at entry=0x7f67e0001a30) at threadpool.c:1024
#12 0x000055ac1cd3908c in dummy_start (data=<optimized out>) at utils.c:1235
#13 0x00007f687358a494 in start_thread (arg=0x7f686e2ae700) at pthread_create.c:333
#14 0x00007f6872194acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
(gdb) list
480 pj_lock_release(tdata->mgr->lock);
481 #endif
482
483 pj_atomic_destroy( tdata->ref_cnt );
484 pj_lock_destroy( tdata->lock );
485 pjsip_endpt_release_pool( tdata->mgr->endpt, tdata->pool );
486 }
(gdb) disassemble
Dump of assembler code for function tx_data_destroy:
0x00007f686cb59c20 <+0>: push %rbx
0x00007f686cb59c21 <+1>: mov %rdi,%rbx
0x00007f686cb59c24 <+4>: callq 0x7f686cb482d0 <pj_log_get_level at plt>
0x00007f686cb59c29 <+9>: cmp $0x4,%eax
0x00007f686cb59c2c <+12>: jle 0x7f686cb59c4b <tx_data_destroy+43>
0x00007f686cb59c2e <+14>: mov %rbx,%rdi
0x00007f686cb59c31 <+17>: callq 0x7f686cb48c70 <pjsip_tx_data_get_info at plt>
0x00007f686cb59c36 <+22>: lea 0x18(%rbx),%rdi
0x00007f686cb59c3a <+26>: lea 0x16701(%rip),%rsi # 0x7f686cb70342
0x00007f686cb59c41 <+33>: mov %rax,%rdx
0x00007f686cb59c44 <+36>: xor %eax,%eax
0x00007f686cb59c46 <+38>: callq 0x7f686cb48100 <pj_log_5 at plt>
0x00007f686cb59c4b <+43>: lea 0x3a8(%rbx),%rdi
0x00007f686cb59c52 <+50>: callq 0x7f686cb48b10 <pjsip_tpselector_dec_ref at plt>
0x00007f686cb59c57 <+55>: mov 0x1b0(%rbx),%rdi
0x00007f686cb59c5e <+62>: callq 0x7f686cb48400 <pj_atomic_destroy at plt>
0x00007f686cb59c63 <+67>: mov 0x180(%rbx),%rdi
0x00007f686cb59c6a <+74>: callq 0x7f686cb48870 <pj_lock_destroy at plt>
0x00007f686cb59c6f <+79>: mov 0x50(%rbx),%rax
0x00007f686cb59c73 <+83>: mov 0x10(%rbx),%rsi
0x00007f686cb59c77 <+87>: pop %rbx
=> 0x00007f686cb59c78 <+88>: mov 0x10(%rax),%rdi
0x00007f686cb59c7c <+92>: jmpq 0x7f686cb48be0 <pjsip_endpt_release_pool at plt>
End of assembler dump.
(gdb) up
#1 0x00007f686cb59cc8 in pjsip_tx_data_dec_ref (tdata=0x7f6814005748) at ../src/pjsip/sip_transport.c:501
501 tx_data_destroy(tdata);
(gdb) print tdata
$1 = (pjsip_tx_data *) 0x7f6814005748
(gdb) print tdata->pool
$2 = (pj_pool_t *) 0x7f6814005645
(gdb) print tdata->mgr
$3 = (pjsip_tpmgr *) 0x554b43415250
(gdb) print tdata->mgr->endpt
Cannot access memory at address 0x554b43415260
It seems like the endpoint struct is gone? But why? Broken pointer? Already free'ed?
Here are the other types of segfaults, which I haven't had a closer look at yet:
2 segfaults occured in function pj_atomic_inc_and_get() in libpj:
(gdb) bt
#0 0x00007fce2dcd4999 in pj_atomic_inc_and_get () from /usr/lib/x86_64-linux-gnu/libpj.so.2
#1 0x00007fcdb878e5a3 in sip_outbound_registration_response_cb (param=0x7fce7467c6e0) at res_pjsip_outbound_registration.c:956
#2 0x00007fce2f250358 in ?? () from /usr/lib/x86_64-linux-gnu/libpjsip-ua.so.2
#3 0x00007fce2f251a2f in ?? () from /usr/lib/x86_64-linux-gnu/libpjsip-ua.so.2
#4 0x00007fce2ee0bb11 in tsx_set_state (tsx=tsx at entry=0x7fce34005988, state=state at entry=PJSIP_TSX_STATE_COMPLETED, event_src_type=event_src_type at entry=PJSIP_EVENT_RX_MSG, event_src=0x7fce180098e8, flag=flag at entry=0) at ../src/pjsip/sip_transaction.c:1234
#5 0x00007fce2ee0d550 in tsx_on_state_proceeding_uac (tsx=0x7fce34005988, event=0x7fce7467ca80) at ../src/pjsip/sip_transaction.c:2958
#6 0x00007fce2ee0d76e in tsx_on_state_calling (tsx=0x7fce34005988, event=0x7fce7467ca80) at ../src/pjsip/sip_transaction.c:2541
#7 0x00007fce2ee0eaef in pjsip_tsx_recv_msg (tsx=tsx at entry=0x7fce34005988, rdata=rdata at entry=0x7fce180098e8) at ../src/pjsip/sip_transaction.c:1788
#8 0x00007fce2ee0ebb5 in mod_tsx_layer_on_rx_response (rdata=0x7fce180098e8) at ../src/pjsip/sip_transaction.c:876
#9 0x00007fce2edf93e6 in pjsip_endpt_process_rx_data (endpt=<optimized out>, rdata=rdata at entry=0x7fce180098e8, p=p at entry=0x7fcded3a8be0 <param>, p_handled=p_handled at entry=0x7fce7467cb94) at ../src/pjsip/sip_endpoint.c:895
#10 0x00007fcded1851bc in distribute (data=0x7fce180098e8) at res_pjsip/pjsip_distributor.c:769
#11 0x000055ef7b8b30e8 in ast_taskprocessor_execute (tps=tps at entry=0x55ef7c9b0e40) at taskprocessor.c:965
#12 0x000055ef7b8bb0a0 in execute_tasks (data=0x55ef7c9b0e40) at threadpool.c:1322
#13 0x000055ef7b8b30e8 in ast_taskprocessor_execute (tps=0x55ef7c101420) at taskprocessor.c:965
#14 0x000055ef7b8baa74 in threadpool_execute (pool=0x55ef7c0ffde0) at threadpool.c:351
#15 worker_active (worker=0x7fcddc00b9e0) at threadpool.c:1105
#16 worker_start (arg=arg at entry=0x7fcddc00b9e0) at threadpool.c:1024
#17 0x000055ef7b8c308c in dummy_start (data=<optimized out>) at utils.c:1235
#18 0x00007fce79959494 in start_thread (arg=0x7fce7467d700) at pthread_create.c:333
#19 0x00007fce78563acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
2 segfaults occured in function pj_pool_alloc_from_block() in libpj:
(gdb) bt
#0 0x00007fdde6efcea4 in pj_pool_alloc_from_block () from /usr/lib/x86_64-linux-gnu/libpj.so.2
#1 0x00007fdde6efd105 in pj_pool_alloc () from /usr/lib/x86_64-linux-gnu/libpj.so.2
#2 0x00007fdde6efd145 in pj_pool_calloc () from /usr/lib/x86_64-linux-gnu/libpj.so.2
#3 0x00007fddfc0550d3 in pj_pool_zalloc (size=288, pool=pool at entry=0x7fdd6400ca18) at ../../pjlib/include/pj/pool.h:476
#4 pjsip_authorization_hdr_create (pool=pool at entry=0x7fdd6400ca18) at ../src/pjsip/sip_auth_msg.c:51
#5 0x00007fddfc0532b0 in auth_respond (req_pool=req_pool at entry=0x7fdd6400ca18, hdr=hdr at entry=0x7fddb8008c00, uri=uri at entry=0x7fdd6400ced0, cred_info=0x7fdd6400c998, method=0x7fdd6400d408, sess_pool=<optimized out>, cached_auth=0x7fdd6400ca48, p_h_auth=0x7fdd3ac205e8) at ../src/pjsip/sip_auth_client.c:694
#6 0x00007fddfc05425c in process_auth (h_auth=0x7fdd3ac205e8, cached_auth=0x7fdd6400ca48, sess=0x7fdd3ac20650, tdata=0x7fdd6400c998, uri=0x7fdd6400ced0, hchal=0x7fddb8008c00, req_pool=0x7fdd6400ca18) at ../src/pjsip/sip_auth_client.c:1115
#7 pjsip_auth_clt_reinit_req (sess=sess at entry=0x7fdd3ac20650, rdata=rdata at entry=0x7fddb8007218, old_request=old_request at entry=0x7fdd6400c998, new_request=new_request at entry=0x7fdd3ac20728) at ../src/pjsip/sip_auth_client.c:1200
#8 0x00007fdd41aa6009 in digest_create_request_with_auth_from_old (auths=<optimized out>, challenge=0x7fddb8007218, old_request=0x7fdd6400c998, new_request=0x7fdd3ac20728) at res_pjsip_outbound_authenticator_digest.c:126
#9 0x00007fdd4189f162 in handle_registration_response (data=0x7fddb8003640) at res_pjsip_outbound_registration.c:811
#10 0x00005651df0160e8 in ast_taskprocessor_execute (tps=tps at entry=0x5651dfdd1690) at taskprocessor.c:965
#11 0x00005651df01e0a0 in execute_tasks (data=0x5651dfdd1690) at threadpool.c:1322
#12 0x00005651df0160e8 in ast_taskprocessor_execute (tps=0x5651e02389b0) at taskprocessor.c:965
#13 0x00005651df01da74 in threadpool_execute (pool=0x5651e0237f30) at threadpool.c:351
#14 worker_active (worker=0x7fdd6c000a80) at threadpool.c:1105
#15 worker_start (arg=arg at entry=0x7fdd6c000a80) at threadpool.c:1024
#16 0x00005651df02608c in dummy_start (data=<optimized out>) at utils.c:1235
#17 0x00007fde02a7a494 in start_thread (arg=0x7fdd3ac21700) at pthread_create.c:333
#18 0x00007fde01684acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
2 segfaults occured in function process_auth() in libpjsip:
(gdb) bt
#0 0x00007f7858a28fab in process_auth (h_auth=0x7f77d83435e8, cached_auth=0x7f7804004698, sess=0x7f77d8343650, tdata=0x7f78040045e8, uri=0x7f7804004b20, hchal=0x7f78200086e0, req_pool=0x7f7804004668) at ../src/pjsip/sip_auth_client.c:1025
#1 pjsip_auth_clt_reinit_req (sess=sess at entry=0x7f77d8343650, rdata=rdata at entry=0x7f7820006cf8, old_request=old_request at entry=0x7f78040045e8, new_request=new_request at entry=0x7f77d8343728) at ../src/pjsip/sip_auth_client.c:1200
#2 0x00007f779e4be009 in digest_create_request_with_auth_from_old (auths=<optimized out>, challenge=0x7f7820006cf8, old_request=0x7f78040045e8, new_request=0x7f77d8343728) at res_pjsip_outbound_authenticator_digest.c:126
#3 0x00007f779e2b7162 in handle_registration_response (data=0x7f7820001c80) at res_pjsip_outbound_registration.c:811
#4 0x00005604f0d320e8 in ast_taskprocessor_execute (tps=tps at entry=0x5604f28b3390) at taskprocessor.c:965
#5 0x00005604f0d3a0a0 in execute_tasks (data=0x5604f28b3390) at threadpool.c:1322
#6 0x00005604f0d320e8 in ast_taskprocessor_execute (tps=0x5604f29697c0) at taskprocessor.c:965
#7 0x00005604f0d39a74 in threadpool_execute (pool=0x5604f2967e00) at threadpool.c:351
#8 worker_active (worker=0x7f77cc0055a0) at threadpool.c:1105
#9 worker_start (arg=arg at entry=0x7f77cc0055a0) at threadpool.c:1024
#10 0x00005604f0d4208c in dummy_start (data=<optimized out>) at utils.c:1235
#11 0x00007f785f44f494 in start_thread (arg=0x7f77d8344700) at pthread_create.c:333
#12 0x00007f785e059acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
I'm not familier and have never hacked on Asterisk/PJSIP code yet, so
I'm happy to be guided by someone who knows the code and where to
look.
-- System Information:
Debian Release: 9.5
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.9.110 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: sysvinit (via /sbin/init)
Versions of packages asterisk depends on:
ii adduser 3.115
ii asterisk-config 1:13.14.1~dfsg-2+deb9u3
ii asterisk-core-sounds-en 1.4.27-1
ii asterisk-modules 1:13.14.1~dfsg-2+deb9u3
ii init-system-helpers 1.48
ii libbsd0 0.8.3-1
ii libc6 2.24-11+deb9u3
ii libcap2 1:2.25-1
ii libedit2 3.1-20160903-3
ii libgcc1 1:6.3.0-18+deb9u1
ii libjansson4 2.9-1
ii libncurses5 6.0+20161126-1+deb9u2
ii libpopt0 1.16-10+b2
ii libsqlite3-0 3.16.2-5+deb9u1
ii libssl1.1 1.1.0f-3+deb9u2
ii libstdc++6 6.3.0-18+deb9u1
ii libsystemd0 232-23
ii libtinfo5 6.0+20161126-1+deb9u2
ii liburiparser1 0.8.4-1
ii libuuid1 2.29.2-1+deb9u1
ii libxml2 2.9.4+dfsg1-2.2+deb9u2
ii libxslt1.1 1.1.29-2.1
ii lsb-base 9.20161125
Versions of packages asterisk recommends:
ii asterisk-moh-opsound-gsm 2.03-1
ii asterisk-voicemail-imapstorage [asterisk-voicemail 1:13.14.1~dfsg-2+deb9u3
ii sox 14.4.1-5+b2
Versions of packages asterisk suggests:
pn asterisk-dahdi <none>
pn asterisk-dev <none>
pn asterisk-doc <none>
pn asterisk-ooh323 <none>
pn asterisk-opus <none>
pn asterisk-vpb <none>
-- no debconf information
More information about the Pkg-voip-maintainers
mailing list