Bug#1109685: libsoup-3.0-0: deadlock with gstreamer1.0-plugins-good 1.26.2-1

[Simon McVittie]

On Tue, 22 Jul 2025 at 13:32:34 +0100, Nick Steel wrote:
>most of Mopidy's value comes from extensions (mostly distributed via PyPI
>etc) and it's these that are exposed to the bug, some are consequently
>rendered unusable

How can this deadlock be reproduced on a Debian system, preferably with 
software from Debian only, or with third-party code if necessary?

GModule uses a recursive lock, so the most obvious reproducers like

     g_module_open_full ("/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstsoup.so", 0, error);

do not deadlock: any reproducer would need to be multithreaded.

In the stack trace linked from 
<https://gitlab.gnome.org/GNOME/libsoup/-/issues/463>, the threads that 
are not idle appear to be:

Thread 1 (Thread 0x7f9414e25bc0 (LWP 737461)):
#0  futex_wait (futex_word=0x7f9414f8aa08 <_rtld_global+2568>, expected=2, private=0) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex at entry=0x7f9414f8aa08 <_rtld_global+2568>, private=0) at lowlevellock.c:49
#2  0x00007f94146a8ec8 in lll_mutex_lock_optimized (mutex=0x7f9414f8aa08 <_rtld_global+2568>) at pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=mutex at entry=0x7f9414f8aa08 <_rtld_global+2568>) at pthread_mutex_lock.c:128
#4  0x00007f94146a1cbe in dlsym_implementation (handle=<optimized out>, name=<optimized out>, dl_caller=<optimized out>) at dlsym.c:52
#5  ___dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:68
#6  0x00007f9413e963f0 in _g_module_symbol (handle=0x55c7f88d3ed0, symbol_name=0x7f94122033b0 "gst_message_get_type") at ../glib/gmodule/gmodule-dl.c:205
#7  g_module_symbol (module=<optimized out>, symbol_name=symbol_name at entry=0x7f94122033b0 "gst_message_get_type", symbol=symbol at entry=0x7fffd0324090) at ../glib/gmodule/gmodule.c:837
#8  0x00007f94132f1e00 in g_typelib_symbol (typelib=0x55c7f860ff40, symbol_name=0x7f94122033b0 "gst_message_get_type", symbol=0x7fffd0324090) at ../gobject-introspection/girepository/gitypelib.c:2522
#9  0x00007f94132ef759 in g_registered_type_info_get_g_type (info=0x55c7f9730610) at ../gobject-introspection/girepository/giregisteredtypeinfo.c:136
#15 0x00007f9412c257c4 in Python Exception <class 'gdb.error'>: value has been optimized out
#16 0x00007f9412086d9f in gst_bus_async_signal_func (bus=0x7f940400c710, message=0x7f9404044340, data=<optimized out>) at ../gstreamer/subprojects/gstreamer/gst/gstbus.c:1286
#17 0x00007f94120875c3 in gst_bus_source_dispatch (source=0x7f9404015bf0, callback=0x7f9412086d40 <gst_bus_async_signal_func>, user_data=0x0) at ../gstreamer/subprojects/gstreamer/gst/gstbus.c:841
#18 0x00007f9412cb087d in g_main_dispatch (context=0x55c7f958a980) at ../glib/glib/gmain.c:3398
#19 0x00007f9412cb1cd7 in g_main_context_dispatch_unlocked (context=0x55c7f958a980) at ../glib/glib/gmain.c:4249
#20 g_main_context_iterate_unlocked (context=0x55c7f958a980, block=block at entry=1, dispatch=dispatch at entry=1, self=<optimized out>) at ../glib/glib/gmain.c:4314
#21 0x00007f9412cb2097 in g_main_loop_run (loop=0x55c7f95a92c0) at ../glib/glib/gmain.c:4516
...

and

Thread 3 (Thread 0x7f940f4486c0 (LWP 737465)):
#0  futex_wait (futex_word=0x55c7f889a520, expected=2, private=0) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex at entry=0x55c7f889a520, private=0) at lowlevellock.c:49
#2  0x00007f94146a8ec8 in lll_mutex_lock_optimized (mutex=0x55c7f889a520) at pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x55c7f889a520) at pthread_mutex_lock.c:128
#4  0x00007f9412ce11ae in g_rec_mutex_lock_impl (mutex=0x7f9413e9a070 <g_module_global_lock>) at ../glib/glib/gthread-posix.c:253
#5  0x00007f9413e96576 in g_module_open_full (file_name=file_name at entry=0x0, flags=flags at entry=0, error=error at entry=0x0) at ../glib/gmodule/gmodule.c:478
--Type <RET> for more, q to quit, c to continue without paging--
#6  0x00007f9413e9707c in g_module_open (file_name=file_name at entry=0x0, flags=flags at entry=0) at ../glib/gmodule/gmodule.c:708
#7  0x00007f93cc73f488 in soup2_is_loaded () at ../libsoup/libsoup/soup-init.c:26
#8  soup_init () at ../libsoup/libsoup/soup-init.c:54
#9  soup_init_ctor () at ../libsoup/libsoup/soup-init.c:96
#10 0x00007f9414f582f7 in call_init (l=<optimized out>, argc=2, argv=0x7fffd03252c8, env=0x7fffd03252e0) at dl-init.c:74
#11 call_init (l=<optimized out>, argc=2, argv=0x7fffd03252c8, env=0x7fffd03252e0) at dl-init.c:26
#12 0x00007f9414f583cd in _dl_init (main_map=0x7f9404052090, argc=2, argv=0x7fffd03252c8, env=0x7fffd03252e0) at dl-init.c:121
#13 0x00007f9414f554b5 in __GI__dl_catch_exception (exception=exception at entry=0x0, operate=operate at entry=0x7f9414f5f160 <call_dl_init>, args=args at entry=0x7f940f446850) at dl-catch.c:215
#14 0x00007f9414f5f0c9 in dl_open_worker (a=a at entry=0x7f940f446850) at dl-open.c:799
#15 0x00007f9414f55416 in __GI__dl_catch_exception (exception=exception at entry=0x7f940f446830, operate=operate at entry=0x7f9414f5f000 <dl_open_worker>, args=args at entry=0x7f940f446850) at dl-catch.c:241
#16 0x00007f9414f5f4de in _dl_open (file=0x7f940c4cfc14 "libsoup-3.0.so.0", mode=-2147483390, caller_dlopen=0x7f940c4c241f <soup_element_init+191>, nsid=-2, argc=2, argv=0x7fffd03252c8, env=0x7fffd03252e0) at dl-open.c:874
#17 0x00007f94146a1b34 in dlopen_doit (a=a at entry=0x7f940f446b00) at dlopen.c:56
#18 0x00007f9414f55416 in __GI__dl_catch_exception (exception=exception at entry=0x7f940f446a40, operate=0x7f94146a1ad0 <dlopen_doit>, args=0x7f940f446b00) at dl-catch.c:241
#19 0x00007f9414f55569 in _dl_catch_error (objname=0x7f940f446aa8, errstring=0x7f940f446ab0, mallocedp=0x7f940f446aa7, operate=<optimized out>, args=<optimized out>) at dl-catch.c:260
#20 0x00007f94146a1623 in _dlerror_run (operate=operate at entry=0x7f94146a1ad0 <dlopen_doit>, args=args at entry=0x7f940f446b00) at dlerror.c:138
#21 0x00007f94146a1beb in dlopen_implementation (file=<optimized out>, mode=<optimized out>, dl_caller=<optimized out>) at dlopen.c:71
#22 ___dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:81
#23 0x00007f940c4c241f in gst_soup_load_library () at ../gstreamer/subprojects/gst-plugins-good/ext/soup/gstsouploader.c:196
#24 soup_element_init (plugin=0x55c7f89ac750) at ../gstreamer/subprojects/gst-plugins-good/ext/soup/gstsoupelement.c:60
#25 0x00007f940c4c5ea1 in souphttpsrc_element_init (plugin=0x55c7f89ac750) at ../gstreamer/subprojects/gst-plugins-good/ext/soup/gstsouphttpsrc.c:2803
...

meaning that thread 3 is holding the libdl lock to load and initialize 
libsoup3, then entering g_module_open_full(), while thread 1 is holding 
the GModule lock to look up a symbol (possibly in some different 
module), then entering dlsym().

I can't see an obvious way to make this happen on-demand: both of those 
operations should normally be quick, so I would expect that you would 
have to be unlucky with timing for the deadlock to appear. I'll see 
whether I can construct an artifical reproducer with a module that is 
intentionally slow to initialize, or intentionally spams GModule calls, 
or something like that...

A likely workaround would be to load some of the relevant modules and/or 
libraries early in the process's lifetime, from single-threaded code, so 
that they cannot possibly be involved in a deadlock.

     smcv