[Syslog-ng-maintainers] Bug#970355: syslog-ng : segfault at 0 ip 00007fa46a2bf7b2 sp 00007ffed353fb30 error 6 in libsyslog-ng-3.19.so.0.0.0[7fa46a2a9000+5a000]

Wed Sep 30 15:15:10 BST 2020

On Wed, 16 Sep 2020 10:16:49 +0200 SZALAY Attila <sasa at debian.org> wrote:
> Hi Jean-Marc,
> 
> Please check if a core file is available related to the segmentation
> fault. If there is any please make it available for me/us.
> 
> Also, can you run syslog-ng-debun with the -r parameter and send the
> generated report bundle?
> 
> Another question, is the segmentation fault reproducible? Is syslog-ng
> crashing frequently?

Dear Maintainer, hello Jean-Marc,
I tried to get some more information from the kernel message,
but found just that it points to this function [1].
There I assume that the argument s is a null pointer.

Bug I fear that without a proper backtrace this might not yet enough to fix the fault.

For getting a coredump or using gdb you might have a look at [2].
For the latter you might want to first install the package syslog-ng-dbg.

Kind regards,
Bernhard

[1] https://sources.debian.org/src/syslog-ng/3.19.1-5/lib/logproto/logproto-server.h/#L163

[2] https://wiki.debian.org/HowToGetABacktrace#Core_dump
    https://wiki.debian.org/HowToGetABacktrace#Installing_the_debugging_symbols
-------------- next part --------------

From submitter:
2020-09-15T08:25:53+02:00 s_dev_kernel_kmsg at asus-190 kernel: 6,35313,87037029084,-;syslog-ng[10311]: segfault at 0 ip 00007fa46a2bf7b2 sp 00007ffed353fb30 error 6 in libsyslog-ng-3.19.so.0.0.0[7fa46a2a9000+5a000]
2020-09-15T08:25:53+02:00 s_dev_kernel_kmsg at asus-190 kernel: 6,35314,87037029092,-;Code: e8 c3 9b fe ff 4c 89 e7 e8 eb dc fe ff 48 89 c7 e8 23 ca fe ff e9 15 ff ff ff 66 0f 1f 44 00 00 48 8b 83 f0 00 00 00 48 89 df <c7> 00 00 00 00 00 48 83 c4 08 5b 5d 41 5c 41 5d e9 b9 fc ff ff 66

https://wiki.debian.org/InterpretingKernelOutputAtProcessCrash

error 6:
    0: no page found
    1: write access
    1: user-mode access

echo -n "find /b ..., ..., 0x" && \
echo "e8 c3 9b fe ff 4c 89 e7 e8 eb dc fe ff 48 89 c7 e8 23 ca fe ff e9 15 ff ff ff 66 0f 1f 44 00 00 48 8b 83 f0 00 00 00 48 89 df <c7> 00 00 00 00 00 48 83 c4 08 5b 5d 41 5c 41 5d e9 b9 fc ff ff 66" \
 | sed 's/[<>]//g' | sed 's/ /, 0x/g'

############

# Buster/stable amd64 qemu VM 2020-09-30

apt update
apt dist-upgrade

apt install systemd-coredump mc gdb syslog-ng syslog-ng-dbg

gdb -q

set width 0
set pagination off
file /usr/sbin/syslog-ng
tb main
run
find /b 0x00007ffff7f37390, 0x00007ffff7f8bf80, 0xe8, 0xc3, 0x9b, 0xfe, 0xff, 0x4c, 0x89, 0xe7, 0xe8, 0xeb, 0xdc, 0xfe, 0xff, 0x48, 0x89, 0xc7, 0xe8, 0x23, 0xca, 0xfe, 0xff, 0xe9, 0x15, 0xff, 0xff, 0xff, 0x66, 0x0f, 0x1f, 0x44, 0x00, 0x00, 0x48, 0x8b, 0x83, 0xf0, 0x00, 0x00, 0x00, 0x48, 0x89, 0xdf, 0xc7, 0x00, 0x00, 0x00, 0x00, 0x00, 0x48, 0x83, 0xc4, 0x08, 0x5b, 0x5d, 0x41, 0x5c, 0x41, 0x5d, 0xe9, 0xb9, 0xfc, 0xff, 0xff, 0x66
b * (0x7ffff7f48788 + 42)
disassemble /r 0x7ffff7f48788, 0x7ffff7f48788 + 62

benutzer at debian:~$ gdb -q
(gdb) set width 0
(gdb) set pagination off
(gdb) file /usr/sbin/syslog-ng
Reading symbols from /usr/sbin/syslog-ng...Reading symbols from /usr/lib/debug/.build-id/53/1963ce8fea48fe705285a1a6f41e34c1fedb6d.debug...done.
done.
(gdb) tb main
Temporary breakpoint 1 at 0x2310: file ../../syslog-ng/main.c, line 207.
(gdb) run
Starting program: /usr/sbin/syslog-ng 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Temporary breakpoint 1, main (argc=1, argv=0x7fffffffe618) at ../../syslog-ng/main.c:207
207     ../../syslog-ng/main.c: Datei oder Verzeichnis nicht gefunden.
(gdb) info target
...
        0x00007ffff7f37390 - 0x00007ffff7f8bf80 is .text in /usr/lib/syslog-ng/libsyslog-ng-3.19.so.0
...
(gdb) find /b 0x00007ffff7f37390, 0x00007ffff7f8bf80, 0xe8, 0xc3, 0x9b, 0xfe, 0xff, 0x4c, 0x89, 0xe7, 0xe8, 0xeb, 0xdc, 0xfe, 0xff, 0x48, 0x89, 0xc7, 0xe8, 0x23, 0xca, 0xfe, 0xff, 0xe9, 0x15, 0xff, 0xff, 0xff, 0x66, 0x0f, 0x1f, 0x44, 0x00, 0x00, 0x48, 0x8b, 0x83, 0xf0, 0x00, 0x00, 0x00, 0x48, 0x89, 0xdf, 0xc7, 0x00, 0x00, 0x00, 0x00, 0x00, 0x48, 0x83, 0xc4, 0x08, 0x5b, 0x5d, 0x41, 0x5c, 0x41, 0x5d, 0xe9, 0xb9, 0xfc, 0xff, 0xff, 0x66
0x7ffff7f48788 <log_reader_work_finished+232>
1 pattern found.
(gdb) b * (0x7ffff7f48788 + 42)
Breakpoint 2 at 0x7ffff7f487b2: file ../../lib/logproto/logproto-server.h, line 163.
(gdb) info b
Num     Type           Disp Enb Address            What
2       breakpoint     keep y   0x00007ffff7f487b2 in log_proto_server_reset_error at ../../lib/logproto/logproto-server.h:163
(gdb) disassemble /r 0x7ffff7f48788, 0x7ffff7f48788 + 62
Dump of assembler code from 0x7ffff7f48788 to 0x7ffff7f487c6:
   0x00007ffff7f48788 <log_reader_work_finished+232>:   e8 c3 9b fe ff          callq  0x7ffff7f32350 <g_cond_signal at plt>
   0x00007ffff7f4878d <log_reader_work_finished+237>:   4c 89 e7                mov    %r12,%rdi
   0x00007ffff7f48790 <log_reader_work_finished+240>:   e8 eb dc fe ff          callq  0x7ffff7f36480 <g_static_mutex_get_mutex_impl at plt>
   0x00007ffff7f48795 <log_reader_work_finished+245>:   48 89 c7                mov    %rax,%rdi
   0x00007ffff7f48798 <log_reader_work_finished+248>:   e8 23 ca fe ff          callq  0x7ffff7f351c0 <g_mutex_unlock at plt>
   0x00007ffff7f4879d <log_reader_work_finished+253>:   e9 15 ff ff ff          jmpq   0x7ffff7f486b7 <log_reader_work_finished+23>
   0x00007ffff7f487a2 <log_reader_work_finished+258>:   66 0f 1f 44 00 00       nopw   0x0(%rax,%rax,1)
   0x00007ffff7f487a8 <log_reader_work_finished+264>:   48 8b 83 f0 00 00 00    mov    0xf0(%rbx),%rax
   0x00007ffff7f487af <log_reader_work_finished+271>:   48 89 df                mov    %rbx,%rdi
   0x00007ffff7f487b2 <log_reader_work_finished+274>:   c7 00 00 00 00 00       movl   $0x0,(%rax)               <<<<<<<<
   0x00007ffff7f487b8 <log_reader_work_finished+280>:   48 83 c4 08             add    $0x8,%rsp
   0x00007ffff7f487bc <log_reader_work_finished+284>:   5b                      pop    %rbx
   0x00007ffff7f487bd <log_reader_work_finished+285>:   5d                      pop    %rbp
   0x00007ffff7f487be <log_reader_work_finished+286>:   41 5c                   pop    %r12
   0x00007ffff7f487c0 <log_reader_work_finished+288>:   41 5d                   pop    %r13
   0x00007ffff7f487c2 <log_reader_work_finished+290>:   e9 b9 fc ff ff          jmpq   0x7ffff7f48480 <log_reader_update_watches>
End of assembler dump.
(gdb) ptype /o LogProtoServer
type = struct _LogProtoServer {
/*    0      |     4 */    LogProtoStatus status;
/* XXX  4-byte hole */
/*    8      |     8 */    const LogProtoServerOptions *options;
/*   16      |     8 */    LogTransport *transport;
/*   24      |     8 */    AckTracker *ack_tracker;
/*   32      |    16 */    LogProtoServerWakeupCallback wakeup_callback;
/*   48      |     8 */    gboolean (*is_position_tracked)(LogProtoServer *);
/*   56      |     8 */    LogProtoPrepareAction (*prepare)(LogProtoServer *, GIOCondition *, gint *);
/*   64      |     8 */    gboolean (*restart_with_state)(LogProtoServer *, PersistState *, const gchar *);
/*   72      |     8 */    LogProtoStatus (*fetch)(LogProtoServer *, const guchar **, gsize *, gboolean *, LogTransportAuxData *, Bookmark *);
/*   80      |     8 */    gboolean (*validate_options)(LogProtoServer *);
/*   88      |     8 */    gboolean (*handshake_in_progess)(LogProtoServer *);
/*   96      |     8 */    LogProtoStatus (*handshake)(LogProtoServer *);
/*  104      |     8 */    void (*free_fn)(LogProtoServer *);

                           /* total size (bytes):  112 */
                         }

https://sources.debian.org/src/syslog-ng/3.19.1-5/lib/logproto/logproto-server.h/#L163

160 static inline void
161 log_proto_server_reset_error(LogProtoServer *s)
162 {
163   s->status = LPS_SUCCESS;
164 }