[Syslog-ng-maintainers] Bug#986748: syslog-ng: stucks on writev(), mostly around nightly rotate and cause _everything_ to stuck until killed

[Peter Gervai]

Package: syslog-ng
Version: 3.28.1-2
Severity: important
Tags: upstream

I sincerely apologise since this is not a good bugreport, so if you really
feel like completely clueless you can close it, but the problem will
probably presist.

The environments use SysV init (none of the systems are systemd-infected).
(It is possible that it have happened on a systemd-infected server, too,
but I am not sure now as it was many weeks before and back then I was not
aware about the cause of the problem.)

What happens: 

* usually around 03:26 (cron.daily / logrotate) syslog-ng gets stuck on a
writev() call, which is supposed to write the actual incoming log line into
a physical file on the system. strace shows that syslog-ng gets signals 
(like HUP or TERM), handles them and ignores them and go back to writev().

* during this time the syslog service blocks (after a while). 

* this in turn blocks everything using syslog service, and I mean everything.
Notable mentions are dhcp server, vpn server, various cron services. (Blocking
cron may also mean system load goes sky high due to many cron spawns running at once.)

* the disk is not full. 

* HUP, TERM and normal signals get ignored. ILL or KILL kills the daemon and the
system starts working again as syslog-ng gets respawned.

Log rotation is done using 'invoke-rc.d syslog-ng reload' which uses 
`start-stop-daemon` to send SIGNAL1 (SIGHUP) then uses 
`syslog-ng-ctl stats` to see when the daemon reports back, which never happens:
the call gets stuck as well.
[I would guess SIGHUP handler runs amok.]

This seems to happen in 3.19.1-5 and 3.28.1-2 as well, but only on the fraction
of the servers, but there it is recurring around weekly. Nothing relevant is in
dmesg or elsewhere. :-(


-- System Information:
Debian Release: bullseye/sid
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.8.6 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_FIRMWARE_WORKAROUND, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF8, LC_CTYPE=en_US.UTF8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/bash
Init: sysvinit (via /sbin/init)
LSM: AppArmor: enabled

Versions of packages syslog-ng depends on:
ii  syslog-ng-core         3.28.1-2+b1
ii  syslog-ng-mod-mongodb  3.28.1-2+b1
ii  syslog-ng-mod-sql      3.28.1-2+b1

Versions of packages syslog-ng recommends:
pn  syslog-ng-mod-add-contextual-data  <none>
pn  syslog-ng-mod-amqp                 <none>
pn  syslog-ng-mod-examples             <none>
pn  syslog-ng-mod-extra                <none>
pn  syslog-ng-mod-geoip2               <none>
pn  syslog-ng-mod-getent               <none>
pn  syslog-ng-mod-graphite             <none>
pn  syslog-ng-mod-http                 <none>
pn  syslog-ng-mod-map-value-pairs      <none>
pn  syslog-ng-mod-python               <none>
pn  syslog-ng-mod-rdkafka              <none>
pn  syslog-ng-mod-redis                <none>
pn  syslog-ng-mod-riemann              <none>
pn  syslog-ng-mod-slog                 <none>
pn  syslog-ng-mod-smtp                 <none>
pn  syslog-ng-mod-snmp                 <none>
pn  syslog-ng-mod-stardate             <none>
pn  syslog-ng-mod-stomp                <none>
pn  syslog-ng-mod-xml-parser           <none>

syslog-ng suggests no packages.

-- no debconf information