[Syslog-ng-maintainers] Bug#996246: syslog-ng: Spams /var/log/syslog with 100 MB of "Destination timeout has elapsed, closing connection" per day after upgrade from Buster to Bullseye

Tue Oct 12 14:55:33 BST 2021

Package: syslog-ng
Version: 3.28.1-2+b1
Severity: important
Tags: fixed-upstream
Control: forwarded -1 https://github.com/syslog-ng/syslog-ng/issues/3506

Dear Syslog-NG Maintainers,

I've run in an annoying bug with syslog-ng in Debian 11 Bullseye which
is already fixed upstream in version 3.30:

   * What led up to the situation?

Upgrading a high-traffic syslog-ng server from Debian Buster to Bullseye.

   * What exactly did you do (or not do) that was effective (or
     ineffective)?

We did not change any of the configuration except what the package
changes (and which was untouched by us).

   * What was the outcome of this action?

About 100 MB of /var/log/syslog within 24h (the upgrade was
on 13:30 UTC yesterday):

-rw-r----- 1 root adm 25M Oct 12 13:11 /var/log/syslog
-rw-r----- 1 root adm 70M Oct 12 07:00 /var/log/syslog.1

I've found tons of the following lines per minute in /var/log/syslog
despite this host is a sinkhole (i.e. only receives messages and writes
them to disk — one file per minute and per sending host):

Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1134'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1138'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1142'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1147'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1148'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1137'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1135'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1153'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1157'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1160'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1158'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1127'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1161'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1144'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1172'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1168'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1177'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1175'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1170'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1184'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1167'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1192'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1182'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1195'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1186'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1206'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1197'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1208'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1216'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1213'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1227'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1200'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1226'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1209'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1223'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1224'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1229'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1217'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1231'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1239'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1238'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1220'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1241'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1225'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1243'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1245'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1232'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1247'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1252'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1255'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1254'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1256'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1244'
Oct 12 13:11:00 s_src at some-syslog-server syslog-ng[1691]: Destination timeout has elapsed, closing connection; fd='1260'

# zfgrep -c 'Destination timeout has elapsed, closing connection' /var/log/syslog*
/var/log/syslog:232553
/var/log/syslog.1:619093
/var/log/syslog.2.gz:0
/var/log/syslog.3.gz:0
/var/log/syslog.4.gz:0
/var/log/syslog.5.gz:0
/var/log/syslog.6.gz:0
/var/log/syslog.7.gz:0

   * What outcome did you expect instead?

Well, syslog-ng not spamming /var/log/syslog. ;-)

This bug has been reported upstream at
https://github.com/syslog-ng/syslog-ng/issues/3506 and a fix is also
available upstream at https://github.com/syslog-ng/syslog-ng/pull/3444
and incorporated in upstream's 3.30 release.

After fixing this in Debian Unstable by simply uploading the current
upstream release, please also backport the upstream fix from 3.30 for
this issue (https://github.com/syslog-ng/syslog-ng/pull/3444) to 3.28 in
Bullseye as this is a quite annoying issue in the new Debian stable
release.

Note: This bug report was not written on the affected server but on a
workstation running the same syslog-ng version (and same Debian release)
albeit with a different configuration and potentially less
syslog-ng-mod-* packages installed. Then again, the upstream fix is
trivial and seems completely unrelated to configuration or modules, so
it shouldn't really matter.

-- System Information:
Debian Release: 11.1
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable-debug'), (500, 'stable'), (400, 'proposed-updates-debug'), (400, 'proposed-updates')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-8-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)
LSM: AppArmor: enabled

Versions of packages syslog-ng-core depends on:
ii  libc6                      2.31-13+deb11u2
ii  libcap2                    1:2.44-1
ii  libelogind0 [libsystemd0]  246.9.1-1+debian1
ii  libglib2.0-0               2.66.8-1
ii  libivykis0                 0.42.4-1
ii  libjson-c5                 0.15-2
ii  libnet1                    1.1.6+dfsg-3.1
ii  libpcre3                   2:8.39-13
ii  libssl1.1                  1.1.1k-1+deb11u1
ii  libwrap0                   7.6.q-31
ii  lsb-base                   11.1.0
ii  util-linux                 2.36.1-8

Versions of packages syslog-ng-core recommends:
ii  logrotate  3.18.0-2

Versions of packages syslog-ng-core suggests:
pn  syslog-ng-mod-add-contextual-data  <none>
pn  syslog-ng-mod-amqp                 <none>
pn  syslog-ng-mod-examples             <none>
pn  syslog-ng-mod-extra                <none>
pn  syslog-ng-mod-geoip2               <none>
pn  syslog-ng-mod-getent               <none>
pn  syslog-ng-mod-graphite             <none>
pn  syslog-ng-mod-http                 <none>
pn  syslog-ng-mod-map-value-pairs      <none>
pn  syslog-ng-mod-mongodb              <none>
pn  syslog-ng-mod-python               <none>
pn  syslog-ng-mod-rdkafka              <none>
pn  syslog-ng-mod-redis                <none>
pn  syslog-ng-mod-riemann              <none>
pn  syslog-ng-mod-slog                 <none>
pn  syslog-ng-mod-smtp                 <none>
pn  syslog-ng-mod-snmp                 <none>
pn  syslog-ng-mod-sql                  <none>
pn  syslog-ng-mod-stardate             <none>
pn  syslog-ng-mod-stomp                <none>
pn  syslog-ng-mod-xml-parser           <none>