[Pkg-matrix-maintainers] Bug#920339: [PATCH] Re: Bug#920339: matrix-synapse: installation process hangs with unknown reason
Axel Beckert
abe at debian.org
Sat Jun 1 16:15:48 BST 2019
Control: retitle -1 matrix-synapse: installation process hangs under sysvinit due to matrix-synapse daemon not closing file descriptor from debconf
Control: tag -1 + patch
Hi again,
Axel Beckert wrote:
> > I tried to reproduce it with sysvinit and still failed :(
>
> I just was able to reproduce it by installing matrix-synapse on a
> Raspberry Pi running Debian Sid with sysvinit as init system and it
> hangs at the configuration stage after "Config is missing
> macaroon_secret_key". (This warning is though a red herring as
> upstream withholds the fact that it will automatically generate a key
> in this case.)
[...]
> It now hangs there for about an hour.
Contents of /var/log/matrix-synapse/homeserver.log:
2019-06-01 13:01:28,548 - root - 209 - WARNING - None- ***** STARTING SERVER *****
2019-06-01 13:01:28,745 - root - 212 - WARNING - None- Server /usr/lib/python3/dist-packages/synapse/app/homeserver.py version 0.99.2
2019-06-01 13:01:28,748 - root - 214 - INFO - None- Server hostname: c3pio.deuxchevaux.org
2019-06-01 13:01:29,523 - synapse.app.homeserver - 242 - WARNING - None- Starting daemon.
2019-06-01 13:01:29,548 - twisted - 240 - WARNING - - /usr/lib/python3/dist-packages/synapse/config/tls.py:300: builtins.UserWarning: Self-signed TLS certificates will not be accepted by Synapse 1.0. Please either provide a valid certificate, or use Synapse's ACME support to provision one.
The time stamps are all from the time of installation.
>From htop:
9500 root 20 0 7684 2608 1352 S 0.0 0.3 25:57.98 ├─ SCREEN -RdU
9501 root 20 0 7264 2512 1772 S 0.0 0.3 0:00.65 │ ├─ bash
2658 root 20 0 178M 125M 49080 S 0.0 13.6 1:24.26 │ │ └─ aptitude -u
4501 root 20 0 12616 7496 1684 S 0.0 0.8 0:00.40 │ │ ├─ dpkg --status-fd 66 --configure --pending
4970 root 20 0 19836 13500 3604 S 0.0 1.4 0:01.32 │ │ │ ├─ perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/matrix-synapse.postinst configure
4502 root 20 0 1468 380 332 S 0.0 0.0 0:00.00 │ │ │ └─ sh -c (test -x /usr/lib/needrestart/dpkg-status && /usr/lib/needrestart/dpkg-status || cat > /dev/null)
4503 root 20 0 1468 68 0 S 0.0 0.0 0:00.00 │ │ │ └─ sh -c (test -x /usr/lib/needrestart/dpkg-status && /usr/lib/needrestart/dpkg-status || cat > /dev/null)
4504 root 20 0 1468 380 332 S 0.0 0.0 0:00.02 │ │ │ └─ sh /usr/lib/needrestart/dpkg-status
3499 root 20 0 178M 125M 49080 S 0.0 13.6 0:00.00 │ │ └─ aptitude -u
[...]
5066 matrix-sy 20 0 69700 46988 6964 S 0.0 5.0 0:19.14 ├─ python3 -m synapse.app.homeserver --config-path /etc/matrix-synapse/homeserver.yaml --config-path /etc/matrix-synapse/conf.d/ --daemonize
5067 matrix-sy 20 0 69700 46988 6964 S 0.0 5.0 0:02.13 │ └─ python3 -m synapse.app.homeserver --config-path /etc/matrix-synapse/homeserver.yaml --config-path /etc/matrix-synapse/conf.d/ --daemonize
So synapse has already forked, the init script seems to have exited
already and even the postinst script has exited already. The question
is what causes debconf to wait?
"ps auxwwwf | fgrep -1 matrix" shows that there's a zombie process
leftover as (former) child of the debconf process:
root 4970 0.0 1.4 19836 13504 pts/0 S+ 13:00 0:01 | \_ /usr/bin/perl -w /usr/share/debconf/frontend /var/lib/dpkg/info/matrix-synapse.postinst configure
root 4977 0.0 0.0 0 0 pts/0 Z+ 13:00 0:00 | \_ [matrix-synapse.] <defunct>
I suspect — based on the name part "matrix-synapse." that this zombie
was eitherr the matrix-synapse.postinst script or the
matrix-synapse.config script.
Looking with lsof what's still opened by the debconf frontend process
(pid 4970), I noticed these two pipes:
frontend 4970 root 7w FIFO 0,8 0t0 1899008614 pipe
frontend 4970 root 8r FIFO 0,8 0t0 1899008615 pipe
Grepping through the whole process space, I noticed that at least one
of these pipes is connected to the python3 process running
matrix-synapse which looks odd to me:
# lsof | egrep '189900861[45]'
frontend 4970 root 7w FIFO 0,8 0t0 1899008614 pipe
frontend 4970 root 8r FIFO 0,8 0t0 1899008615 pipe
python3 5066 matrix-synapse 3w FIFO 0,8 0t0 1899008615 pipe
python3 5066 5067 python3 matrix-synapse 3w FIFO 0,8 0t0 1899008615 pipe
And indeed, calling "service matrix-synapse stop" on another console
makes the hanging install process to continue.
So it seems that matrix-synapse's --daemonize doesn't properly close
file descriptors upon forking — which is likely an upstream issue.
With systemd, systemd probably covers that. With init scripts we have
start-stop-daemon to fix that, so let's do that:
diff --git a/init.d/matrix-synapse b/init.d/matrix-synapse
index c836728..6c4988b 100755
--- a/init.d/matrix-synapse
+++ b/init.d/matrix-synapse
@@ -103,8 +103,8 @@ do_start()
chown $USER:nogroup $SHAREDIR/media/
chown $USER:nogroup $SHAREDIR/uploads/
- start-stop-daemon --start --pidfile $PIDFILE --chuid $USER \
- --exec $PYTHON -- -m "synapse.app.homeserver" $CONFIGS --daemonize || return 2
+ start-stop-daemon --start --background --pidfile $PIDFILE --chuid $USER \
+ --exec $PYTHON -- -m "synapse.app.homeserver" $CONFIGS || return 2
return 0
}
This uses start-stop-daemon's --background instead of matrix-synapse's
--daemonize.
Without this modification, "dpkg-reconfigure matrix-synapse" hangs as
upon installation while with the patch, "dpkg-reconfigure
matrix-synapse" exits properly.
Regards from the Hack4Glarus.ch Hackathon, Axel
--
,''`. | Axel Beckert <abe at debian.org>, https://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
`- | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE
More information about the Pkg-matrix-maintainers
mailing list