[Freedombox-discuss] Matrix-synapse not working since last reboot, and how to fix it.

[A. F. Cano]

Hi,

The latest reboot of the Freedombox was on:

2022-06-25 02:02:41,330 - synapse.handlers.presence - 748 - INFO - presence.on_shutdown-0- Performing _on_shutdown. Persisting 1 unpersisted changes
2022-06-25 02:02:41,331 - synapse.handlers.presence - 761 - INFO - presence.on_shutdown-0- Finished _on_shutdown
2022-06-25 02:02:41,334 - synapse.app._base - 428 - INFO - sentinel- Shutting down...

and the matrix server shut down.

When I tried to test the latest Element it said "connection to server
has been lost" and sure enough, it wasn't running.

$ sudo /etc/init.d/matrix-synapse start
Starting matrix-synapse (via systemctl): matrix-synapse.serviceJob for matrix-synapse.service failed because the control process exited with error code.
See "systemctl status matrix-synapse.service" and "journalctl -xe" for details.
failed!

$ systemctl status matrix-synapse.service
● matrix-synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/matrix-synapse.service.d
             └─freedombox.conf
     Active: activating (start-pre) since Sun 2022-06-26 18:52:55 EDT; 850ms ago
       Docs: file:/usr/share/doc/matrix-synapse/README.rst
Cntrl PID: 100752 (python3)
      Tasks: 1 (limit: 4641)
     Memory: 12.7M
        CPU: 820ms
     CGroup: /system.slice/matrix-synapse.service
             └─100752 /usr/bin/python3 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate

$ sudo journalctl -xe

Says partially:

Jun 26 18:54:59 fbx python3[100847]: ERROR:root:Synapse 1.57.1 needs matrix-common~=1.1.0, but got matrix-common==1.2.1
Jun 26 18:54:59 fbx python3[100847]: Missing Requirements: "matrix-common"
Jun 26 18:54:59 fbx python3[100847]: To install run:
Jun 26 18:54:59 fbx python3[100847]:     pip install --upgrade --force "matrix-common"
Jun 26 18:54:59 fbx systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

so it looks like an incompatible version of python.

Also, cockpit reports that the "Network Manager Wait Online" service
"Failed to start".  Probably unrelated.  Checked for upgrades and there
were 2 security upgrades.  Installed them, rebooted and that made the
"Failed to start" message disappear.  After the reboot, the message
about matrix-synapse was:

Jun 26 19:14:55 fbx sshd[3623]: Received disconnect from 120.48.47.189 port 36394:11: Bye Bye [preauth]
Jun 26 19:14:55 fbx sshd[3623]: Disconnected from authenticating user root 120.48.47.189 port 36394 [preauth]
Jun 26 19:14:57 fbx python3[3627]: ERROR:root:Synapse 1.57.1 needs matrix-common~=1.1.0, but got matrix-common==1.2.1
Jun 26 19:14:57 fbx python3[3627]: Missing Requirements: "matrix-common"
Jun 26 19:14:57 fbx python3[3627]: To install run:
Jun 26 19:14:57 fbx python3[3627]:     pip install --upgrade --force "matrix-common"

There is no such package/command installed, so started up aptitude and
found that the metapackage pip points to python3-pip.  It has 19
dependencies and 4 suggestions.  After playing a bit with aptitude and
noticing that matrix-synapse can be upgraded, I deselected pip and all
its dependencies.  In the end matrix-synapse is being upgraded along
with python3-unpaddedbase64, along with debian-keyring being installed.

And this seems to have solved it.  Matrix-synapse is again running:

matrix-synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled; vendor preset: enabled)
    Drop-In: /usr/lib/systemd/system/matrix-synapse.service.d
             └─freedombox.conf
     Active: active (running) since Sun 2022-06-26 19:32:09 EDT; 55s ago
       Docs: file:/usr/share/doc/matrix-synapse/README.rst
    Process: 5044 ExecStartPre=/usr/bin/python3 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
   Main PID: 5059 (python3)
      Tasks: 7 (limit: 4641)
     Memory: 77.6M
        CPU: 19.500s
     CGroup: /system.slice/matrix-synapse.service
             └─5059 /usr/bin/python3 -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d

It is possible that the upgrades would have caught up eventually with
the auto-upgrades, but in case anyone encounters this issue.  This is
how to solve it.