Bug#923508: systemd not restarting sshd on Debian 9 Stretch i386 due to RestartPreventExitStatus=255 in sshd.service file

[Andrew Roberts]

Package: systemd
Version: 232-25+deb9u

On Debian 9 stretch with the latest updates systemd no longer restarts 
sshd due to changes to the sshd.service unit file.

The line:

RestartPreventExitStatus=255 has been added to the master sshd unit file 
in recent systemd/sshd updates.

This means if the network interface is not up by the time sshd tries to 
start, sshd will return a result code of 255, and systemd will NOT try 
to restart it.

This is obviously a HUGE problem for anybody using sshd as their 
exclusive means of accessing a system. On many systems the network can 
take a considerable amount of time to start.

When you finally do get physical access to a system, and try to override 
the default unit file by adding:

RestartPreventExitStatus=

to the end of /etc/systemd/system/sshd.service.d/override.conf

and running systemctl daemon-reload and rebooting

You find that the default of 255 is NOT overridden. I already had:

[Service]

Restart=always

RestartSec=5

in the override.conf file. What part of ALWAYS does systemd not understand!

I was forced to comment out the RestartPreventExitStatus=255 line in the 
/etc/systemd/system/sshd.service file, which is NOT a solution.

Also the version of systemd currently being used has bugs which when you 
manually restart a unit using systemctl restart sshd.service it warns 
about the configuration needing reloading using systemctl daemon-reload, 
even though you have just done that. This bug appears to have been fixed 
once in systemd in 2016, so either an old version is being used, or it 
has crept back in.

systemctl daemon-reload

systemctl restart sshd.service

Warning: sshd.service changed on disk. Run 'systemctl daemon-reload' to 
reload units.

Of course if systemd didn't start networking dependent units until the 
network was actually up then none of this would be an issue... Of course 
then you get into arguments about which network interface...

Clearly much more testing of systemd needs to be done before pushing 
updates of it. But the bottom line is NEVER break sshd.

Systemd Version:

||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  systemd        232-25+deb9u i386         system and service manager

Openssh Server version:

||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  openssh-server 1:7.4p1-10+d i386         secure shell (SSH) server, 
for se

System Version:

uname -a
Linux sempron 4.9.0-8-686-pae #1 SMP Debian 4.9.144-3.1 (2019-02-19) 
i686 GNU/Linux

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"

apt show libc6 | grep ^Version

Version: 2.24-11+deb9u4

Error messages from sshd when booting using default config 
(RestartPreventExitStatus=255 in /etc/systemd/system/sshd.service file)

journalctl -b0 | grep ssh

Mar 01 07:32:02 sempron sshd[432]: error: Bind to port XXXX on 
192.168.1.54 failed: Cannot assign requested address.
Mar 01 07:32:02 sempron sshd[432]: fatal: Cannot bind any address.
Mar 01 07:32:02 sempron systemd[1]: ssh.service: Main process exited, 
code=exited, status=255/n/a
Mar 01 07:32:02 sempron systemd[1]: ssh.service: Unit entered failed state.
Mar 01 07:32:02 sempron systemd[1]: ssh.service: Failed with result 
'exit-code'.
Mar 01 07:32:17 sempron systemd[700]: Listening on GnuPG cryptographic 
agent (ssh-agent emulation).

where XXXX is the sshd port number

/etc/systemd/system/sshd.service file used for above log file:

[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
#Restart=on-failure
Restart=always
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service

/etc/systemd/system/sshd.service.d/override.conf

[Unit]
After=network.target

[Service]
Restart=always
RestartSec=5
RestartPreventExitStatus=

Commenting out RestartPreventExitStatus=255 in sshd.service, doing 
systemctl daemon-reload, and rebooting gives:

journalctl -b0 | grep ssh

Mar 01 07:38:24 sempron sshd[426]: error: Bind to port XXXX on 
192.168.1.54 failed: Cannot assign requested address.
Mar 01 07:38:24 sempron sshd[426]: fatal: Cannot bind any address.
Mar 01 07:38:24 sempron systemd[1]: ssh.service: Main process exited, 
code=exited, status=255/n/a
Mar 01 07:38:24 sempron systemd[1]: ssh.service: Unit entered failed state.
Mar 01 07:38:24 sempron systemd[1]: ssh.service: Failed with result 
'exit-code'.
Mar 01 07:38:24 sempron systemd[1]: ssh.service: Service hold-off time 
over, scheduling restart.
Mar 01 07:38:25 sempron sshd[437]: error: Bind to port XXXX on 
192.168.1.54 failed: Cannot assign requested address.
Mar 01 07:38:25 sempron sshd[437]: fatal: Cannot bind any address.
Mar 01 07:38:25 sempron systemd[1]: ssh.service: Main process exited, 
code=exited, status=255/n/a
Mar 01 07:38:25 sempron systemd[1]: ssh.service: Unit entered failed state.
Mar 01 07:38:25 sempron systemd[1]: ssh.service: Failed with result 
'exit-code'.
Mar 01 07:38:25 sempron systemd[1]: ssh.service: Service hold-off time 
over, scheduling restart.
Mar 01 07:38:25 sempron sshd[454]: Server listening on 192.168.1.54 port 
XXXX.
Mar 01 07:38:26 sempron sshd[454]: Received SIGHUP; restarting.
Mar 01 07:38:26 sempron sshd[454]: Server listening on 192.168.1.54 port 
XXXX.
Mar 01 07:38:26 sempron sshd[454]: Received SIGHUP; restarting.
Mar 01 07:38:26 sempron sshd[454]: Server listening on 192.168.1.54 port 
XXXX.
Mar 01 07:38:53 sempron systemd[941]: Listening on GnuPG cryptographic 
agent (ssh-agent emulation).

where XXXX is the sshd port number

/etc/systemd/system/sshd.service file used for above log file:

[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
#Restart=on-failure
Restart=always
#RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service

/etc/systemd/system/sshd.service.d/override.conf

[Unit]
After=network.target

[Service]
Restart=always
RestartSec=5
RestartPreventExitStatus=