Bug#923508: systemd not restarting sshd on Debian 9 Stretch i386 due to RestartPreventExitStatus=255 in sshd.service file

[Michael Biebl]

Am 01.03.19 um 08:59 schrieb Andrew Roberts:
> Package: systemd
> Version: 232-25+deb9u
> 
> On Debian 9 stretch with the latest updates systemd no longer restarts
> sshd due to changes to the sshd.service unit file.
> 
> The line:
> 
> RestartPreventExitStatus=255 has been added to the master sshd unit file
> in recent systemd/sshd updates.


ssh.service is shipped by openssh-server, not by the systemd package.

RestartPreventExitStatus=255 was added in 2015 (in openssh 1:7.1p1-5).
So this is not a recent change

http://bugs.debian.org/778913 contains more information why it
was added to ssh.service. Short version: sshd will return with an exit
code of 255 when your config file contains errors. Repeatedly restarting
sshd in that case is not a sensible thing to do as you will hit a
restart limit and ssh.service will be marked as failed.

In any case, you need to discuss changes to ssh.service with the
openssh-server maintainer.

> This means if the network interface is not up by the time sshd tries to
> start, sshd will return a result code of 255, and systemd will NOT try
> to restart it.

See above why the exit code 255 is ignored.
If sshd also returns 255 when it fails to bind, then this is unfortunate.
I'm not sure if continuously restarting in that case is helpful though
(and hoping you don't hit the restart limit until network is up)


> This is obviously a HUGE problem for anybody using sshd as their
> exclusive means of accessing a system. On many systems the network can
> take a considerable amount of time to start.


From the error message you posted, it seems you configured sshd to bind
to a specific IP address (probably via ListenAddress in sshd_config).
This is not the default sshd config.

If you bind to a specific address, you need to make sure, that your
network is properly configured before sshd starts.
Assuming you use ifupdown, the simplest way to achieve that is to change
allow-hotplug to auto.
interfaces marked as allow-hotplug are brought up by ifupdown
asynchronously.

It seems to me, you use the restart mechanism of systemd to work around
this broken network configuration.


> When you finally do get physical access to a system, and try to override
> the default unit file by adding:
> 
> RestartPreventExitStatus=
> 
> to the end of /etc/systemd/system/sshd.service.d/override.conf

> and running systemctl daemon-reload and rebooting
> 
> You find that the default of 255 is NOT overridden. I already had:
> 
> [Service]
> 
> Restart=always
> 
> RestartSec=5
> 
> in the override.conf file. What part of ALWAYS does systemd not understand!
> 
> I was forced to comment out the RestartPreventExitStatus=255 line in the
> /etc/systemd/system/sshd.service file, which is NOT a solution.


If you look closely, you'll see that /etc/systemd/system/sshd.service is
a symlink to /lib/systemd/system/ssh.service.
So if you edit /etc/systemd/system/sshd.service directly you actually
edit the file in /lib and any changes you make will be overwritten on
packages update. So don't do that.
Instead use the canonical file name, i.e. ssh.service.

If you want to override ssh.service in full, copy
/lib/systemd/system/ssh.service to /etc/systemd/system/ssh.service and
edit that file (or use systemctl edit --full ssh.service)
This way changes are preserved.

If you want to override only certain aspects of ssh.service, create a an
override file for ssh.service (e.g. via systemctl edit ssh.service).
Keep in mind, that you need to clear certain variables, before you can
set them anew.

Say you want to override Restart= via a drop-in config snippet, you need
to use something like
/etc/systemd/system/ssh.service.d/override.conf
[Service]
Restart=
Restart=always

> Also the version of systemd currently being used has bugs which when you
> manually restart a unit using systemctl restart sshd.service it warns
> about the configuration needing reloading using systemctl daemon-reload,
> even though you have just done that. This bug appears to have been fixed
> once in systemd in 2016, so either an old version is being used, or it
> has crept back in.
> 
> systemctl daemon-reload
> 
> systemctl restart sshd.service
> 
> Warning: sshd.service changed on disk. Run 'systemctl daemon-reload' to
> reload units.

Can't reproduce this.

> Of course if systemd didn't start networking dependent units until the
> network was actually up then none of this would be an issue... Of course
> then you get into arguments about which network interface...
> 
> Clearly much more testing of systemd needs to be done before pushing
> updates of it. But the bottom line is NEVER break sshd.

From what I can see or reproduce, there is nothing that can be addressed
on the systemd side but rather looks like local (mis)configuration issues.

I'll leave it up to Colin (our ssh maintainer), to decide what to do
about RestartPreventExitStatus=255 in ssh.service

Colin, do you want this bug report to be reassigned?



> Systemd Version:
> 
> ||/ Name           Version      Architecture Description
> +++-==============-============-============-=================================
> 
> ii  systemd        232-25+deb9u i386         system and service manager
> 
> Openssh Server version:
> 
> ||/ Name           Version      Architecture Description
> +++-==============-============-============-=================================
> 
> ii  openssh-server 1:7.4p1-10+d i386         secure shell (SSH) server,
> for se
> 
> System Version:
> 
> uname -a
> Linux sempron 4.9.0-8-686-pae #1 SMP Debian 4.9.144-3.1 (2019-02-19)
> i686 GNU/Linux
> 
> cat /etc/os-release
> PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
> 
> apt show libc6 | grep ^Version
> 
> Version: 2.24-11+deb9u4
> 
> Error messages from sshd when booting using default config
> (RestartPreventExitStatus=255 in /etc/systemd/system/sshd.service file)
> 
> journalctl -b0 | grep ssh
> 
> Mar 01 07:32:02 sempron sshd[432]: error: Bind to port XXXX on
> 192.168.1.54 failed: Cannot assign requested address.
> Mar 01 07:32:02 sempron sshd[432]: fatal: Cannot bind any address.
> Mar 01 07:32:02 sempron systemd[1]: ssh.service: Main process exited,
> code=exited, status=255/n/a
> Mar 01 07:32:02 sempron systemd[1]: ssh.service: Unit entered failed state.
> Mar 01 07:32:02 sempron systemd[1]: ssh.service: Failed with result
> 'exit-code'.
> Mar 01 07:32:17 sempron systemd[700]: Listening on GnuPG cryptographic
> agent (ssh-agent emulation).
> 
> where XXXX is the sshd port number
> 
> /etc/systemd/system/sshd.service file used for above log file:
> 
> [Unit]
> Description=OpenBSD Secure Shell server
> After=network.target auditd.service
> ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
> 
> [Service]
> EnvironmentFile=-/etc/default/ssh
> ExecStartPre=/usr/sbin/sshd -t
> ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
> ExecReload=/usr/sbin/sshd -t
> ExecReload=/bin/kill -HUP $MAINPID
> KillMode=process
> #Restart=on-failure
> Restart=always
> RestartPreventExitStatus=255
> Type=notify
> 
> [Install]
> WantedBy=multi-user.target
> Alias=sshd.service
> 
> /etc/systemd/system/sshd.service.d/override.conf
> 
> [Unit]
> After=network.target
> 
> [Service]
> Restart=always
> RestartSec=5
> RestartPreventExitStatus=
> 
> Commenting out RestartPreventExitStatus=255 in sshd.service, doing
> systemctl daemon-reload, and rebooting gives:
> 
> journalctl -b0 | grep ssh
> 
> Mar 01 07:38:24 sempron sshd[426]: error: Bind to port XXXX on
> 192.168.1.54 failed: Cannot assign requested address.
> Mar 01 07:38:24 sempron sshd[426]: fatal: Cannot bind any address.
> Mar 01 07:38:24 sempron systemd[1]: ssh.service: Main process exited,
> code=exited, status=255/n/a
> Mar 01 07:38:24 sempron systemd[1]: ssh.service: Unit entered failed state.
> Mar 01 07:38:24 sempron systemd[1]: ssh.service: Failed with result
> 'exit-code'.
> Mar 01 07:38:24 sempron systemd[1]: ssh.service: Service hold-off time
> over, scheduling restart.
> Mar 01 07:38:25 sempron sshd[437]: error: Bind to port XXXX on
> 192.168.1.54 failed: Cannot assign requested address.
> Mar 01 07:38:25 sempron sshd[437]: fatal: Cannot bind any address.
> Mar 01 07:38:25 sempron systemd[1]: ssh.service: Main process exited,
> code=exited, status=255/n/a
> Mar 01 07:38:25 sempron systemd[1]: ssh.service: Unit entered failed state.
> Mar 01 07:38:25 sempron systemd[1]: ssh.service: Failed with result
> 'exit-code'.
> Mar 01 07:38:25 sempron systemd[1]: ssh.service: Service hold-off time
> over, scheduling restart.
> Mar 01 07:38:25 sempron sshd[454]: Server listening on 192.168.1.54 port
> XXXX.
> Mar 01 07:38:26 sempron sshd[454]: Received SIGHUP; restarting.
> Mar 01 07:38:26 sempron sshd[454]: Server listening on 192.168.1.54 port
> XXXX.
> Mar 01 07:38:26 sempron sshd[454]: Received SIGHUP; restarting.
> Mar 01 07:38:26 sempron sshd[454]: Server listening on 192.168.1.54 port
> XXXX.
> Mar 01 07:38:53 sempron systemd[941]: Listening on GnuPG cryptographic
> agent (ssh-agent emulation).
> 
> where XXXX is the sshd port number
> 
> /etc/systemd/system/sshd.service file used for above log file:
> 
> [Unit]
> Description=OpenBSD Secure Shell server
> After=network.target auditd.service
> ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
> 
> [Service]
> EnvironmentFile=-/etc/default/ssh
> ExecStartPre=/usr/sbin/sshd -t
> ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
> ExecReload=/usr/sbin/sshd -t
> ExecReload=/bin/kill -HUP $MAINPID
> KillMode=process
> #Restart=on-failure
> Restart=always
> #RestartPreventExitStatus=255
> Type=notify
> 
> [Install]
> WantedBy=multi-user.target
> Alias=sshd.service
> 
> /etc/systemd/system/sshd.service.d/override.conf
> 
> [Unit]
> After=network.target
> 
> [Service]
> Restart=always
> RestartSec=5
> RestartPreventExitStatus=
> 
> _______________________________________________
> Pkg-systemd-maintainers mailing list
> Pkg-systemd-maintainers at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-systemd-maintainers


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://alioth-lists.debian.net/pipermail/pkg-systemd-maintainers/attachments/20190302/6f0cc91c/attachment.sig>