Bug#873613: systemd gets confused at shutdown time

[Felipe Sateler]

On Fri, Sep 1, 2017 at 7:27 AM, Harald Dunkel <harald.dunkel at aixigo.de> wrote:
> On Wed, 30 Aug 2017 10:17:55 -0300
> Felipe Sateler <fsateler at debian.org> wrote:
>
>> On Wed, Aug 30, 2017 at 6:53 AM, Harald Dunkel <harald.dunkel at aixigo.de>
>> wrote:
>>
>> > On Tue, 29 Aug 2017 11:17:03 -0300
>> > Felipe Sateler <fsateler at debian.org> wrote:
>> > >
>> > > Please attach the full configuration for your mount points.
>> > >
>> >
>> > /proc/mounts is attached (as it is now).
>> >
>>
>> Looks like your nfs mounts are missing the _netdev option. Does the problem
>> persist if you add that option? Is this generated from fstab or mount units?
>>
>
> Its in /etc/fstab. Is the "nfs" or "nfs4" in /proc/mounts somehow
> ambiguous?
>
> Anyhow, the (tiny) problem is that systemd ignored the NFS mount point
> completely. The local mount points were released, even though they
> don't have the _netdev attribute set, either.
>
> Even if systemd would have managed to unmount the NFS mount points,
> the bigger problem is still that systemd stops basic services like
> portmap very early at shutdown time.

AFAICS, the umount is attempted at the correct place, but it fails:

Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target Remote File Systems.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Unmounting /home...
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Unmounting /data...
Aug 29 15:08:40 dpcl082.example.com systemd[1]: home.mount: Mount process exited, code=exited status=16
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Failed unmounting /home.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Unmounted /data.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target Network is Online.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped Network Manager Wait Online.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target Remote File Systems (Pre).
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target NFS client services.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopping RPC security service for NFS client and server...


It appears something is keeping /home still open at the time of the
unmount. Maybe some service has an undeclared dependency on /home?
Finding out who is keeping it open can be done with a unit like the
following:

=== list-open.service
[Unit]
Description=List Open files in /home
RequiresMountsFor=/home
Before=remote-fs.target

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=-/usr/bin/lsof +f -- /home

[Install]
WantedBy=multi-user.target
===


Anyway, I think this general problem is helped by this patch:

https://github.com/systemd/systemd/pull/6588

At the time of systemd-shutdown, because the network is disconnected,
any operations on the nfs mount will fail.

>
>>
>> >
>>  [...]
>> > >
>> > > Looks like systemd shut down your network before it unnmounted remote
>> > > filesystems.
>> > >
>> >
>> > Maybe it should have tried to run "umount -f" or to kill user processes
>> > keeping the mount point busy? Anyway, if you look at the log file you
>> > will notice that portmap was stopped *very* early at shutdown time, even
>> > though /home was still mounted via NFS.
>> >
>> > >
>> > > Could you attach full logs? Attaching the info generated by reportbug
>> > > would be useful too.
>> > >
>> >
>> > journalctl.log is attached (in ASCII). Unfortunately journald stopped
>> > logging. The long delay at shutdown time doesn't show, but I took a
>> > photo.
>> >
>>
>> The full info that reportbug would have generated is important. Please
>> attach it.
>>
>
> Attached.
>
> On Wed, 30 Aug 2017 11:53:38 +0200 I had sent some attachments to this
> bug (the output of journalctl and a screen snapshot). What happened to
> these?
>

I received them, and made a cursory look only. The information from
reportbug is very complete, and avoids a lot of back-and-forth with
questions. For this reason I don't look into much detail without the
full info available. There is a reason we tell reportbug to include
it, and it is that it saves everybodys time.

-- 

Saludos,
Felipe Sateler