[Pkg-zfsonlinux-devel] Bug#900656: Bug#900656: Make ZFS upgrades in backports more reliable

[Craig Sanders]

tl;dr version: DON'T UPGRADE YOUR KERNEL until you know for sure that the
ZFS packages are compatible with the kernel you're about to install.


On Sat, Jun 02, 2018 at 10:38:09PM +0200, Michael Wyraz wrote:
> I do run zfs-dkms on several servers and on my desktop (currently on
> stretch-backports). I am also running the stretch-backport kernel because
> I need some recent kernel features. Unfortunately, zfs-dkms builds broke
> several times in the past. The reason was always that the backport kernel
> was upgraded but this upgrade was not synchronized with a zfs-dkms upgrade.

Yes, this happens. On almost every new kernel release.  It will continue to
happen until Oracle chooses to re-license ZFS so that it is compatible with
both the GPL (for Linux) **AND** the BSD & CDDL licenses (for Open ZFS) and
can be merged into the mainline kernel.

Oracle has shown no interest in doing this and probably never will, so don't
hold your breath waiting for it.

And even if it does happen, the Open ZFS that ZFS on Linux is based on would
also have to re-license and merge with Oracle's ZFS, and that would likely
take a year or three.

> My big wish is that zfs-dkms upgrades on backports and kernel upgrades on
> backports could be coordinated in a way that prevents that a kernel upgrade
> breaks the zfs file system.

This will likely never happen.  The kernel package maintainers have their own
release schedule and will almost certainly not delay or alter it for the sake
of an optional contrib package with license-compatiblity issues that is of
interest to only a tiny minority of debian users.

> Otherwise it's not possible to use this combination in a production
> environment (or any other kind of environment that is important to work).

It's easily possible.  In fact, it's easier now than it has ever been,
certainly since I started running zfsonlinux on debian boxes in 2010 or so.

The important thing to do is to use apt-mark to hold all ZoL packages **AND**
linux-image-* and linux-header-* packages.

You should not be allowing apt-get dist-upgrade to automatically upgrade a
kernel on production servers anyway.  Kernel upgrades, and spl-dkms & zfs-dkms
upgrades should be done on production severs only a) when they're compatible,
b) when they're known to work reliably together, and c) after you've tested
the upgrade procedure on an unimportant machine or VM.

This is true for any package on a production server, and especially true for
kernel and related packages which could cause the server to fail to reboot
correctly, and for packages which implement the server's primary purpose.

Debian provides the tools to make a sysadmin's job easier to do and easier to
repeat reliably on many machines. It does not and can not automagically know
what to do in every possible sitution or combination of incompatible packages.
That's your job.

anyway, for example:

    apt-mark hold linux-image-amd64 linux-headers-amd64

    apt-mark hold spl spl-dkms zfs-dkms zfsutils-linux libnvpair1linux \
                  libuutil1linux libzfs2linux libzfslinux-dev libzpool2linux


Then apt-get install the same packages when you KNOW it's safe to upgrade
them, and apt-mark hold them again **IMMEDIATELY** after the upgrade.  I've
been doing this without problems for years, since before the zfs-dkms etc
packages were officially in debian.

Put the upgrade and hold commands in a script.  NOTE: the script should
upgrade the zfs packages **before** the kernel packages.  For example:

    #!/bin/bash

    declare -a ZFS KERNEL

    ZFS=(spl spl-dkms zfs-dkms zfsutils-linux libnvpair1linux
         libuutil1linux libzfs2linux libzpool2linux)

    KERNEL=(linux-image-amd64 linux-headers-amd64)

    apt-get update

    apt-get install "${ZFS[@]}"
    apt-get install "${KERNEL[@]}"

    apt-mark hold "${ZFS[@]}" "${KERNEL[@]}"



ps: Theoretically, it's possible that only the kernel packages need to be
held. In practice, I wouldn't want to be running a server (or any machine)
where the running kernel has had its dkms modules upgraded without rebooting,
or where zfsutils-linux and the zfs libs have been upgraded past what's in the
current running kernel.  So I co-ordinate the upgrades to happen at the same
time, when I can afford some downtime to reboot, and after I've tested the
upgrade on another machine.

craig