Bug#785445: systemd: with 2 RTC devices -- need way to chose which one sets system clock on boot

[Andreas Henriksson]

Hello Rick Thomas,

Thanks for filing #785445. I think this bug is in util-linux but
it's not clear to me exactly how we should handle it. We should
probably consider reassigning it to util-linux though...
(Also: reassign 785419 util-linux + forcemerge 785445 785419)

More information inline.

On Sat, May 16, 2015 at 04:54:25AM -0700, Rick Thomas wrote:
> Package: systemd
> Version: 215-17
> Severity: normal
> 
> My cubox-i4pro armhf device has two real-time-clocks.  One, snvs, is not battery backed,
> hence is not useful for setting the system clock on boot after a power failure.
> The other, pcf8523, does have battery backup.
> 
> Unfortunately, the snvs is recognzed first, so it become /dev/rtc0 (the pcf8523 becomes /dev/rtc1)
> and the kernel at boot time uses /dev/rtc0 by default to set the system time from.
> 
> There does not seem to be any way to over-ride this. 

Not sure, but if I read rtc_device_register(...) in kernel correctly
you should be able to order your rtcs in the device-tree.

http://sources.debian.net/src/linux/4.8.5-1/drivers/rtc/class.c/#L163

I however don't think that playing with the rtc order is the correct
solution. (We live in a hotplug world and even if you probably don't
physically hotplug RTCs that often, apparently these days drivers
are being built as modules and loading of the driver might happen
when we didn't expect it.)

> There's code in /etc/default/hwclock that would do part of the work in
> a sysvinit setup, but it seems to be ignored under systemd.
> 
> Presumably, there is systemd magic that could do the same thing as was
> available under sysvinit.  Is there anybody out there with enough
> systemd fu to tell me how to do that?

I don't think there's much difference between sysvinit and systemd
other than the hwclock init script being run at bootup under
sysvinit while it's masked under systemd.
(Disabling it also under sysvinit would probably be a good idea as
the rationale for it being masked under systemd isn't in any way
systemd-specific.)

> 
> Other ways to attack this problem may involve something with udev
> rules, but I'm not savvy enough to figure that out for myself, either.

The udev-rules is AFAIK the only way to attack this problem under
systemd.

See:
https://anonscm.debian.org/cgit/collab-maint/pkg-util-linux.git/commit?id=7c0dbd842ba49c55ac899ca9c9455013bf0edabe

> 
> In googling about, I stumbled across a proposed patch that would add a kernel command line
> parameter, "hctosys=rtc<#>' that would over-ride the default, but it seems not
> to have ever been implemented ( http://lkml.iu.edu/hypermail/linux/kernel/1407.0/03989.html )
[...]


On Sat, May 16, 2015 at 01:19:16PM -0700, Rick Thomas wrote:
> 
> On May 16, 2015, at 12:58 PM, Rick Thomas <rbthomas at pobox.com> wrote:
> 
> > 
> > On May 16, 2015, at 6:02 AM, Ben Hutchings <ben at decadent.org.uk> wrote:
> > 
> >> This is not implemented directly by the init system.  util-linux
> >> installs the script/lib/udev/hwclock-set and a udev rule that runs it
> >> for each RTC device.  However, the hwclock-set script does nothing if
> >> systemd is running.
> > 
> > curiouser and curiouser…
> > 
> > Looking at the code in /lib/udev/hwclock-set, I can’t see that it would would ever do anything useful (except by chance) in the case like mine where there are two rtc devices, only one of which should actually be used to set system time at boot.
> > 
> > In particular, it goes to some effort to source /etc/default/rcS and
> > /etc/default/hwclock, but it pays no attention to the HCTOSYS_DEVICE
> > parameter.

I think it's unfortunate that /etc/default/rcS is duplicating the
HCTOSYS_DEVICE setting which is also coming from the kernel
configuration. The kernel also does some sanity checks no the rtc
before giving it the "hctosys" attribute.

(Ignoring settings under /etc/default is unfortunate and makes it harder
than needed for the user to figure out what's going on.... On the
other hand I think the fault lies on whoever didn't think carefully
enough about this before introducing it to /etc/default.)

> > It appears to set the system time from each RTC device in turn as it discovers them.  So system time ends up set by the last RTC to be discovered.  If the right one happens to be last, that’s good.  But that’s not guaranteed.
> 
> Looking further, I find that what I said is not quite true.
> hwclock-set only gets called for /dev/rtc0, i.e. the *first* one to be
> discovered.  This happens in /lib/udev/rules.d/85-hwclock.rules.
> There is provision in /lib/udev/rules.d/50-udev-default.rules to swing
> the /dev/rtc symlink to the device that has ATTR{hctosys}==“1”, but
> that doesn’t fix the problem at hand, because the symlink is not used
> anywhere in hwclock-set.
> 
> Fascinating!

I think the main issue is that there are two different udev rules:

The one shipped by udev itself in
/lib/udev/rules.d/50-udev-default.rules contains:

# select "system RTC" or just use the first one
SUBSYSTEM=="rtc", ATTR{hctosys}=="1", SYMLINK+="rtc"
SUBSYSTEM=="rtc", KERNEL=="rtc0", SYMLINK+="rtc", OPTIONS+="link_priority=-100"


And the one shipped by util-linux at /lib/udev/rules.d/85-hwclock.rules
which contains:

# Set the System Time from the Hardware Clock and set the kernel's
# timezone
# value to the local timezone when the kernel clock module is loaded.

KERNEL=="rtc0", RUN+="/lib/udev/hwclock-set $root/$name"


The hwclock rule should probably be more similar to the udev default
rule. ie. try to use the "hctosys" attribute.

At the same time I see a problem with the udev rule which I'm not sure
how to handle. Consider the second rule, which is supposed to handle
the case where you configured your homebrew kernel with
CONFIG_RTC_HCTOSYS{,_DEVICE} unset.
The problem I see it that the rule will also match in the case when
kernel skips setting the attribute because there's an error with the
RTC.

Consider the case when you have CONFIG_RTC_HCTOSYS_DEVICE set to rtc1
and that one is marked as having an error (personal experience is
for example because the rtc has not been initialized with a time).
The kernel will then skip setting the "hctosys" attribute and the
udev default rule will point the /dev/rtc symlink to /dev/rtc0 instead
of your desired /dev/rtc1. That might in turn cause other unknown
consequenses. I don't know of a way to handle this in udev/userspace and
simply blame the kernel for making (non-availability of) hctosys
attribute mean multiple things.

Also, for arm (and others) it would probably be useful to have
a way to specify which device to use as "hctosys device" via
devicetree if it isn't already possible. Rather than building
into the kernel binary, as the same kernel probably runs on
many different systems with different rtc setups.
(I think this would be better than passing it on the kernel command
line, but maybe there are still cases where the kernel command line
patch you mentioned before is still useful.)

No matter if the kernel is perfect or not, using udev default rules
naive approach also in hwclock.rules is probably an improvement over
the current situation (and hopefully good enough in general).

Would be great if you where willing to work on this and send me
a tested patch for the hwclock.rules. ;)
(If you could get Aurelien in the loop and have him review your patch
as well that would be awesome.)

HTH.

Regards,
Andreas Henriksson