[Pkg-sysvinit-devel] Bug#777113: invoke-rc.d, service under systemd: Avoid job dependency loops during boot/shutdown
Martin Pitt
mpitt at debian.org
Thu Feb 5 08:57:50 UTC 2015
Package: sysvinit-utils
Version: 2.88dsf-58
Severity: important
Tags: patch
Hello all,
Due to the way systemd job transactions work, we had and still have
several occasions where DHCP/ifupdown and similar hooks try to reload
or restart a service and cause a deadlock. That's because especially
during boot and shutdown there is a big transaction of services and
their dependencies which want to start or stop, and if within that
transaction you synchronously request another one you get a circular
dependency loop which causes such a deadlock. Examples are
https://bugs.debian.org/652942 (still open and reproducible), and
https://bugs.debian.org/635777 and https://bugs.debian.org/624599. For
the latter two the systemd package currently has workarounds, but they
don't always apply.
This is an emergent problem as we try to combine two completely
different paradigms:
* Under SysV init, commands like "service foo restart" are always
synchronous, i. e. they wait for the operation to finish. But they
don't take any dependencies into account, i. e. restarting "foo"
won't ensure that foo's dependencies are already started.
* Under systemd, starting a unit always ensures that its dependencies
(and ordering constraints) are fulfilled. But in that world, things
like DHCP hooks should reload services asynchronously. (E. g. with
systemctl --no-block reload foo)
Both are valid, the trouble comes with trying to combine them. Our
"service" and "invoke-rc.d" hooks work under all init systems and
don't offer async operations. This was discussed several times on the
systemd ML, like on
http://lists.freedesktop.org/archives/systemd-devel/2014-July/021457.html
(continuation: http://lists.freedesktop.org/archives/systemd-devel/2014-August/022048.html)
http://lists.freedesktop.org/archives/systemd-devel/2015-February/027966.html
But in the end all of those are just hacks which break systemd's
semantics. There is fundamentally no good way how we can fulfill both
the SysV and the systemd semantics, so we need some
heuristics/workarounds.
One such thing would be that service/invoke-rc.d always call systemctl
with --job-mode=ignore-dependencies which would make it do what SysV
does: only operate on the given job but not its dependencies. But
often this behaviour is actually handy, and the lockups realistically
only occur on boot and shutdown, so we can restrict it to these times.
That's what the attached patch does. After long discussions with
both systemd upstream as well as Michael Biebl I am now convinced that
this is the most sensible approach, doesn't break either SysV or
systemctl semantics, and is a good enough heuristics to avoid
deadlocks. Such deadlocks in principle happen during runtime as well,
but there's no evidence of that so far; if it does happen, we need to
adjust the scope of the "ignore-dependencies", but I don't think it
will be necessary.
With this I verified that all of the bugs above (plus
https://launchpad.net/bugs/1417010, which is the Ubuntu equivalent of
https://bugs.debian.org/652942) are fixed with a pristine upstream
systemd, i. e. we can drop our insufficient patches.
Thanks for considering,
Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
-------------- next part --------------
diff -Nru sysvinit-2.88dsf/debian/changelog sysvinit-2.88dsf/debian/changelog
--- sysvinit-2.88dsf/debian/changelog 2014-11-11 20:34:28.000000000 +0100
+++ sysvinit-2.88dsf/debian/changelog 2015-02-05 09:51:50.000000000 +0100
@@ -1,3 +1,16 @@
+sysvinit (2.88dsf-59) UNRELEASED; urgency=medium
+
+ * service, invoke-rc.d: Avoid deadlocks during bootup and shutdown from
+ units/hooks which call "invoke-rc.d service reload" and similar, since the
+ synchronous wait plus systemd's normal behaviour of transactionally
+ processing all dependencies first easily causes dependency loops. Thus
+ during boot/shutdown operate only on the unit and not on its dependencies,
+ just like SysV behaves.
+ * Make sysvinit-utils and sysv-rc break systemd << 215 to ensure we have the
+ "systemctl is-system-running" command.
+
+ -- Martin Pitt <mpitt at debian.org> Thu, 05 Feb 2015 09:48:40 +0100
+
sysvinit (2.88dsf-58) unstable; urgency=low
* Fix typo in invoke-rc.d breaking upstart installations (Closes:
diff -Nru sysvinit-2.88dsf/debian/control sysvinit-2.88dsf/debian/control
--- sysvinit-2.88dsf/debian/control 2014-10-25 23:24:19.000000000 +0200
+++ sysvinit-2.88dsf/debian/control 2015-02-05 09:51:15.000000000 +0100
@@ -65,7 +65,7 @@
Replaces: last, sysvinit (<= 2.86.ds1-65)
Depends: ${shlibs:Depends}, ${misc:Depends}
, startpar
-Breaks: upstart (<< 1.5-0ubuntu5)
+Breaks: upstart (<< 1.5-0ubuntu5), systemd (<< 215)
Suggests: bootlogd, sash
Description: System-V-like utilities
This package contains the important System-V-like utilities.
@@ -85,7 +85,7 @@
sysvinit-utils (>= 2.86.ds1-62),
insserv (>> 1.12.0-10)
, startpar
-Breaks: initscripts (<< 2.86.ds1-63)
+Breaks: initscripts (<< 2.86.ds1-63), systemd (<< 215)
Description: System-V-like runlevel change mechanism
This package provides support for the System-V like system
for booting, changing runlevels, and shutting down,
diff -Nru sysvinit-2.88dsf/debian/service/service sysvinit-2.88dsf/debian/service/service
--- sysvinit-2.88dsf/debian/service/service 2014-10-25 23:20:33.000000000 +0200
+++ sysvinit-2.88dsf/debian/service/service 2015-02-05 09:31:14.000000000 +0100
@@ -184,6 +184,15 @@
if [ -n "$is_systemd" ]
then
UNIT="${SERVICE%.sh}.service"
+ # avoid deadlocks during bootup and shutdown from units/hooks
+ # which call "invoke-rc.d service reload" and similar, since
+ # the synchronous wait plus systemd's normal behaviour of
+ # transactionally processing all dependencies first easily
+ # causes dependency loops
+ if ! systemctl is-system-running; then
+ sctl_args="--job-mode=ignore-dependencies"
+ fi
+
case "${ACTION}" in
restart|status)
exec systemctl ${ACTION} ${UNIT}
@@ -195,10 +204,10 @@
# Users who need more control will use systemctl directly.
for unit in $(systemctl list-unit-files --full --type=socket 2>/dev/null | sed -ne 's/\.socket\s*[a-z]*\s*$/.socket/p'); do
if [ "$(systemctl -p Triggers show $unit)" = "Triggers=${UNIT}" ]; then
- systemctl ${ACTION} $unit
+ systemctl $sctl_args ${ACTION} $unit
fi
done
- exec systemctl ${ACTION} ${UNIT}
+ exec systemctl $sctl_args ${ACTION} ${UNIT}
;;
reload)
_canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)"
@@ -208,7 +217,7 @@
# specific service.
run_via_sysvinit
else
- exec systemctl reload "${UNIT}"
+ exec systemctl $sctl_args reload "${UNIT}"
fi
;;
force-stop)
@@ -217,9 +226,9 @@
force-reload)
_canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)"
if [ "$_canreload" = "CanReload=no" ]; then
- exec systemctl restart "${UNIT}"
+ exec systemctl $sctl_args restart "${UNIT}"
else
- exec systemctl reload "${UNIT}"
+ exec systemctl $sctl_args reload "${UNIT}"
fi
;;
*)
diff -Nru sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d
--- sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d 2014-11-08 21:38:18.000000000 +0100
+++ sysvinit-2.88dsf/debian/src/sysv-rc/sbin/invoke-rc.d 2015-02-05 09:32:35.000000000 +0100
@@ -519,16 +519,24 @@
# pick up any changes.
systemctl daemon-reload
fi
+ # avoid deadlocks during bootup and shutdown from units/hooks
+ # which call "invoke-rc.d service reload" and similar, since
+ # the synchronous wait plus systemd's normal behaviour of
+ # transactionally processing all dependencies first easily
+ # causes dependency loops
+ if ! systemctl is-system-running; then
+ sctl_args="--job-mode=ignore-dependencies"
+ fi
case $saction in
start|stop|restart|status)
- systemctl "${saction}" "${UNIT}" && exit 0
+ systemctl $sctl_args "${saction}" "${UNIT}" && exit 0
;;
reload)
_canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)"
if [ "$_canreload" = "CanReload=no" ]; then
"${INITDPREFIX}${INITSCRIPTID}" "${saction}" "$@" && exit 0
else
- systemctl reload "${UNIT}" && exit 0
+ systemctl $sctl_args reload "${UNIT}" && exit 0
fi
;;
force-stop)
@@ -537,9 +545,9 @@
force-reload)
_canreload="$(systemctl -p CanReload show ${UNIT} 2>/dev/null)"
if [ "$_canreload" = "CanReload=no" ]; then
- systemctl restart "${UNIT}" && exit 0
+ systemctl $sctl_args restart "${UNIT}" && exit 0
else
- systemctl reload "${UNIT}" && exit 0
+ systemctl $sctl_args reload "${UNIT}" && exit 0
fi
;;
*)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-sysvinit-devel/attachments/20150205/3e62a0cf/attachment.sig>
More information about the Pkg-sysvinit-devel
mailing list