Bug#815020: breaks coredump handling for systems without systemd-coredump

[Felipe Sateler]

On 17 February 2016 at 21:03, Wouter Verhelst <wouter at debian.org> wrote:
> On Wed, Feb 17, 2016 at 06:33:59PM -0300, Felipe Sateler wrote:
>> On 17 February 2016 at 13:12, Wouter Verhelst <wouter at debian.org> wrote:
>> > Package: systemd
>> > Version: 229-1
>> >
>> > Hi,
>> >
>> > systemd contains the following line (src/core/main.c):
>> >
>> >                         (void) write_string_file("/proc/sys/kernel/core_pattern", "|/bin/false", 0);
>> >
>> > The intent is that at some later point, systemd-coredump is started,
>> > which then sets it to something more useful.
>>
>> Note that "systemd-coredump is started" means exactly "set it to
>> something more useful" systemd-coredump does not run as a daemon, but
>> rather is invoked by the kernel.
>
> That's not what the NEWS file seems to suggest, but I'll take your word
> for it.

Do you refer to the systemd service bit? The change mentioned is that
previously, the systemd-coredump binary would process the core file
itself while being invoked by the kernel. Now, it will instead trigger
a oneshot service that can be monitored with systemd, and thus can be
subject to resource limits, niceness, etc.

>
>> > However, in Debian, systemd-coredump is in a separate package that
>> > systemd itself does not depend on. As a result, if systemd is used as
>> > PID1, but systemd-coredump is not installed, all core dumps are thrown
>> > into the bit bucket; not just those that are generated during bootup,
>> > but all core dumps after boot, too.
>> >
>> > They're still generated, however, since systemd also sets the default
>> > soft resource limit on core file size ("ulimit -c") to the "unlimited"
>> > value, rather than having the hard limit be unlimited and the soft limit
>> > be set at 0.
>>
>> Apparently, if the core_pattern is a program, then the limit is
>> ignored by linux, and should instead be honored by the called program.
>
> yes, well, so they're generated at any rate, even if the limit is set
> :-)
>
>> > This is not an ideal situation:
>> > - Generating a core dump, especially in case of a program that uses a
>> >   lot of memory at the time where the dump is generated, requires some
>> >   processing. If the core dump is thrown away, this is wasted processing
>> >   time. If the system is loaded (e.g., because something is forking and
>> >   segfaulting a lot), this is problematic behaviour. By setting the soft
>> >   core file size resource limit to 0, core dumps aren't even generated,
>> >   improving performance.
>>
>> How much resources does this actually consume? If the receiving
>> process does nothing, I don't think there would be much effect going
>> on.
>>
>> According to this (anectodic) upstream list post, the cost is negligible
>>
>> http://article.gmane.org/gmane.comp.sysutils.systemd.devel/31244/match=core_pattern
>
> It may be "negligible", it won't be "nothing". If the system is under
> high load, thousands of "negligible" isn't that any longer.

Right, the question is whether this new load makes the system
noticeably worse than just the restart/abort loop.

>
>> > - The kernel.core_pattern sysctl setting is a system-wide setting; it is
>> >   not possible to modify this other than by using CAP_SYS_ADMIN (IIUC)
>> >   permissions. In contrast, the resource limits are per-process
>> >   configuration; any process can change its resource limits at will.
>> >   This allows the system to have a default of "no core dumps", but at
>> >   the same time, allows a developer or a system administrator to
>> >   temporarily enable core dumps for a particular process, without
>> >   requiring administrator capabilities.
>>
>> Because systemd-coredump respects rlimit, if systemd did not change
>> the rlimit then it wouldn't work by default: meaning that
>> systemd-coredump would have to somehow tell systemd (maybe at install
>> time) to change the default rlimit to something saner, but without
>> interfering with any admin changes.
>>
>> Because systemd conf snippets are always read after the main
>> configuration file, it means that systemd-coredump cannot ship a
>> system.conf snippet altering the default rlimit, because it would
>> override admin changes in system.conf.
>
> I'm not sure I understand this bit. Can you clarify?

Systemd-coredump, as of systemd 229, respects the rlimit of the
crashed process. This means that if the limit is not set to a positive
value, no coredumps will be stored.

This means that systemd-coredump should somehow instruct systemd to
set the default core rlimit to a high value when it is installed. This
cannot be done via a system.conf.d snippet, as that would override any
changes made by the admin in /etc/systemd/system.conf. I do not know
of any other way to set a system-wide rlimit other than telling pid1
to do it. I hope this is clearer now.

>
>> > I'm sure systemd-coredump replicates some of the above, but it isn't
>> > always the best choice to have a core dump be piped into a process, and
>> > therefore using the traditional "just dump everything to disk" logic can
>> > still be beneficial.
>>
>> It can still be restored via a sysctl.d snippet, and the rlimit via system.conf.
>>
>> Maybe this should be documented somewhere on upgrades?
>
> Possibly.
>
> An alternative solution would be that systemd-sysv shipped with a unit
> which would set the core_pattern back to default, which could be
> overridden or disabled by a unit shipped by a unit shipped with
> systemd-coredump? That would be less surprising -- I have to say I spent
> a long time tracking down what happened.

I think the bigger problem is setting the default core rlimit, not the
core_pattern. The /bin/false core_pattern is being set because the
default rlimit was raised, otherwise you would get core files
everywhere on your system. In turn, the rlimit was raised because now
systemd-coredump respects the limit (previously it would ignore it and
save larger than allowed dumps).

-- 

Saludos,
Felipe Sateler