<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<pre>Sorry for the late reply here. Got caught up in other stuff. I did
some digging around. Set the loglevel with systemd-analyze to debug
and also added the debug flag to pam_systemd.so.
What I noticed is that on boot, when 5000+ machines try to
authenticate at once, pam_systemd seems to fall on its ass and fail
due to resource exhaustion. An excerpt from journalctl is attached
below.
I thought this might've been the reason behind sshd processes not
being assigned to the correct slice, but the processes for which
these log entries are generated are not available on the system
anymore, which I take as meaning that the sshd process exited
because it couldn't open a session.
I tried to go about it the other way around and search for logs
generated by any sshd process which is under the system.slice. I
used this oneliner to do so:
for i in $(systemctl status ssh | grep 'sshd: <user>' | sed -E 's/[^0-9]*([0-9]*)[^0-9]*/\1/'); do echo "==> logs for process $i"; journalctl | grep '\[$i\]'; done | less
The search came up empty. For none of the currently 7000+ sshd
processes which aren't in the correct user slice, there are any logs
in journalctl. I verified and all logs since boot time are currently
still in the journal, it hasn't rotated yet.
Jun 04 09:50:17 <hostname> sshd[8415]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8691]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8717]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28538]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8652]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8721]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8716]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28121]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[28551]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8682]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8396]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28524]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8719]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28546]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8690]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8697]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[7350]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8522]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8406]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28636]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[28620]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[28669]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8693]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8723]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28633]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8401]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8692]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28670]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8543]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8234]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8654]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28545]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8791]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8840]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8815]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28710]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8541]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28705]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[28269]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
Jun 04 09:50:17 <hostname> sshd[8725]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8786]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8849]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8562]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8572]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8816]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8870]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[8560]: pam_systemd(sshd:session): Failed to create session: Connection timed out
Jun 04 09:50:17 <hostname> sshd[28759]: pam_systemd(sshd:session): Failed to connect to system bus: Resource temporarily unavailable
</pre>
</body>
</html>