Bug#840056: shibboleth-sp2-utils: upgrade attempt of shibboleth-sp2-utils gets hung at restart of shibd service

[S. Banerian]

On 10/09/2016 05:25 PM, Ferenc Wágner wrote:
> "S. Banerian" <banerian at u.washington.edu> writes:
> 
>> On 10/07/2016 02:04 PM, Ferenc Wágner wrote:
>>
>>>> Manually trying to start /usr/sbin/shibd -f -F never produces the
>>>> /run/shibboleth/shibd.sock nor any other files in that dir.
>>>
>>> This is really disturbing.  You don't except the above command to give
>>> back your prompt, right?  
>>
>> correct.
>>
>>> Could you please make sure shibd isn't running
>>> then show me the output of
>>>
>>> # sudo -u _shibd strace shibd -f -F
>>
>> the output is rather long...
>> I shall attach.
> 
> Thanks.  Does it hang there indefinitely, with no further output for
> minutes?  

after some 12 hours of trying to start, failing, it finally started,
created shibd.sock, and under a test, worked.

Its dpkg status is still iF


Does shibd still consume significant CPU according to top?

at this moment, no.
> What are its memory figures?

nothing big.

> Isn't your computer swapping heavily?

no

> Can
> you provide a full GDB backtrace (after installing
> shibboleth-sp2-utils-dbgsym; please yell if you need precise
> instructions).

does not appear to be in stretch. so i need the instructions.


>> Note: prior to the upgrade, shibboleth was working.
> 
> Which version of shibboleth was working for you?

the version just prior to this one 2.6.0+dfsg1-3+b1 on stretch.

> Can you share your shibboleth2.xml?

I'm a bit reluctant to provide some of the information in the
RequestMapper sections.



>> We did note that sometimes shibd was more well-behaved when starting
>> from shell than via systemctl.
> 
> Could you please make this more precise?  How did it misbehave when
> started by systemctl?

this is where it hangs for minutes, says it might start, may or may not
create /run/shibboleth/shibd.sock  and the apache application only
returns with shibboleth errors.

Now, however, as it has sat over the weekend, and finally started, dpkg
has let the upgrade complete, with no errors.

When I force a restart, systemctl restart shibd.service I get the issue
as before, where

\_ /bin/systemd-tty-ask-password-agent --watch

stays there for a looong time, and is not returning, systemctl says it
is started, but journalctl -xe gives:

Oct 10 14:00:35 epics systemd[1]: shibd.service: Killing process 30980
(shibd) with signal SIGKILL.
Oct 10 14:00:35 epics systemd[1]: shibd.service: Main process exited,
code=killed, status=9/KILL
Oct 10 14:00:35 epics systemd[1]: Failed to start Shibboleth Service
Provider Daemon.
-- Subject: Unit shibd.service has failed

there is a shibd -f -F process running, but no shibd.sock file

I'm not convinced that systemd is behaving well.


-- 
Stefani Banerian
UW Clinical Cyclotron www.uwmcf.org
UW School of Medicine
UW Box 356043
206-598-0302
gpg key 6642E7EE
fingerprint = BD13 875D 2D03 5E1D 1E3B  8BF7 F4B8 63AD 6642 E7EE