[Pkg-freeipa-devel] Bug#970880: Bug#970880: Bug#970880: freeipa-server: FreeIPA server installation fails with Certificate issuance failed (CA_REJECTED)

[Spencer Olson]

Since it doesn't look like any progress has been made on this, I've
started to work through some debugging.

Right now, it looks like the problem is probably actually due to a
change in libnss3.  In fact, the problem appears to be specifically in
libssl3.so from the libnss3 package.

The problem:
  * certmonger has a hard time finishing the certificate requests
because it can't seem to authenticate to the dogtag PKI server.

Observations:
 * When certmonger attempts to request a signed certificate for the
renewal agent, it temporarily explicitly uses the ipa-ca-agent
certificate which has been temporarily extracted from the
/root/ca-agent.p12 storage.
 * dogtag-submit attempts to use the CURL library to submit the
request, subsequently approve the request, and then poll for its
finish.
 * The initial request does not use/require an encrypted channel, but
the approval and subsequent queries do.
 * These attempts to authenticate over this encrypted channel using
the client certificate are rejected.

Hacks & tests:
 * By creating a very small c-program that does the same CURL commands
as dogtag-submit from the certmonger package, this same authorization
denied can be seen.
 * By simply replacing the libssl3.so library, using either LD_PRELOAD
or LD_LIBRARY_PATH, from a prior version, the requests succeed.  As of
now, I've tried only one other version of libssl3.so (libnss3 3.35
from ubuntu 18.04).
 * Also, instead of linking against libcurl-nss and manualy replacing
the libssl3.so library, success can be found by linking to
libcurl-gnutls or libcurl-openssl

I suspect that a compile option in libnss3 has to be changed in order
for this to work again.

Still todo:
 * I haven't fully discovered which part/option from libnss3 might have changed.
 * I haven't yet successfully had libnss3 emit much
debugging--probably have to recompile with DEBUG=1.