[pkg-gnupg-maint] Bug#790665: Bug#790665: gpg2 fails to import gpg2 created keyring in a gpg1 created keyring

[David Kalnischkies]

On Mon, Jul 06, 2015 at 03:19:04PM -0400, Daniel Kahn Gillmor wrote:
> On Sun 2015-07-05 11:10:59 -0400, David Kalnischkies wrote:
> 
> > I don't know where you get the "stdin" from. What I am trying here is
> > merging a keybox into a simple keyring in a single command, no funky
> > business with stdin.
> 
> heh.  i consider "messing with another program's internal data storage"
> to be the "funky business", while reading standardized file formats from
> a file on stdin to be "standard unix practices". :)
> 
> Let's see if we can get both habits to work :)
> 
> > As basically every documentation I came across says "gpg --import
> > [filename]" I kinda expect that to work with whatever file gpg ends up
> > producing.
> 
> gpg will produce files to stdout (and with --output, if you prefer) that
> will work fine for stdin (and as a named argument to --import, if you
> prefer).  Its internal key storage, however, is not guaranteed to be
> either of those formats.

That is an acceptable expectation, I would just be a lot happier if that
would be documentated somewhere. Maybe I am the only idiot, but at least
for this idiot it is surprising that "[filename]" isn't including
a keybox, but expecting a very specific file format which just happened
to be the same for (simple) keyring files and hence were accepted as
well instead of just a (maybe temporarily saved) stream. That it is
binary doesn't help me in making it more discoverable (nor is it very
"standard unix practice" like for that matter to have a binary stream,
text is very much preferred… </nitpick>).


> > "OpenPGP packets" on the other hand makes me immediately think its an
> > internal detail I shouldn't concern myself with and documentation
> > seems to agree as there is no mention of this either.
> 
> We agree that external code shouldn't depend on internal details of
> other programs.  "OpenPGP packets", on the other hand, is a
> standardized, well-known, interoperable data structure defined in RFC
> 4880.

I would concur with the "well-known" part of that sentence, but claiming
that on a single datapoint (me) is bit weak.

All I am trying to say here I guess is that this is something I wasn't
expecting and why I wasn't. What you end up doing (or not) with that
information is your choice. No problem for me either way now that I am
"in the know"; I am just entertaining the possibility that I am not the
least knowledgeable gpg user… but I might be.


> > [dkg wrote:]
> >> hmmm -- if the goal is to merge simple keyrings together into a larger
> >> simple keyring, then the simplest merge mechanism is /bin/cat:
> >> 
> >>   /bin/cat /etc/apt/trusted.gpg.d/*.gpg > $TMPDIR/pubring.gpg
> >
> > Huh, this (like many of your other suggestions I snipped) is
> > interesting. I just naturally assumed that simple appending wouldn't
> > work. Is this a dependable interface or is it just working for now™?
> 
> if *.gpg files are simple keyrings (concatenated OpenPGP packets) then
> yes, /bin/cat is a dependable interface, as it is for concatenating any
> arbitrary data.
> 
> The only difference will be if there are duplicate keys in the exposed
> keyset, in which case gpg would accumulate/merge them, producing minimal
> output, while cat will produce a larger keyring with duplicate keys in
> them.  Such a situation could arguably be a bug in the multiple packages
> providing duplicate authoritative keys in the first place, though.

A bit of playing with this reveals that if you have the same key in
multiple keyrings, merge them together with cat and you try to delete
the duplicated key with --delete-keys gpg will delete only the first
occurrence of it.

Not really a problem for apt-key as the implementation prevents this "by
accident", but I will ensure it keeps that way as while it is indeed
unlikely (and bug worthy) to have the same key multiple times in
a keyring, it is a situation which is hard to prevent while merging
multiple keyrings from different sources together. A user could have
imported a key manually before and later on the same key appears in
a packaged keyring for example.


> > All other usecases are kinda optional, but as this is security related
> > even the more obscure ones are on a keep-forever list. "update" is on
> > the way out through (no point of this if a package can just add/remove
> > entire keyrings on its own).
> 
> what does "on the way out out through" mean?  do you mean "on the way
> out though" ?  can we ignore "apt-key update" then?

Well, for certain levels of "ignore". The debian archive keyring isn't
using it anymore (for years), but even Debian still ships some packages
(I thought it would be one, but that one isn't even doing it. It just
mentions it and uses add (#390449), but codesearch.d.n has some real
hits, so some more patches to write… :/ ) and with derivatives the state
is even more uncertain.

The plan is to weed out the usage of update/add/del in maintainer
scripts in the VERY long run. Given that you likely need del to undo
whatever you did before with update/add in scripts that can take
a while.

If that is done we can demote the gnupg dependency to a recommends (or
suggests). Well, slightly earlier I guess as we can let the -keyring
packages recommend gnupg for the migration handling as we did for
debian-archive-keyring.


> > "add" has mostly the same faith, but as it is you can change keys this
> > way as well (import new expire dates, sigs, …). Not that you should,
> > but you can. "del" of course should work, even if rarely used (yes,
> > people complain if they can't remove the debian archive keyring – or
> > at least some of them… *shrug*).
> 
> do these need any sort of additional work?  if you're doing keyring
> management here through gpg (of whatever version), the right thing would
> be to make a new temporary keyring, import all known updates of the key
> for merging, and export to a single new simple keyring file.

The thing is that we have multiple keyrings and need to preserve them as
best as possible. Consider #558784 where you remove one out of the
"many" Debian keys. That change has to be preserved. On the other hand,
if all these keys are removed later by the package (next release) you
want those keys gone, not that they are bundled still in some other
keyring just because you removed one of them before. Same on package
purge and similar situations. So in the (unlikely) event of a user
modifying the keyrings, we deal with it in-place and with gpg, so that
in the common case of changes to entire keyrings based on packaging
changes we don't need gpg (given that the first one is done by very few
users manually, while the later is done by postinst scripts for all
users, so if we ever want to drop gpg hard-dependency, those have to
work without it).


> > net-update was never a thing for Debian and as far as I know it
> > going anywhere either (better plans mostly).  That leaves the basic
> > readonly wrappers list, finger(print) and export. And then there is
> > the bad apple 'adv' which can be used to do all the good/bad things
> > you could come up with.  Mostly used by users to --recv-key from the
> > net into apts trusted keyring or --refresh-keys. Both are not exactly
> > on my favorite list either, but people wanna do it and who am I to
> > forbit/break it…
> 
> I'm happy to ignore net-update (which afaict has been a source of
> security problems for those distros which have enabled it) and to let
> users of adv be responsible for their own decisions for now -- if any
> tool in debian is programmatically using "apt-key adv" then i'd be
> interested in knowing more about it.

codesearch.d.n suggests a few places. Most are in documentation (install
instructions) and testcode, but at the very least salt, ansible and
puppet seem to provide modules to use apt-key adv to import keys from
remote servers…

And I know people who do --refresh-keys in cron jobs. There is actually
a bug requesting apt to do this.

I am not a huge fan of either as this means there is another vector
where MITM and such things are a threat, but oh well.


> > There is only a very minor thing I tried this gpg/gpgv shenanigan
> > before for which is that apt-cdrom traditionally shows output from
> > gpgv which doesn't care for trust, while gpg seems rather hard to be
> > convinced to not care about it (e.g. not printing "gpg: WARNING: Using
> > untrusted key!" as that is quiet scary), regardless of how hard you
> > try (--no-auto-check-trustdb --trust-model always). That was at least
> > the plan until you came along with 'cat'…
> 
> I'd be happy to talk more about trust models if you want to.  At the
> moment, my understanding is that the specific keys shipped are all
> simple keyrings consisting of explicitly trusted apt archive signing
> keys.

Yeap. We circumvent the gpg/web of trust model entirely and simply say:
Every key we know (= is in /etc/apt/trusted.gpg file or in a file
matching /etc/apt/trusted.gpg.d/*.gpg) is a trusted key. Everyone who
can sneak a key into this root-owned location could just as well
circumvent any additional limitation anyhow. Besides, which limitation
could we possibly enforce (it is not like all Debian users will have
a trustpath to the Debian archive keyring. Even less so for other
archives) and gpgv has no concept of trust either and that is what we
aspire to use. :)


> > Just as a summary: apt/experimental works fine with 2.0 already, 2.1 has
> > these two rather minor issues we can easily solve one way or the other.
> > It isn't "optimized" yet and your tips give a good idea what to do in
> > this regard, but at least functional and (hopefully) good enough to make
> > apt/stretch deal with newer gpg's for the forseeable future (worked
> > "well" in the 2.0 to 2.1 jump, but I can't be unlucky twice, can I…).
> 
> let me know if you see other things that need fixing, but i'd be happy
> to see apt drop all dependencies on gnupg and just rely on gpgv for its
> checks, which i think is likely to be both easier and safer.

I just implemented the cat-merging, so Release file verification is
again a gpgv-only operation in /experimental soon. /sid and earlier are
anyway.


> > Sidenote: The apt-key above can be told to use gpg or gpg2 and gpg2
> > (2.1, not 2.0) seems to be considerably slower. Is this known and/or do
> > you have an idea why this is so? Its not like apt-key operation would be
> > really time-critical, so usually nobody would notice, but our apt-key
> > testcase makes it visible as it runs slower…
> 
> There has been some discussion of a slowdown due to the check-trustdb
> step recently:
> 
>  https://bugs.gnupg.org/gnupg/issue2019
> 
> I don't think this should be a critical performance issue for apt at the
> moment, though, and it shouldn't be relevant at all for the gpgv case.

Yeah, could be as we have an explicit --check-trustdb call as earlier
gpg versions were very unhappy if no trustdb was around (just like with
secring, which now seems to be a trigger for migration handling, even if
it is an empty file…).

As said, not really a problem. I just noticed it in our testcases.


> I'm now no longer sure what followup this ticket needs, though.  Should
> i close it now, or do you need anything else from gnupg/gnupg2?

As said, apt needs nothing, with the cat-merging and a more careful
importing (so that the result is always a simple keyring)
apt-key/experimental works now fine with gpg 2.1. Earlier versions
should be fine with gpg 2.1 anyway (just that it has "unrelated" bugs).

The bug was about the importing of keyboxes. If that isn't supposed to
work that isn't a bug (expect documentation maybe). You reported your
observation that gpgv can't deal with keyboxes already upstream. The
slowdown is known upstream as well. --delete-keys only working on the
first key is a bit strange from a user POV, but well, I can see why
a unique fingerprint is a stop word…

So, nothing unique to this bugreport I presume,
but while talking into the blue:

There is something I found just now which is a bit strange: gpgv
--status-fd is documented in /usr/share/doc/gnupg/DETAILS.gz.

In GOODSIG it says that each sig has either "GOODSIG, BADSIG or ERRSIG"
code, which isn't true as an expired sig comes with EXPSIG. Worse, in
VALIDSIG it is said that "This is the same as GOODSIG […] Both status
lines are emitted for a good signature.", while as mentioned earlier
gpgv does not emit a GOODSIG for expired sigs, but does emit VALIDSIG.
Would be nice if the documentation could be updated to match reality.

Oh, and I wonder, are there any plans to "replace" the long keyids with
fingerprints here at some point (new names, new parameter, whatever)?
Unlikely perhaps, but archives with the same long keyid could be fun…


Best regards

David Kalnischkies
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-gnupg-maint/attachments/20150707/de15ba3f/attachment.sig>