[pkg-gnupg-maint] Bug#790665: Bug#790665: gpg2 fails to import gpg2 created keyring in a gpg1 created keyring

[Daniel Kahn Gillmor]

Hi David--

On Sun 2015-07-05 11:10:59 -0400, David Kalnischkies wrote:

> I don't know where you get the "stdin" from. What I am trying here is
> merging a keybox into a simple keyring in a single command, no funky
> business with stdin.

heh.  i consider "messing with another program's internal data storage"
to be the "funky business", while reading standardized file formats from
a file on stdin to be "standard unix practices". :)

Let's see if we can get both habits to work :)

> As basically every documentation I came across says "gpg --import
> [filename]" I kinda expect that to work with whatever file gpg ends up
> producing.

gpg will produce files to stdout (and with --output, if you prefer) that
will work fine for stdin (and as a named argument to --import, if you
prefer).  Its internal key storage, however, is not guaranteed to be
either of those formats.

> "OpenPGP packets" on the other hand makes me immediately think its an
> internal detail I shouldn't concern myself with and documentation
> seems to agree as there is no mention of this either.

We agree that external code shouldn't depend on internal details of
other programs.  "OpenPGP packets", on the other hand, is a
standardized, well-known, interoperable data structure defined in RFC
4880.

> This side-comment was mainly about partial upgrades and such. For
> example, using gnupg (>2.1) and gpgv(<2) together doesn't work that
> well. Neither does a *-archive-keyring package which happens to ship
> a keybox while you haven't the tools to deal with it.

I know of no *-archive-keyring packages that ship keyboxes.  I would
recommend against any *-archive-keyring package shipping a keybox
format, fwiw.  Do you know of any?  Please file a bug (and feel free to
cc me on it) if they start doing so.

> [dkg wrote:]
>> hmmm -- if the goal is to merge simple keyrings together into a larger
>> simple keyring, then the simplest merge mechanism is /bin/cat:
>> 
>>   /bin/cat /etc/apt/trusted.gpg.d/*.gpg > $TMPDIR/pubring.gpg
>
> Huh, this (like many of your other suggestions I snipped) is
> interesting. I just naturally assumed that simple appending wouldn't
> work. Is this a dependable interface or is it just working for now™?

if *.gpg files are simple keyrings (concatenated OpenPGP packets) then
yes, /bin/cat is a dependable interface, as it is for concatenating any
arbitrary data.

The only difference will be if there are duplicate keys in the exposed
keyset, in which case gpg would accumulate/merge them, producing minimal
output, while cat will produce a larger keyring with duplicate keys in
them.  Such a situation could arguably be a bug in the multiple packages
providing duplicate authoritative keys in the first place, though.

> If its the former I would be tempted to declare keyboxes a no-go for
> -keyring packages dropping files into that directory and use this + gpgv
> to lessen our dependency on gnupg in the future.

i think we're in agreement here.  hopefully we will end up providing
gpgv from the 2.1.x branch soon (1.4.x will move to gpgv1 if we find we
still need to keep it around), which means that apt-key can silently
transition to using the more modern codebase.

> After all, that was the plan behind introducing trusted.gpg.d/, to get
> to a point where normal operations (-keyring package installation/
> removal and Release file verification) do not require gpg anymore.
> I dropped that plan after we hit the 40 keyring limit thinking that we
> would need gpg to merge them together…

making apt depend only on gpgv2 sounds good to me, modulo concerns about
apt-key management below.

> Setup:
> * We have a bunch of keyrings: trusted.gpg more or less under our
>   control and here for compatibility reasons mostly while distributions
>   as well as users are supposed to drop keyrings into trusted.gpg.d/.
>   That can be freakishly many, so at some point people were hitting 40
>   limit and if 1 becomes a reality… some sort of merging seems needed.
> * As far as I understood you so far I can end up with a lovely mixture
>   of simple keyrings and keyboxes in this directory depending on what
>   gpg version happens to create these keyrings and how. The less
>   strict the better usually, but that is a bonus objective.

all supported versions of gpg (and probably all versions til the end of
time) can create simple keyrings via "gpg --export".

> * apt-key is supposed to handle all the nitty gritty details of working
>   with gpg/gpgv/gpg2/gpgv2 in whatever version we have to deal with in
>   oldstable and stable (so that (partial) upgrades either way work).

This is a good policy.  It might even be that that means "no using gpg
at all" at some point. :)

> Usecases:
> * apt-key verify – undocumented as it is supposed to be used only
>   internally by apt to verify Release files (this was hardcoded before
>   in libapt directly, I just moved it out for the merging). Wrapping
>   around "something" which verifies the given file + detached sig
>   (clearsigned files are split up before) and gives some info about
>   which is the result of that: good sig, bad sig, unknown key, that sort
>   of stuff.

This should use gpgv, as you've said above.  the other usecases below
might be trickier.


> All other usecases are kinda optional, but as this is security related
> even the more obscure ones are on a keep-forever list. "update" is on
> the way out through (no point of this if a package can just add/remove
> entire keyrings on its own).

what does "on the way out out through" mean?  do you mean "on the way
out though" ?  can we ignore "apt-key update" then?

> "add" has mostly the same faith, but as it is you can change keys this
> way as well (import new expire dates, sigs, …). Not that you should,
> but you can. "del" of course should work, even if rarely used (yes,
> people complain if they can't remove the debian archive keyring – or
> at least some of them… *shrug*).

do these need any sort of additional work?  if you're doing keyring
management here through gpg (of whatever version), the right thing would
be to make a new temporary keyring, import all known updates of the key
for merging, and export to a single new simple keyring file.

> net-update was never a thing for Debian and as far as I know it isn't
> going anywhere either (better plans mostly).  That leaves the basic
> readonly wrappers list, finger(print) and export. And then there is
> the bad apple 'adv' which can be used to do all the good/bad things
> you could come up with.  Mostly used by users to --recv-key from the
> net into apts trusted keyring or --refresh-keys. Both are not exactly
> on my favorite list either, but people wanna do it and who am I to
> forbit/break it…

I'm happy to ignore net-update (which afaict has been a source of
security problems for those distros which have enabled it) and to let
users of adv be responsible for their own decisions for now -- if any
tool in debian is programmatically using "apt-key adv" then i'd be
interested in knowing more about it.

> Well, /sid and earlier all use gpgv exclusive for verification, but that
> has the limited number of keyrings problem, so in /experimental "apt-key
> verify" is taught to use gpg to first merge the keyrings and then calls
> gpgv with this big keyring. Expect that this isn't working well if you
> end up on systems with gpg2 used for the merging and gpgv for the
> verify, so I am probably going to drop that in favor of just using "gpg
> --verify" instead of gpgv all the time.

Please don't do that.  Please instead aggregate the simple keyrings with
cat and use gpgv.

> There is only a very minor thing I tried this gpg/gpgv shenanigan
> before for which is that apt-cdrom traditionally shows output from
> gpgv which doesn't care for trust, while gpg seems rather hard to be
> convinced to not care about it (e.g. not printing "gpg: WARNING: Using
> untrusted key!" as that is quiet scary), regardless of how hard you
> try (--no-auto-check-trustdb --trust-model always). That was at least
> the plan until you came along with 'cat'…

I'd be happy to talk more about trust models if you want to.  At the
moment, my understanding is that the specific keys shipped are all
simple keyrings consisting of explicitly trusted apt archive signing
keys.

> Just as a summary: apt/experimental works fine with 2.0 already, 2.1 has
> these two rather minor issues we can easily solve one way or the other.
> It isn't "optimized" yet and your tips give a good idea what to do in
> this regard, but at least functional and (hopefully) good enough to make
> apt/stretch deal with newer gpg's for the forseeable future (worked
> "well" in the 2.0 to 2.1 jump, but I can't be unlucky twice, can I…).

let me know if you see other things that need fixing, but i'd be happy
to see apt drop all dependencies on gnupg and just rely on gpgv for its
checks, which i think is likely to be both easier and safer.

> Sidenote: The apt-key above can be told to use gpg or gpg2 and gpg2
> (2.1, not 2.0) seems to be considerably slower. Is this known and/or do
> you have an idea why this is so? Its not like apt-key operation would be
> really time-critical, so usually nobody would notice, but our apt-key
> testcase makes it visible as it runs slower…

There has been some discussion of a slowdown due to the check-trustdb
step recently:

 https://bugs.gnupg.org/gnupg/issue2019

I don't think this should be a critical performance issue for apt at the
moment, though, and it shouldn't be relevant at all for the gpgv case.
if you think it is, clearer profiling is probably the way to go
(e.g. what specifically is gpg being asked to do that is slower?) I can
help you follow up with upstream about that if you want to do the
profiling.

I'm now no longer sure what followup this ticket needs, though.  Should
i close it now, or do you need anything else from gnupg/gnupg2?

All the best,

    --dkg