[DRE-maint] googleapis packaging

Mon Sep 27 03:27:03 BST 2021

On 2021-06-03 02:05 +0100, Wookey wrote:
> On 2021-04-23 02:46 +0100, Wookey wrote:

So, I got back to having another look at this.

[Background for those new to this conversation: I'm working on
packaging tensorflow and that has googleapis as a build-dep. Looking
at this package it seems like we have a coresponding package for
golang (which to be fair is separate upstream now) and subset packages
for ruby and java, but not the other supported-by-upstream
languages). Perhaps it makes sense to tidy this up to be more
consistent in debian and build the full googleapis set of
headers/classes/libraries for all the supported languages from one
consistent source?]

> > There is a package in debian built from
> > https://github.com/googleapis/googleapis
> > ruby-googleapis-common-protos-types but its 'source' is the _output_
> > of processing upstream with the protocol buffer compiler. Arguably
> > that's not the 'preferred form of modification' and that package should be fixed?
> > it's also a version from 2018 so probably quite a lot has changed.
> > 
> > It seems to me that the right way to package this is to generate all
> > the language outputs from one source package (ruby, java, gp, python,
> > nodejs, php, csharp) maybe there are more?,

> > I did try just building the package from upstream and had some success
> > after installing protobuf-compiler and protobuf-compiler-grpc and
> > python3-grpc-tools, ruby-google-protobuf and protobuf-compiler-grpc-java-plugin
> > 
> > 3 out of 7 languages build. I don't know which language bindings tensorflow actually wants.

Subsequent research suggests that it wants C/C++ headers for
'google/monitoring/v3', (defined in third_party/googleapis/BUILD)
although I've still not worked out where they get used.

> I've committed these experiments to a salsa repo here:
> https://salsa.debian.org/deeplearning-team/googleapis
> 
> Anyone is welcome to see if they can get some of the broken languages
> building, or otherwise fill out the debian dir.

So, I've now got 6 languages working (cpp, java, php, python, csharp, ruby).
node, and objectivec still fail, and remain to be grokked.

cpp was failing due to a bug in the compiler which fell over when
'$field' was mentioned in a C++ style comment.  Upstream's fix for
this for the time being is to remove that $ in the comment. Not very
impressive, but it'll do for now.
https://github.com/GoogleCloudPlatform/cpp-samples/issues/113
It was nominally fixed in grpc back in March but hasn't been
incorporated or released yet, so we get to work around it.
https://github.com/grpc/grpc/pull/25538

ruby and php were both failing because their grpc compilers/parsers
only support proto3 format, not proto2 format, and there are a couple
of proto2 format files, notably in the grpc compiler files itself. I
don't know if those are just out of date or if there is a good reason
that grpc still has proto2 files? I modified the makefile to check for
the 'proto2' descriptor and if it was present, to remove those files
from the list to be processed.

ruby is more picky than php as the code generator parsed dependencies
and then checked those for being proto2 type. So I had to just resort
to specifically excluding
google/cloud/bigquery/storage/v1alpha2/protobuf.pb.cc from the
processed files list (as writing a recursive parser to do the checks
myself in the makefile seemed excessive.)

node: gives this error from make OUTPUT=./debian/node LANGUAGE=node
 "protoc-gen-node: program not found or is not executable"
 Now there is a /usr/bin/grpc_node_plugin and a /usr/bin/protoc-gen-go
 (in golang-goprotobuf-dev), and a /usr/bin/protoc-gen-rust (in
 protobuf-codegen), but I can't find a /usr/bin/protoc-gen-go.
 Anyone know where this should come from?

objectivec: has the same issue as node:
 protoc-gen-objectivec: program not found or is not executable
 (again there is a /usr/bin/grpc_objective_c_plugin)

Where to put these files?

This is presumably different or each language. I know how things work
in C/C++ (and perl), and have some idea of java, but everything else
is a bit of a mystery to me.

Existing ruby package (ruby-googleapis-common-protos-types) puts ruby stuff in:
/usr/lib/ruby/vendor_ruby/google

Existing golang package (golang-google-genproto-dev) puts them in:
/usr/share/gocode/src/google.golang.org/genproto/googleapis/   
(and also has some extra stuff in
/usr/share/gocode/src/google.golang.org/genproto/protobuf - is that important I wonder?)

There is a java package libgoogle-common-protos-java which contains
proto-google-common-protos-1.16.0.jar, which has a subset of the output from 
make OUTPUT=./debian/java LANGUAGE=java
compiled into classes

So we have this 'common' subset packaged for both ruby and java. The
java one comes from
https://github.com/googleapis/java-common-protos. Is there a good
reason to package these subsets but not the whole googleapis repo,
or was it just expedient? Are they both being kept in sync upstream?

Trying to build and install the full set I have these packages and install directories:
Do they make sense to people?

googleapis-dev (C/C++)
/usr/include/google/

libgoogleapis-java
Where should this go? Java policy suggests libgoogleapi-0.jar in /usr/share/java/?
(I have no idea what the version number of this package is - googleapis repo doesn't
seem to go in for versions/releases).

python3-googleapis
/usr/lib/python3/dist-packages/google?

ruby-googleapis
/usr/lib/ruby/vendor_ruby/google

php-googleapis
Where does php look for libraries/headers/definitions?

csharp-googleapis
And csharp?

So, this seems to be quite a large rabbithole I have fallen down. I'm
still not sure that this is the right approach, but it seems like this
supplies the general case, rather than various subsets, so is probably
worth doing. But I could be confused as I don't actually know how this
stuff is used in practice.

I've cc:ed maintainers for some of the packages mentioned above as
they presumably know more about what is what.

The salsa repo is out of date as send this. I'll update it to match my state tomorrow.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/pkg-ruby-extras-maintainers/attachments/20210927/7e135ecb/attachment-0001.sig>