[PKG-Openstack-devel] ongoing OpenStack packaging in Debian

[Thomas Goirand]

On 02/21/2017 07:58 PM, Allison Randal wrote:
> (Brief note here: the OpenStack CI/CD infrastructure uses
> git-buildpackage to build the "deb-" prefixed repositories into actual
> Debian packages. The current usage of git-buildpackage is not using the
> overlay option, which is why it requires a complete copy of the upstream
> source code in the git repository hosted by OpenStack. It is possible
> for us to choose to use the overlay option in the future.)

I would recommend against using the overlay option, because it makes it
very complicated to manage quilt patch then. The current way things are
setup is the most convenient for the package maintainers. The only issue
is that infra people don't like it, because of the duplication itself.

> The "deb-" prefixed repositories are what OpenStack Infra calls "managed
> projects", meaning that they automatically sync changes from the
> mainline OpenStack repositories. Which simplifies things for us.

You should also keep in mind the generalization. The packages don't all
have git.openstack.org as upstream. Sometimes, it's github, or others,
for 3rd party libs.

> When you submit a patch to a "deb-" repo in Gerrit, the package gets
> rebuilt in OpenStack CI/CD in the check pipeline and the gate pipeline.
> The package is only built (on an untrusted worker for safety), it
> doesn't run any tests or lintian checks.

Not correct. There's lintian checks, and the build is failed if the
package doesn't pass the lintian checks.

> The gate build uploads the built packages to:
> 
> http://tarballs.openstack.org/packaging-deb/

Technically, the gate build, but it's the POST job that builds and
upload to tarballs.o.o.

> In subdirectories named for the repository built, and the git hash of
> the specific build, for example:
> 
> http://tarballs.openstack.org/packaging-deb/deb-nova/uploads/2fb1c3bc96a52721b6f05bbdfaf0306b8705ac2e/
> 
> These packages are signed by the Infra key

Nop, they aren't signed with the infra key. They are signed with a key
which is discarded after build, so the signing part is something that
would needs to be rethought. That is, if we care about it. In fact, I
don't think we need to care about package signatures: the signature of
packages is never used for any check, only the *repository* key is.

> and then loaded into a
> Debian mirror used by OpenStack Infra for testing. This mirror has two
> archive areas for each OpenStack release, one for the packages
> maintained by OpenStack PKG, and the other for custom backports of
> dependencies for those packages. For Newton these are:
> 
> http://mirror.dfw.rax.openstack.org/debian-openstack/dists/jessie-newton/
> http://mirror.dfw.rax.openstack.org/debian-openstack/dists/jessie-newton-backports/
> 
> (It also hosts a full unmodified mirror of Jessie, for use by the build
> slaves at http://mirror.dfw.rax.openstack.org/debian/dists/)

Which unfortunately can't be used by openstack pkg (it'd be useful for
example when building the chroot), because that Jessie mirror is made
with reprepro, and therefore isn't signed. :(

> Paul needs to do some manual work for us here, because the directories
> for Ocata packages and backports have to be created before we can start
> publishing packages.

Correct. This was discussed in Barcelona. Basically, it would just be a
reprepro command to type, to import all Newton packages into the Ocata
repository. Thanks to the pool structure, this only adds indices files,
we don't need to actually copy all packages to a new place.

> The CI/CD process is not running integration tests on packages, so ther
> are no cross-package dependencies. But, from what I understand, Thomas
> was manually timing his changes to Gerrit to control for dependencies.

I'm not sure what you're saying here.

For running tempest tests and checking everything works, I was running
them on a separate hardware using a Debian live system. All of it is
included in the source packages (see openstack-meta-packages). I don't
think it would be too hard to move it all to a VM in infra, though I
doubt it'd be useful to run the tempest tests on each build (this would
be too resource intensive and would take too long to run).

> Also, sometimes, this whole process can be quite slow. The patch builds
> 3 times before it lands, which can take a long time if the gate is under
> heavy usage. It might only take 10 minutes for a change to get into the
> gate, but take an entire day to get the package published.

I *never* took a day.

The quickest build I could observe was about 7 minutes, on which maybe 6
are spent creating the build environment (ie: installing sbuild,
setting-up the chroot, etc.). There's room for improvement, and I'm sure
we could get it as quick as 1 or 2 minutes for the simplest packages. I
didn't have time to focus on this, as I was in a hurry to deliver Newton
on time, which I did.

Then, there's the fact that packages are built once when someone sends a
change request, then once for the gating, and once more for the POST
job. Only the files from the 3rd build are used to publish the actual
package. I've been told by the infra team that, with zuul 3, we would be
able to keep the files from the gating and use them for the POST job,
which would reduce the number of build to 2 only. IMO, that's the only
improvement we should attempt.

> That's all for now. I'm working on getting a first few Ocata packages
> through, and will let you know how that goes.

Once Paul has enabled the Ocata repo, the first patch that should land
is for openstack-pkg-tools, so that it uses the correct debian/ocata
branch. So branch openstack-pkg-tools to debian/ocata, and work on it
there. A simple "grep -r newton *" will IMO be enough to get you going.
Then the new openstack-pkg-tools will be pushed to the newly created
ocata repo, and you should be good to go for working on other packages.

If you need me for more explanation, let me know, I'll be very happy to
help if I can.

Cheers,

Thomas Goirand (zigo)