[PKG-Openstack-devel] ongoing OpenStack packaging in Debian

[Allison Randal]

Hi all,

I'm at the OpenStack PTG meeting this week, and actively working on the
Ocata packages for Debian. It's a little bit of a recovery effort, so
I'll share what I discover and what I'm working on here, so the entire
team can help fill in missing pieces and contribute to the effort.

I'm CC'ing Spyros Trigazis, who has volunteered to help with the Magnum,
Heat, and Barbican packages (and will join this mailing list). And, also
Omer Anson and Lihi Wishnitzer, who expressed an interest in packaging
Dragonflow for Debian.

In the past, Thomas posted some information about the new OpenStack
CI/CD setup for Debian packages here:

http://thomas.goirand.fr/blog/?p=278

And, there are some even older instructions on the OpenStack PKG pages
on Alioth, that predate the OpenStack CI/CD setup and so aren't accurate
anymore:

https://openstack.alioth.debian.org/

That wasn't quite enough information to pick up the work, so I spent
some time today with Paul Belanger from the OpenStack Infra team, who
did the majority of the work in setting up OpenStack CI/CD to handle
Debian packages. Here's a brief summary of my notes from talking with
Paul. (This may seem overly detailed, but I figure it's best to start
out explicitly, both for new team volunteers, and so we can all agree up
front if this is an accurate understanding of the way OpenStack PKG has
done things so far, and also if it's the way we want to continue doing
things.)

=========
All package development for Debian OpenStack packages is done in
OpenStack's git/gerrit setup. Each project has a separate git repo,
prefixed with "deb-". So, for example, OpenStack hosts a repository:

https://git.openstack.org/cgit/openstack/deb-nova

Which contains a complete copy of the mainline OpenStack nova
repository, including the master branch and stable branches for liberty,
mitaka, newton, and ocata.

The Debian packaging work is done on a separate branch for each
OpenStack release, the branches for Newton are named debian/newton, and
the branches we need to create for Ocata will be named debian/ocata. The
Debian packaging branch contains a complete copy of the source code for
the OpenStack upstream code, plus the addition of the debian/ directory
containing the usual set of packaging file.

(Brief note here: the OpenStack CI/CD infrastructure uses
git-buildpackage to build the "deb-" prefixed repositories into actual
Debian packages. The current usage of git-buildpackage is not using the
overlay option, which is why it requires a complete copy of the upstream
source code in the git repository hosted by OpenStack. It is possible
for us to choose to use the overlay option in the future.)

The "deb-" prefixed repositories are what OpenStack Infra calls "managed
projects", meaning that they automatically sync changes from the
mainline OpenStack repositories. Which simplifies things for us.

When you submit a patch to a "deb-" repo in Gerrit, the package gets
rebuilt in OpenStack CI/CD in the check pipeline and the gate pipeline.
The package is only built (on an untrusted worker for safety), it
doesn't run any tests or lintian checks.

The gate build uploads the built packages to:

http://tarballs.openstack.org/packaging-deb/

In subdirectories named for the repository built, and the git hash of
the specific build, for example:

http://tarballs.openstack.org/packaging-deb/deb-nova/uploads/2fb1c3bc96a52721b6f05bbdfaf0306b8705ac2e/

These packages are signed by the Infra key, and then loaded into a
Debian mirror used by OpenStack Infra for testing. This mirror has two
archive areas for each OpenStack release, one for the packages
maintained by OpenStack PKG, and the other for custom backports of
dependencies for those packages. For Newton these are:

http://mirror.dfw.rax.openstack.org/debian-openstack/dists/jessie-newton/

http://mirror.dfw.rax.openstack.org/debian-openstack/dists/jessie-newton-backports/

(It also hosts a full unmodified mirror of Jessie, for use by the build
slaves at http://mirror.dfw.rax.openstack.org/debian/dists/)

Paul needs to do some manual work for us here, because the directories
for Ocata packages and backports have to be created before we can start
publishing packages.

Once the package is available in the CI/CD package archive, the package
will be imported/installed on a trusted slave.

The CI/CD process is not running integration tests on packages, so ther
are no cross-package dependencies. But, from what I understand, Thomas
was manually timing his changes to Gerrit to control for dependencies.


Also, sometimes, this whole process can be quite slow. The patch builds
3 times before it lands, which can take a long time if the gate is under
heavy usage. It might only take 10 minutes for a change to get into the
gate, but take an entire day to get the package published.


That's all for now. I'm working on getting a first few Ocata packages
through, and will let you know how that goes.

Allison