[Openstack-devel] Current state of the Folsom packaging

[Thomas Goirand]

Hi,

As requested by Loic Dachary, here's a report after 2 weeks of my work 
on the Folsom packaging for Debian. I also think it's a good idea to 
tell where I am at.

I'm talking about the things I've been working on, and I would like to 
highlight the fact that Roland Mas (aka lolando) worked and debugs some 
of my work, so he's to be associated as well with what's bellow, even if 
I'm talking at the first person.

1/ Python libraries we depend on

I made sure that all packages that we need are there, including some of 
the python libraries that aren't in Wheezy. This includes: python-cliff, 
python-cloudfiles, python-django-appconf, python-django-compressor, 
python-django-openstack-auth, python-tox, python-warlock.

I know that some of these libraries are being packaged by others, and 
even some are probably (at the time of writing) either in the NEW queue 
or already in SID. But I needed them to be able to build the rest of 
Openstack, and I couldn't wait, so I made the missing bits. (I can't 
remember which one I built from scratch, and which one were already on 
Alioth though, probably half/half...)

We will need to make check the state of all these libs at some point, 
and get in touch with the python module team, check ITPs, etc.

2/ Global state of our git repository on Alioth

All packages now point to the correct branch by default (eg: 
debian/unstable if we have the package in Wheezy, debian/experimental 
otherwise). In some packages, the default branch wasn't set at all. It's 
not easy to find, so here's how to do (for the record):

git symbolic-ref HEAD refs/heads/debian/experimental

If you do that, then debian/experimental will be the default branch that 
is checkout after an initial clone.

I also made sure that absolutely all of our packages had a debian/rules 
get-vcs-source target that works. This Makefile target fetches the 
upstream branch, and creates the orig.tar.xz out of it (using the 
upstream tags).

There's at least 1 or 2 packages for which I couldn't do this, because 
there was no upstream tags. I just had this fixed upstream for 
pythong-django-openstack-auth (eg: upstream added a tag), but I believe 
there's one more without upstream tags (we will need to ping upstream 
and explain why tags are mandatory for us...).

3/ Automatic build of all packages

I wrote a script to rebuild everything. It's a simple script to build 
all sources that we maintain in the PKG Openstack team. I wrote it 
because it was difficult to know what the build order is, and we needed 
to make sure that there was no recursive build-depends.

Currently, all packages are building fine automatically, except some 
unit tests which are failing, notably in Glance and Nova. Ghe Rivero 
wrote on IRC that he will have a look and try to find why they are failing.

The result is a usable Debian repository which you can add to your 
sources.list.

To use the script simply do:
git clone \
   http://anonscm.debian.org/git/openstack/openstack-auto-builder.git
cd openstack-auto-builder
./build_openstack

Once the script has started, go get yourself some coffee, it takes a bit 
of time to build everything. Note that it will fail on Glance and Nova 
at least because of the unit tests. To solve that, do:

cd source/nova/nova
DEB_BUILD_OPTIONS=nocheck git-buildpackage

and once done, return to the build script.

As a bonus, once everything is built, you can do:

echo deb file://`pwd`/repo/debian experimental main \
   >>/etc/apt/sources.list

and start installing packages with apt.

Note that the build_openstack script expect a gnupg key. What I do is 
that I generate a temporary one on my test server and fill up the 
PGP_SIGN_KEY_ID in the built script.

4/ Debconf and pkgos functions

Many packages had non policy compliant maintainer scripts when dealing 
with Debconf. Just as a reminder, our scripts *have* to:
- *NOT* have the config file marked as conffiles, which means 
installation and removal of the config file has to be done in postinst / 
postrm.
- Read the (eventually modified by hand) config file in the config 
script and use db_set accordingly.
- Work even in non-interactive mode

In many ways, both nova, glance, keystone and cinder were not respecting 
the above rules from the Debian policy. This is all fixed now.

Since we had a lot of redundancy in both Cinder, Nova, Glance and 
Keystone, I decided to write functions that would be reused from one 
package to another. This has many advantage:
- Less code duplication
- More testings of functions that are more and more robust
- Faster addition / porting of features from package to package

Currently, we have this in the form of a debian/pkgos_func file which is 
present in all these 4 packages. There has been talks in IRC about 
creating an openstack-pkg-tools package on which we would build-depend, 
to store the files there. I was not really convinced that it was already 
the time to do it, and that it was better to first port all 4 package to 
the new system, but now that it's done (eg: all of the 4 packages are 
using pkgos_* functions), it might be worth to start cleaning-up and 
build such package. It also might be really good to have a common 
Makefile, since we have a lot of redundancy there as well (like, the 
get-vcs-source target and many more could be stored ONCE and FOR ALL 
packages, using an include: that'd be an awesome time saver...).

Anyway, I'm quite satisfied with the result. There is of course room for 
improvements in the functions of debian/pkgos_func, but the point being 
that 
{keystone,cinder-common,nova-common,glance-common}.{postinst,config} 
files are now *very* small and calling high level functions, and there's 
at least 3 times less code redundancy now. It took time to first design 
these, but now it's a way faster to write debconf stuff, and we could 
add a way more if needed. I like to have more things in debconf, because 
then it's easy to script with preseeds. I wouldn't mind add more of that 
if anyone spots packages where we should add some (probably

5/ Testing

I could test and check that all packages are installing. The debconf 
stuff is working well and seem to do what we shall expect from it.

But we're not up to a functional test of all Openstack yet. I'm 
currently a bit stuck with Glance, with my client returning an HTTP 500 
error with not much more info... Help would be appreciated here.

6/ KGB bot

Since it was tedious to tell the world what has been pushed to our 
Alioth repository, I installed an instance of the KGB bot, after I read 
about it on planet.debian.org. I thought this was a gadget, but in fact, 
I'm very happy I did this setup, this eases communication on IRC a lot. 
If you add a package to our Alioth Git repo, please let me know, and 
I'll add it to the KGB bot.

7/ Things to work on

The only service which I haven't really worked on yet is swift. It's 
unfortunate since Loic just wrote me he needs it. We need to talk about 
it a bit more probably.

I also need to work on Quantum. It has also an api-paste.ini file, and 
that one would also need the debconf thingy. Many packages have some 
sql_connection fields which we could also configure using 
dbconfig-common. I need input and advices on this, to know what I should 
be working on.

I haven't worked at all yet on horizon, this is to be done as well.

So, all together, I'd say that Folsom for Debian is getting into shape! 
Please help me to do tests with what we have already.

Cheers,

Thomas Goirand (zigo)