[Openstack-devel] Current state of the Folsom packaging

[Nicolas Marchal]

Hello Thomas,

Thanks for the hard work !

Do you know when the swift packages will be ready ?

Thanks
Nicolas

----- Mail original -----
> De: "Thomas Goirand" <zigo at debian.org>
> À: "PKG OpenStack" <openstack-devel at lists.alioth.debian.org>
> Cc: "ifremer" <ifremer at enovance.com>, "Ignace Mouzannar" <ignace.mouzannar at enovance.com>
> Envoyé: Vendredi 26 Octobre 2012 14:41:03
> Objet: Current state of the Folsom packaging
> 
> Hi,
> 
> As requested by Loic Dachary, here's a report after 2 weeks of my
> work
> on the Folsom packaging for Debian. I also think it's a good idea to
> tell where I am at.
> 
> I'm talking about the things I've been working on, and I would like
> to
> highlight the fact that Roland Mas (aka lolando) worked and debugs
> some
> of my work, so he's to be associated as well with what's bellow, even
> if
> I'm talking at the first person.
> 
> 1/ Python libraries we depend on
> 
> I made sure that all packages that we need are there, including some
> of
> the python libraries that aren't in Wheezy. This includes:
> python-cliff,
> python-cloudfiles, python-django-appconf, python-django-compressor,
> python-django-openstack-auth, python-tox, python-warlock.
> 
> I know that some of these libraries are being packaged by others, and
> even some are probably (at the time of writing) either in the NEW
> queue
> or already in SID. But I needed them to be able to build the rest of
> Openstack, and I couldn't wait, so I made the missing bits. (I can't
> remember which one I built from scratch, and which one were already
> on
> Alioth though, probably half/half...)
> 
> We will need to make check the state of all these libs at some point,
> and get in touch with the python module team, check ITPs, etc.
> 
> 2/ Global state of our git repository on Alioth
> 
> All packages now point to the correct branch by default (eg:
> debian/unstable if we have the package in Wheezy, debian/experimental
> otherwise). In some packages, the default branch wasn't set at all.
> It's
> not easy to find, so here's how to do (for the record):
> 
> git symbolic-ref HEAD refs/heads/debian/experimental
> 
> If you do that, then debian/experimental will be the default branch
> that
> is checkout after an initial clone.
> 
> I also made sure that absolutely all of our packages had a
> debian/rules
> get-vcs-source target that works. This Makefile target fetches the
> upstream branch, and creates the orig.tar.xz out of it (using the
> upstream tags).
> 
> There's at least 1 or 2 packages for which I couldn't do this,
> because
> there was no upstream tags. I just had this fixed upstream for
> pythong-django-openstack-auth (eg: upstream added a tag), but I
> believe
> there's one more without upstream tags (we will need to ping upstream
> and explain why tags are mandatory for us...).
> 
> 3/ Automatic build of all packages
> 
> I wrote a script to rebuild everything. It's a simple script to build
> all sources that we maintain in the PKG Openstack team. I wrote it
> because it was difficult to know what the build order is, and we
> needed
> to make sure that there was no recursive build-depends.
> 
> Currently, all packages are building fine automatically, except some
> unit tests which are failing, notably in Glance and Nova. Ghe Rivero
> wrote on IRC that he will have a look and try to find why they are
> failing.
> 
> The result is a usable Debian repository which you can add to your
> sources.list.
> 
> To use the script simply do:
> git clone \
>    http://anonscm.debian.org/git/openstack/openstack-auto-builder.git
> cd openstack-auto-builder
> ./build_openstack
> 
> Once the script has started, go get yourself some coffee, it takes a
> bit
> of time to build everything. Note that it will fail on Glance and
> Nova
> at least because of the unit tests. To solve that, do:
> 
> cd source/nova/nova
> DEB_BUILD_OPTIONS=nocheck git-buildpackage
> 
> and once done, return to the build script.
> 
> As a bonus, once everything is built, you can do:
> 
> echo deb file://`pwd`/repo/debian experimental main \
>    >>/etc/apt/sources.list
> 
> and start installing packages with apt.
> 
> Note that the build_openstack script expect a gnupg key. What I do is
> that I generate a temporary one on my test server and fill up the
> PGP_SIGN_KEY_ID in the built script.
> 
> 4/ Debconf and pkgos functions
> 
> Many packages had non policy compliant maintainer scripts when
> dealing
> with Debconf. Just as a reminder, our scripts *have* to:
> - *NOT* have the config file marked as conffiles, which means
> installation and removal of the config file has to be done in
> postinst /
> postrm.
> - Read the (eventually modified by hand) config file in the config
> script and use db_set accordingly.
> - Work even in non-interactive mode
> 
> In many ways, both nova, glance, keystone and cinder were not
> respecting
> the above rules from the Debian policy. This is all fixed now.
> 
> Since we had a lot of redundancy in both Cinder, Nova, Glance and
> Keystone, I decided to write functions that would be reused from one
> package to another. This has many advantage:
> - Less code duplication
> - More testings of functions that are more and more robust
> - Faster addition / porting of features from package to package
> 
> Currently, we have this in the form of a debian/pkgos_func file which
> is
> present in all these 4 packages. There has been talks in IRC about
> creating an openstack-pkg-tools package on which we would
> build-depend,
> to store the files there. I was not really convinced that it was
> already
> the time to do it, and that it was better to first port all 4 package
> to
> the new system, but now that it's done (eg: all of the 4 packages are
> using pkgos_* functions), it might be worth to start cleaning-up and
> build such package. It also might be really good to have a common
> Makefile, since we have a lot of redundancy there as well (like, the
> get-vcs-source target and many more could be stored ONCE and FOR ALL
> packages, using an include: that'd be an awesome time saver...).
> 
> Anyway, I'm quite satisfied with the result. There is of course room
> for
> improvements in the functions of debian/pkgos_func, but the point
> being
> that
> {keystone,cinder-common,nova-common,glance-common}.{postinst,config}
> files are now *very* small and calling high level functions, and
> there's
> at least 3 times less code redundancy now. It took time to first
> design
> these, but now it's a way faster to write debconf stuff, and we could
> add a way more if needed. I like to have more things in debconf,
> because
> then it's easy to script with preseeds. I wouldn't mind add more of
> that
> if anyone spots packages where we should add some (probably
> 
> 5/ Testing
> 
> I could test and check that all packages are installing. The debconf
> stuff is working well and seem to do what we shall expect from it.
> 
> But we're not up to a functional test of all Openstack yet. I'm
> currently a bit stuck with Glance, with my client returning an HTTP
> 500
> error with not much more info... Help would be appreciated here.
> 
> 6/ KGB bot
> 
> Since it was tedious to tell the world what has been pushed to our
> Alioth repository, I installed an instance of the KGB bot, after I
> read
> about it on planet.debian.org. I thought this was a gadget, but in
> fact,
> I'm very happy I did this setup, this eases communication on IRC a
> lot.
> If you add a package to our Alioth Git repo, please let me know, and
> I'll add it to the KGB bot.
> 
> 7/ Things to work on
> 
> The only service which I haven't really worked on yet is swift. It's
> unfortunate since Loic just wrote me he needs it. We need to talk
> about
> it a bit more probably.
> 
> I also need to work on Quantum. It has also an api-paste.ini file,
> and
> that one would also need the debconf thingy. Many packages have some
> sql_connection fields which we could also configure using
> dbconfig-common. I need input and advices on this, to know what I
> should
> be working on.
> 
> I haven't worked at all yet on horizon, this is to be done as well.
> 
> So, all together, I'd say that Folsom for Debian is getting into
> shape!
> Please help me to do tests with what we have already.
> 
> Cheers,
> 
> Thomas Goirand (zigo)
>