[PKG-Openstack-devel] Migration of packaging development workflow to OpenStack project

Sat Sep 3 00:15:28 UTC 2016

On 09/01/2016 04:44 PM, James Page wrote:
> Hi Team
> 
> As you may (or may not) know, Thomas has been working on enabling
> package build and publication processes in the OpenStack project
> infrastructure

No. This isn't about publication. The intend *has never* and *never will
be* to have OpenStack infra as the place to publish packages. The only
reason why there is a Debian repository is because we have
interconnected build-depenencies. But The intend really is to continue
to publish within Debian.

> with the intent of migrating *all* packaging development
> workflow under pkg-openstack to a git/gerrit based workflow.

Yes, and for the good. (almost?) Everyone in Debian understand that just
having a write access control for Git is not enough. Gerrit is so much
better in many ways.

> This means that day-to-day packaging changes, currently done directly in
> git on alioth, will all be done using git/gerrit, inline with other
> OpenStack projects.

Yes, and that makes a lot of sense. Projects within OpenStack will also
be able to gate on packaging, and the packaging will be able to have
gating based on puppet-openstack, for example. Projects like Fuel could
also re-use the packages, and the packages could use Fuel as tests too.
I'd love to see JuJu doing the same.

> I'm concerned that, to me at least, it feels like this plan has never
> been formally ratified by the members of pkg-openstack in Debian.

I see absolutely no reason why we would keep using the old Git setup on
Alioth. What we're slowly shaping within OpenStack infra is so much
better. Alioth is clucky, sometimes very slow (think load average going
up to 30 regularly every day) and even down. It even happened that
Alioth was down for more than 10 days in a raw. I don't want to
experience this again. I also don't even feel safe with the data stored
on Alioth: when it was down, it was because the RAID array failed. It's
not even a distributed storage.

Having only the possibility to grant write access to *everything* or
*nothing* in the pkg-openstack team on Alioth is really bad in many
ways. Say a new contributor shows up that we don't know, then I wouldn't
trust him, and then wouldn't provide write access. Then he will not
contribute (ie: he's feeling not welcome). Say I'm giving him write
access, then that's a big risk (he could rm -rf /git/openstack/* on
Alioth). Also, multiple times, we had potential contributors that were
afraid of breaking something, and therefore didn't dare to commit to the
git. Having a full CI (with in the short term a full Tempest test suite)
would make them feel more confident.

Another thing is that current OpenStack contributors are actually
*expecting* to use a Gerrit with a full set of checks and gating. We're
here opening up the door for all of them, and many have packaging skills
that we're not able to enjoy.

Also, a lot of packages are maintained with Git, outside of Alioth.
Alioth isn't mandatory at all. Even though, I would like to get any
merged CR pushed to Alioth, just because that's possible. I'm not sure
yet how. Though that would help, because on Alioth, there's git receive
hooks that will trigger builds for arm64 on the Jenkins provided by
Linaro. So that will also be very good to have.

As for the CI part, what happened before is: someone was making a commit
in the Git, checking or not the result in the Jenkins that I have setup.
Sometimes. It could break things hard, and nobody would notice.
Considering how complex all of OpenStack is, using a real CI with gating
is a huge improvement. Having an always working package repository is
also a goal which we can't have with the current setup.

Now, if what is bothering you is that the packaging is hosted by
OpenStack infra, then I would very much welcome producing the same kind
of Gerrit / Zuul / Nodepool setup that OpenStack infra has for Debian as
a whole. I've been thinking about this for more than a year already. And
I discussed it during my talk in Debconf. Some other DDs are also
interested by the idea.

The reply I had from the DSA team was that they needed a proof of
concept first. Which is exactly what I'm doing here. Once we have all up
and running, and even after Newton is released this way, we will have a
very good show case for Debian. Hopefully, the DSA team will then agree
to allocate some hardware, and we will be able to have such a setup for
the whole Debian archive, interfaced with something like Gerrit. At
least, that's my plan for the longterm. Hopefully, I'll have enough time
to implement it.

> I've been fairly vocal on the openstack-dev ML that I don't think this
> is the right direction to go in (see [0]), as I feel it moves packaging
> and distribution work outside of either Debian or Ubuntu, resulting in
> more hops to get work done and uploaded to either distribution. I still
> stand by this sentiment, and feel that this change is not been driven by
> the team as a whole.

The only person who's not happy about the move is yourself, and that
*after* you first approved the idea. Everyone else is every enthusiastic
about the move.

So far, you haven't given any reason but "it feels like it's done
outside of Debian/Ubuntu", which doesn't feel right to me. The feeling I
have is that this move doesn't fit in Canonical's agenda. Of course I
have no way to know, and this is just a double-guess gratuitous
sentence. But I've been thinking about it over and over again, and I
don't see any other reason why you wouldn't want to do it. If I'm wrong
here, feel free to say it, and do not take it the wrong way: it's
perfectly understandable that companies have a (commercial) agenda.

But if I am right, then I would like to kindly ask you to reconsider.
Ubuntu will always be the winner distro in OpenStack in many, many ways.
I've tried to push for Debian, and even within my own company. It's a
disaster: from sales to PM, they all reject the idea, on the grown that
they believe customers would run away because of the lack of commercial
support. Whatever we do to actually create the packages for both Debian
and Ubuntu, this will never change. There is absolutely nothing that
Ubuntu would loose in moving to Gerrit. The only way you would loose
would be if Ubuntu was not supported there. And because you don't seem
to care, and I have no time (yet) to take care of it, that's
unfortunately what's currently happening. Though it's far from too late.

Also, the Debian repositories hosted in OpenStack infra will only be for
development purposes. They are in no way to be considered for
production. I will make a big emphasis on it in the docs that they can
be used for testing, but that's the only purpose. Like you, I believe
packages belong to the respective distributions.

> Thoughts?  Is this something that all team members feel is a change for
> the better and is something they can stand behind?  I appreciate that
> adding things like code review and CI are all helpful to development,
> but I don't think that doing that all outside of Debian is the right
> approach.
> 
> Cheers
> 
> James
> 
> [0] http://lists.openstack.org/pipermail/openstack-dev/2015-June/thread.html#65298

We now have a *real* CI in place. We (or at least *I*) will continue to
upload to Debian. Adding a real CI with gating will add a lot of good
things. Just today, I had an external contribution:

https://review.openstack.org/364994

This isn't much (ie: just a watch file update), but it shows a lot. That
contributor never used Gerrit before, it was a first for him. Within 5
minutes, he was able to do all the steps to be in. This clearly shows
it's very easy to do. I don't think setting-up an account on Alioth is
easier and/or faster. In fact, it is both slower (you have to wait for
some cron job, and wait for the team approval, otherwise you don't have
write access), and Fusion-forge GUI is sometimes very confusing.

I do expect many other contributors to show up. For example, the Melanox
guys that want to maintain their networking-mlnx driver.

BTW, I don't like the CLA we're having in OpenStack. I really hate it.
But frankly, if that it touches *also* the packaging too isn't a big
issue, since everything else already has this CLA. If you want to fight
getting the OpenStack CLA out of the way, I'd very much welcome you to
do so, I believe a vast majority of OpenStack contributors would like it
see it gone as well. BTW, if someone wants to contribute, but doesn't
want to sign the CLA, I'd happily do that commit for that person.

Hoping my answer is driving you to the right direction of reconsidering
your last year back-up,
Cheers,

Thomas Goirand (zigo)