[PKG-Openstack-devel] Improving the infra CI/CD for Deb packaging

[Thomas Goirand]

On 02/22/2017 09:42 PM, Ondrej Novy wrote:
> Hi,
> 
> 2017-02-22 18:37 GMT+01:00 Thomas Goirand <zigo at debian.org
> <mailto:zigo at debian.org>>:
> 
>     In which way does this differ from what we had on Alioth + Jenkins build
>     server? (This is retorical, let me answer) Well, before, we could push a
>     merge, but then there was a chance the package couldn't build. Now, we
>     make sure than 100% of the time, the package *can* build. So IMO it's a
>     very good thing that now, the merge commit must contain fixes so the
>     package continues to build. On top of this, we have a consistent git
>     history, were we always can build.
> 
> 
> On alioth I can do
> many commits and push "when it's ready" and when it can be build. This
> makes git history much cleaner.

It is my view that it makes the git history a lot more dirty.

> It needs to be buildable after push, not
> after commit.

My view is the opposite one. I believe it is a very good thing that each
and every commit can be built.

> Now I need to make all changes in one commit. Git history
> is completely useless/unreadable now.

useless/unreadable is probably a bit strong. To me, a debian/changelog,
debian/control and a few patch fixes are still very easy to read in a
single commit. I guess YMMV.

> BTW: Same flow are used in github+travis. Merge request (which can be
> more than one commit) needs to be buildable, not all single commits.

Maybe if you think that this workflow is wrong, you should discuss this
problem in a more broader scope, with all the rest of the OpenStack
project. All over on all of OpenStack, we insist that every commit
should pass the tests. I don't see why packaging should be an exception,
leading me to believe that probably, you're raising the point that the
whole of OpenStack gerrit workflow is wrong.

>     There's 2 options for us to use: --download-from-jessie-backports (which
>     I used for a few packages which we couldn't build), and -d
>     jessie-backports. If you and everyone else believe it's the best option
>     for us, then I don't have any objection to use extensively the -d
>     jessie-backports (which I prefer over --download-from-jessie-backports
>     so we check if the package can be built). The reason why I think it's a
>     good idea, is because this way, we take advantage of the maintenance
>     work done within Debian directly, which is a very good thing.
> 
> 
> I mean something different. I would use jessie-backports official
> backports (in /etc/apt/sources.list) during our build and only if
> package is not in official j-bpo repo, add them to auto-backports. And
> only for short time, because we should upload them to j-bpo.
> 
> So simple flow:
> * build our packages using official j + j-bpo repos
> * when i need newer/completly new b-d which is not in j-bpo, add it to
> auto-backports
> * upload it to j-bpo
> * remove from auto-backports
> 
> So keep auto-backports as small as possible. Example: Why we should
> build debhelper for j-bpo ourself if there is working+tested version in
> j-bpo already? We are uploading final packages to j-bpo repo, so it MUST
> work with other j-bpo packages. And if we don't want wait for dak/other
> maintainers, use auto-backports.

The idea was to have a self-contained full set of backports, so that we
can work on it fully, including tempest functional tests. Then we can
upload *at once* all of the new OpenStack release, once we know it
works, and without breaking our users. Otherwise, you're backporting
things for the current in development release to the official
stable-backports, which may break what is there already (ie: the
previously already released stable version of OpenStack). There are very
important incompatibilities and API breakage between versions which we
need to deal with (think SQLA transition, etc.).

With what you're proposing, there's also the issue that we may not be
able to provide one repository per OpenStack release (ie: there's only a
single stable-backports repository in Debian, not multiple), while
currently, we do have one repository per release.

This may change when we have the Debian bikesheds, but we don't have
them yet.

Cheers,

Thomas Goirand (zigo)