[PKG-Openstack-devel] Improving the infra CI/CD for Deb packaging

[Thomas Goirand]

On 02/22/2017 01:23 PM, Ondrej Novy wrote:
>     >  3. it's always not possible to have one commit per change (i need to
>     >     git commit --amend after merge)
> 
>     Could you expand on this? I don't understand why there's a problem, and
>     why one would need to do a --amend after merge.
> 
> 
> ehm, https://wiki.openstack.org/wiki/DEB-packaging#Releasing_a_new_upstream_release_package
> 
> You MUST use --amend after git merge.
> 
> Cite: Note that the fix of debian/changelog and debian/control should be
> done in the same single commit as the merge commit, otherwise the
> package will fail to build.
> 
> So in 1 commit I need:
> 
>   * merge upstream
>   * change d/changelog
>   * change d/control for deps
>   * add d/patches which fixies build
> 
> This is crazy. I want to do as small commits as possible, at least one
> commit = one changelog entry.

In which way does this differ from what we had on Alioth + Jenkins build
server? (This is retorical, let me answer) Well, before, we could push a
merge, but then there was a chance the package couldn't build. Now, we
make sure than 100% of the time, the package *can* build. So IMO it's a
very good thing that now, the merge commit must contain fixes so the
package continues to build. On top of this, we have a consistent git
history, were we always can build.

>     >  4. it's hard to manage deb-auto-backports. It took me a day to fix
>     >     auto-backports just to upload new Swift version
> 
>     True. Though it has the huge advantage that it keeps backports
>     up-to-date, which was never the case with the hack system of doing it by
>     hand. Also, you never had to take care of it, because *I* was doing it
>     by hand, sometimes cheating and just ignoring build problems (shame on
>     me). It was even more painful by hand...
> 
> 
> but that's theory. I waited ~ week for fix and then i fixed it myself.

I don't remember what that was. Can you refresh my memory?

> We can't depend on single man to do critical job. We need solution which
> anyone can fix :). Current solution is hackish. Example: If you need to
> add new package for auto backports, every other package is checked for
> update and if there is update, it gets updated.

This is a very good thing, so we stay current. Meaning that if there's a
security issue fixed in Sid, we get it too. The original plan was to
rebuild all packages as a periodic job. It is my opinion that such a
periodic job should be setup, because otherwise we may miss some
security patches and bugfixes.

Yes, it's inconvenient, and sometimes even painful. Though I don't think
we have a choice here.

> Next: We should
> prefer depedency packages from official jessie backports if they are
> newer and (auto-)remove old one from our repo.

There's 2 options for us to use: --download-from-jessie-backports (which
I used for a few packages which we couldn't build), and -d
jessie-backports. If you and everyone else believe it's the best option
for us, then I don't have any objection to use extensively the -d
jessie-backports (which I prefer over --download-from-jessie-backports
so we check if the package can be built). The reason why I think it's a
good idea, is because this way, we take advantage of the maintenance
work done within Debian directly, which is a very good thing.

We could even upload 3rd party libs to official jessie-backports first,
and *then* use the -d jessie-backports if we believe it's a good
workflow, however, this would mean waiting for a dak run and mirror
sync, which could take up to a day. I'd say this last option is for the
cases were we really need specific stable-backports patch.

Please do answer here and voice your opinion.

>     >  5. it doesn't build packages for unstable - which is important,
>     because
>     >     we are uploading to unstable and instead it builds/checks Jessie
>     >     backports (which is less important)
> 
>     We could add that feature. Also, it is my view that one should attempt
>     an sbuild build on Sid before sending a CR. With the old system, we also
>     had the issue.
> 
> yep i can say anyone should do sbuild on Jessie backports before doing
> CR too. So why we have gating for Jessie bpo?

stable-backports has always been what Debian users are consuming. At
least, it was my view that it was how it should have been. Also because
Sid *does* break from times to times (think Python 3.5 transition, for
example...), so it's harder to keep that gating stable.

> Because gates are here for
> checking if people are doing correct thing. And because we are uploading
> to Sid, we must be sure it builds on sid (with deps package only from sid).

I also agree we should be checking with Sid, probably as non-voting
first, and see how it goes.

> so:
> 
> 8. add repo for Ocata

Hopefully, Alison will be able to corner someone from infra this week,
and make them create the ocata package repos.

Cheers,

Thomas Goirand (zigo)