[Pkg-xen-devel] debian-xen git workflow

[Hans van Kranenburg]

Hi Ian, team,

    In the week of Feb 23 (time flies!:), we had a very productive day
\o/ working on the stretch security/stable package and some irc
discussions in oftc #debian-xen which ended with an open end.

Ian, you were a bit surprised that 1) my debian-xen repo started with an
import of a 4.8 package 2) it was not a git-pbuilder style repo with
upstream source included, and asked me to at least make a conscious
decision about why not doing those two things instead.

To start with the conclusion: I indeed want to continue in the same way
as the current repository on salsa is set up, since I believe it fits
the type of package and work (and workflow for contributors) quite well.

Summarizing that's:
 * packaging-only repo
 * debian changelog version is an unambiguous reference to an upstream
version/commit
 * generated files (4.x version dependent) are in the git ignore
 * branching done like explained at
https://salsa.debian.org/xen-team/debian-xen/wikis/home

I'd appreciate if you just would let me do this for a while and see what
track record we achieve while rolling forward.

The rest of this wall of text is a collection of thoughts and things
that happened in the last months. Feel free to read or ignore (but not
ask questions which are answered here already) or let me know how I
could have provided the same information using less words.

-------- >8 --------

For Xen packaging, there are two different types of work running in
parallel:
 * Following the latest greatest releases and putting them in Debian
unstable (yes, there's actually an audience for that, and for
stable-backports)
 * Provide fixes for security and stability issues for Debian stable.

Also, for Xen packaging, there are two peculiarities which make it look
a bit different from many other packages in Debian:
 * Instead of simple a linear development model, upstream supports
multiple releases in parallel (e.g. 4.8, 4.9, 4.10). There is no path
forward from a stable branch to the next release (e.g 4.10-stable ->
4.11) in upstream version control.
 * There are quite some binary packages, and a bunch of them have a
version number in them, which changes when moving to a new 4.x release.

    In Nov/Dec 2017, the first thing I tried was searching for an
existing packaging git repo.

I could find an old repository on alioth, which had some packaging from
2014 and 2015, but nothing recent. Since the way of working in there was
very different again (git-dpm I guess), and about 75% of the full commit
history is the same set of patches over and over and over and over and
over again, and since I could find only exacty one commit in there that
has a bit more explanation than a single meaningless line of ("Update
XYZ") which describes why the change is done and what it solves
(092785a0d8), I respectfully ignored this and later archived it at
https://salsa.debian.org/xen-team/xen-2014-2015

    Since I couldn't find anything else, I assumed the current packages
were just "based on changing the previous version" and then uploading
that. Ubuntu already had a Xen 4.9 package, and based on that work, the
first thing I tried was to create a git-pbuilder style repo with
upstream + debian packaging.

But, I quickly ran into a set of problems and inconveniences:
 * git-pbuilder can't handle the git-attributes replacement magic when
creating an orig tar, so it always ends up with a failing build because
of unexpected upstream changes. I couldn't figure out a way to work
around this.
 * Having the upstream source attached to the packaging is not a good
match for the parallel-development style repo that Xen upstream has.
There is no path forward from e.g 4.10-stable to 4.11. Merge commits of
upstream source make wip-rebasing, branching and merging of packaging
changes between versions very awkward, noisy and highly meaningless.
 * The ubuntu package obviously had been constructed by modifying
generated content (all the generated files and files with the 4.x
version in their name) in place, and having these files around meant I
had to rename them from 4.9 to 4.10, resulting in noisy error-prone
non-optimal git commits with no default history for the new files.

    At that moment I discovered these version-dependent files could be
generated, and when taking a closer look, I also discovered that the
packaging looked much alike the packaging repo of the debian linux
kernel team. Since I've been following and using their packaging for
quite a bunch of years, I was pleasantly surprised about this. BOOM!

What I did directly after this is weeding out all generated content, and
that's where the repo started.

I documented a few things in messages to the team mailing list:

https://alioth-lists.debian.net/pipermail/pkg-xen-devel/2017-December/007118.html
https://alioth-lists.debian.net/pipermail/pkg-xen-devel/2018-January/007153.html

Those messages got zero feedback from anyone(!!), so I just continued
working in that way.

It was only in March 2018 when I learned about the dgit system and
discovered the part of the packaging for the current Stretch release.
However, after playing around for a few hours with dgit, I wasn't able
to construct a local repository which made much sense. What I ended up
with was multiple initial commits and unrelated branches based without
finding out how to combine them into a meaningful history.

There's undoubtedly a lot more to say, but this message is already too
long :)

Let useful discussions continue, but aside from that, I would really
like to end up with an upload of Xen 4.10 to Debian Unstable soon.

Hans