[Pkg-xen-devel] debian-xen git workflow

[Hans van Kranenburg]

On 05/14/2018 04:52 PM, Ian Jackson wrote:
> Hans van Kranenburg writes ("debian-xen git workflow"):
>> I'd appreciate if you just would let me do this for a while and see what
>> track record we achieve while rolling forward.
> 
> OK.

Thanks for the feedback.

> I have read everything you wrote.  Thanks for the writeup.  I will
> only reply to the bits I had any kind of comment on:

Doing the same now.

>>  * Having the upstream source attached to the packaging is not a good
>> match for the parallel-development style repo that Xen upstream has.
>> There is no path forward from e.g 4.10-stable to 4.11. Merge commits of
>> upstream source make wip-rebasing, branching and merging of packaging
>> changes between versions very awkward, noisy and highly meaningless.
> 
> I have a new tool (`git-debrebase') in development which may make some
> of this easier, but I don't get the impression that you want to be an
> early adopter of novel git tools, so think it would be best for you to
> avoid it until it's more mature.

For sure I'm interested in learning more about this, and if I can,
contribute some testing time and ideas to help.

One of the things I really like about the dgit approach is that you work
towards getting rid of manual actions needed to manage debian/patches. I
think we agree that just using git is less of a headache than vim and
quilt if you want to make custom changes to upstream.

However, the downside of this approach is that it instantly gets less
obvious for someone browsing the history to understand which git commits
are part of the real upstream code or were cherry-picked by a
maintainer, because they look exactly the same. This works when you're a
sole maintainer and have this information in your head, or you can
reverse engineer this information by looking at debian/patches again,
but meh... :|

And, updating to a new upstream version means that you can end up with
conflicts caused by changes to upstream added by yourself...

In the case of Xen, I've been looking at and thinking about the current
patches a bit.

* Most of the patches are related to the abiname/prefix thing, and we
already know that we'd like to find a solution for this which can be
upstreamed. I'm not an expert in this area, but I will try to help in
whatever way I can when it's going to happen. Also: [0]
* There might be some cherry-picked patches, like including the xen-diag
tool for 4.8 now (#880554), but those kind of patches are not the
painful ones to add manually to debian/patches. Just use the right git
format patch command and boom!, it's done.
* And, I'd really like to support a "fewer patches is better" culture,
trying to build a good relationship with upstream, so that instead of
piling up patches, there can be some discussion back and forth about
what issues users run into and how they can be solved upstream. Adding
patches to the debian packaging should be a last resort.

>> It was only in March 2018 when I learned about the dgit system and
>> discovered the part of the packaging for the current Stretch release.
>> However, after playing around for a few hours with dgit, I wasn't able
>> to construct a local repository which made much sense. What I ended up
>> with was multiple initial commits and unrelated branches based without
>> finding out how to combine them into a meaningful history.
> 
> The stretch branches are not in a very convenient state for anyone but
> the most hardcore git expert.  My apologies for that.

Well, as you told me earlier, you kind of adopted the packaging because
it got abandoned, and at least made sure there were new security updates
for users. Yay for that!

If I was in the same position and also developing new tools for package
management at the same time, I think I would also have gone the way of
trying them on whatever I was working on. :)

>> Let useful discussions continue, but aside from that, I would really
>> like to end up with an upload of Xen 4.10 to Debian Unstable soon.
> 
> Yes.

\o/

Can you share your thoughts about what work we need to do to reach this
goal?

I'm currently already running the Xen 4.10 packages at work in a
pre-production environment. Except for some serious major breakage [1]
it just works (tm) as long as we don't try to use live migrate.

One thing I noticed is that there are an awful lot of warnings and
interesting messages spewed out during the build, also related to
debian-specific things. And, the packages I end up also report a quite
interesting list of lintian errors and warnings.

I think the lintian report is at least something that we should have a
look at. I can run this on my most recent packages and compose a message
to the list in which I try to triage them. Or, I could just split them
out and create gitlab issues for every one of them, in which anyone can
add a thought about how important it is and how it could be solved?

What do you think?

Hans

[0] I have been wondering... and I haven't tried to test this myself
yet... This hypervisor/utils ABI which the debian packaging relies on...
Is there some upstream test in the test suite to make sure this does not
change during a 4.x release? I can imagine that making huge changes like
the spectre/meltdown stuff might accidentally change this. How is this
managed?

[1] At Mendix, we've ran into a very nasty live migration problem,
leading to corruption in the dom0 network stack (or worse things looking
like random memory corruption causing spectacular crashes and even disk
curruption in the domU) and haven't been able to trace down the exact
cause yet. However, we can reproduce the same issue with Xen 4.8 from
Debian Stable and are really puzzled about what's so different about our
setup that causes this.

Since we can reproduce using the current stretch xen 4.8 and kernel in
dom0, I don't think this should be a total blocker for 4.10 to enter
unstable, because all other users of it are probably not hitting this
problem? (insert puzzled face icon here)