git based autoforwardporter.

[peter green]

On 19/01/17 07:00, Robie Basak wrote:
> Hi Peter,
>
> On Thu, Jan 19, 2017 at 02:16:27AM +0000, peter green wrote:
>> Some time ago I put together what I call the "autoforwardporter". The
>> aim of this is to take downstream changes and apply them on top of new
>> debian uploads.
> Can you expand on this?

Here is my attempt at describing how the system works.

Throughout the process "local" versions are distinguished from "debian" versions by looking for tags (in our case +rpi and +rvt) in the version number. The +rpi tag is used where we have local modifications, the +rvt tag is used where we previously had local modifications, have since removed them but for version numbering reasons we can't just use the Debian version directly.

The tag is normally located at the end of the version number, but occasionally it is located earlier in the version number for version ordering reasons.

The system relies on a whitelist of source package names that it should consider processing (some local changes are short term hacks that should probbablly not be carried forward)

When the system tries to make a forwardport it creates a logfile. It will only try to make a forwardport with a given version number once unless the logfile is manually removed.

Stage1: local repo on autoforwardporter box

We use reprepro to maintain local copies of both our own suites and the corresponding suite from Debian on the box that runs the autoforwardporter. On our own side we have a "main" suite and a "staging suite" (similar in concept at least to ubuntu's "proposed"). For stretch these suites are called "stretch" (our main suite) "stretch-staging" (our staging suite) and "stretch-deb" (debian stretch)

Stage1: import.

I am using dgit import-dsc for this with a custom driver script that takes a directory full of dscs, imports and tags them. If the package has a "dgit:" header we just let dgit import it with whatever history is in the git repo.

If the dsc does not use dgit then the import script figures out the history shape it should use from debian/changelog. For locally modified packages the immediate parent must be present or the import fails. For non locally modified packages we are far more lenient allowing version skipping and even orphan histories.

Stage2: decide whether to forward port and figure out version numbers

For each package in the whitelist we check the version numbers. Figure out whether there are local changes in the staging or main suites that should be brought forward to a new version from Debian and if-so what the new version number should be. The code is quite tricky because it has to handle some weird corner cases (for example if we have a version 1.0+rpi1 and debian introduces 1.0+deb9u1 then the new version needs to be numbered 1.0+rpi1+deb9u1 since 1.0+deb9u1+rpi1 < 1.0+rpi1 )

Stage3: potential psuedomerge.

First we grab the changelog from our old local version and look for the first entry in the changelog that does not have one of our local tags on it. This tells us the version from Debian that it was based on.

In most cases the old version from Debian will be an ancestor of the new version from Debian but sometimes it isn't. In this case we do a psuedomerge to create a commit that has the tree from the new debian version and has the old debian version as an ancestor. This avoids trying to merge changes from the old Debian version into the new Debian version when we do the main merge.

Stage 4: main merge

This bit is a simple git merge. The "ours" side of the merge is our old version and the "theirs side" is the new version from Debian with the psudomerge if needed.

We use the "diff3" conflictstyle so our later fixup tools can get the context they need.

Stage 5: post merge fixup

It is necessary to fix up debian/changelog so it doesn't contain conflict markers and so it has the right versoin number. Exactly how this is done is a matter of taste, personally I squash together all the "local" changes into a single changelog entry on top of the Debian changelog. Ubuntu seem to be a fan of mergechangelogs (personally I find the result of mergechangelogs to be a confusing mess).

It is also frequently needed to fix up debian/patches/series . The script tries to retain order with local patches sorted after debian ones.

I expect further fixup will be needed, in particular I am expecting to have to add some further quilt-related hackery. I also want to take a look at debian/control merge failures, at least the simple cases.

Stage 6: commit results to working branch for archival

We commit the results to a working branch and push them to a suitable (local) repo to preserve them. If there were merge failures that could not be auto-resolved we commit the files with conflict markers still in them and put a list at the top of debian/changelog. This allows a human to clone the working branch, finish the fixup manually and amend the merge commit.

Stage 7: build

Assuming we were able to automatically resolve the merge conflicts we build a source package with dgit and then call sbuild to build that source package.

Assuming it built succesfully we copy it to our "output" directory.

Stage 8: dgit push

Assuming the source package builds successfully we do a dgit push to add the dgit: tag to the dsc and push the git stuff to our dgit server. We suppress the actual uploading for the moment as I like to make some final manual checks before uploading.

If the dgit push succeeds we re-copy the package with the dgit: header added to our "output" directory.