[Pkg-zsh-devel] pkg-zsh.git workflow, Second Draft

[Frank Terbeck]

* Branches

  This is a quick overview of the branches that are being used for
  debian's zsh package maintenance. This part does not go into much
  detail on how the workflow works. For details about that, see the
  `Workflow' part below.


** upstream

   The upstream sources from <git://zsh.git.sf.net/gitroot/zsh/zsh>.


** debian

   The debian changes for the debian `zsh' package.


** others

   These branches are rather optional and not required for basic
   maintenance for of the `zsh' package.


*** master

    This is the old repository's main branch. Kept for historical
    reasons.


*** debian-beta

    This could basically do *exactly* the same thing as `debian' does,
    but just merge more often. Such a branch *could* then be used to
    keep the packaging information for `zsh-beta'.


* Workflow

  This is a diagram, that outlines the proposed workflow. Basically,
  the `debian' branch walks alongside the `upstream' branch and
  upstream changes get merged into `debian' at defined points (for the
  non-beta zsh package, that should be upstream commit tags).


            * debian/4.3.11-4
            |
            |   * debian/4.3.11-5
            |   |
            |   |       * debian/4.3.12-1
            |   |       |
            |   |       |   * debian/4.3.12-2
            |   |       |   |
            |   |       |   |           * debian/4.3.13-1
            |   |       |   |           |
            |   |       |   |           | * debian/4.3.13-2
            |   |       |   |           | |
            |   |       |   |           | |   * debian/4.3.13-3
            |   |       |   |           | |   |
            |   |       |   |           | |   |
          a-b-c-d---e-f-g-h-i-------j-k-l-m-n-o-p   `debian'
         /         /               /
      A-B-C-D-E-F-G-H-I-J-K-L-M-N-O-P-Q             `upstream'
        |         |               |
        |         |               |
        |         |               * zsh-4.3.13
        |         * zsh-4.3.12
        |
        * zsh-4.3.11


** Working on the package


*** Changes

    Every change to the source (non-debian-dir) should be done by
    adding quilt patches. Permanent patches (patches that would be
    kept in 4.3.12-1) should be named specially ("deb_*" comes to
    mind).

    To keep order for cases in which a vast number of patches are
    required, any quilt patch needs to be numbered according to the
    order in which "quilt series" would list them. The format looks
    like this:

       - deb_NNNN_<name>.diff
       - NNNN_<name>.diff

    Where `N' is a digit from 0 to 9. Counting starts at `0000'.

    Every patch should have a short annotation above the actual
    unified diff content, outlining the status of the patch. It is
    especially important to know whether a change is already applied
    upstream because if so, the patch in question must be dropped
    before merging a new upstream release into the `debian' branch.


**** Dealing with quilt's .pc directory

     When quilt works, it keeps track of what it does in a directory
     by the name `.pc'. This directory will show up in the output of
     "git status" etc. It should *never* *ever* by committed to the
     git repository at any point.

     We cannot add the directory to `.gitignore' because we would
     change the zsh source directly instead of via `debian/patches'.

     To deal with the directory, there is another git-facility, that
     we can use for fun and profit.

     % echo .pc/ >> .git/info/exclude

     Now git will ignore quilt's directory for this
     repository. Unfortunately, this has to be done for each
     checkout. Luckily, this only has an impact for people who want to
     work on `pkg-zsh'. Everyone else can savely ignore the directory.


**** Example of adding a fix via a quilt patch

     Let's say, there is an issue `#12345' which can be fixed by
     patching `Functions/Misc/colors'. Here is how you could add that
     patch (assuming clean git working directory):

     ## First, add all exising patches, so the new one is added on top.
     % quilt push -a

     ## Add the new patch (say, the topmost patch is 0002-foo.diff).
     % quilt new 0003-fix-colors.diff

     ## Tell quilt which files are going to be changed.
     % quilt add Functions/Misc/colors

     ## Import the fix (manually by editor or by patching).
     % vi Functions/Misc/colors

     ## Pop all patches again to clean up the upstream source.
     % quilt pop -a

     ## Commit the new patch and the changes `series' file to git.
     % git add debian/patches/0003-fix-colors.diff
     % git add debian/patches/series
     % git commit -a -m'Fixing foo in colors function (Closes: #12345)'

     That's all.


**** Keeping the local repository clean

     Before making changes of any kind, it should be made sure that
     the local repository you are working on is in a clean state. To
     clean up the local repository do this:

     % git clean -xdf
     % git reset --hard

     That will make sure that any non-tracked files are removes and
     that any changes in tracked files are reverted. The latter will
     also make sure that no parts of a quilt patch-queue are still
     applied.


*** Releases

    When a change justifies the release of a new package version, the
    debian/changelog file should be updated and the resulting commit
    should be tagged debian/<zsh-version>-n+1.


*** Updating debian/changelog

    This file should *not* be updated manually. The changes should be
    inserted by running the `git-dch' tool from the `git-buildpackage'
    before a new release is about to be made.

    Changelog entries should be prefixed by a "[hashsum] " string,
    where `hashsum' is a string that represents the first eight
    characters of commit the changelog entry was generated from.

    Also, if multiple authors are involved in a changelog entry-set,
    each author should only appear once in the series with all her/his
    changes listed below her/him in chronological order.

    Here is a command line which will result in the desired changelog
    format:

    % git-dch --debian-branch=debian --id-length=8 --multimaint-merge


** Transitioning to a new upstream version

   When upstream releases a new version, we should follow these steps:


*** Removing non deb_* quilt patches

    All non deb_* patches should be removed from `debian/patches'
    directory, unless they fix an issue that was *not* addressed
    upstream and is therefore missing from upstream's code base.

    If such a change should prove to be required to be kept with the
    package permanently (e.g. because upstream refuses to apply the
    patch), the patch should eventually be renamed to match the
    "deb_*" nameing convention.


*** Merging `upstream' into `debian'

    After the `debian/patches' directory was cleaned up in the
    previous step, merging `upstream' into `debian' should generally
    lead to a working package again.

    If old patches were still around, that could lead to conflicts
    when those would be applied during the build process.

    The message for the merge commit should be set to "New upstream
    release" to allow `git-dch' to pick it up correctly later.


*** Update the autotools files for the new release

    This should be done to make sure, the build systems is always kept
    up to date. For convenience, there is a script that automates this
    process:

    % ./debian/at2quilt

    After that, the involved quilt patches are updated, but not yet
    committed to the git repository to allow for review.


*** Updating autotools files whenever input files are changing

    Sometimes, it will be necessary to backport changes to autotools
    input files (such as `configure.ac'). As with any other change,
    such a task should be done by adding a patch to the packages quilt
    queue.

    Obviously, if an input file changes, the autotools related patches
    for the package need to be updated. Again, the `at2quilt' script
    from the `debian/' subdirectory helps. The only difference to the
    way it was used before is that you need to tell the script which
    patches in the patch-queue are relevant for autotools. Say there
    is a patch `0017-autotools-fix-gnu-hurd.diff' which alters
    `configure.ac', here is what would need to be done:

    % ./debian/at2quilt debian/patches/0017-autotools-fix-gnu-hurd.diff

    If there is more than one patch `at2quilt' need to consider, list
    all of them in the order in which `quilt' would apply them (if in
    doubt, ask "quilt series").


*** Fix outstanding bug

    If *any* outstanding bugs are known, they should be fixed before
    releasing a new package. Obviously, if any of the known bugs are
    very hard to fix and the issue is not serious in nature, releasing
    the package with the issue may be more important.

    Again, all changes to non `debian/*' files should be done via
    quilt patches.


*** Verify that the package builds

    % git reset --hard
    % git clean -xdf
    % QUILT_PATCHES=debian/patches
    % export QUILT_PATCHES
    % quilt push -a
    % ./configure
    % make all test


*** Tag debian/<new-zsh-version>-1

    After fixes for all serious and trivially fixable issues have been
    added and it has been verified that the package builds, `git-dch'
    should be used to update `debian/changelog' and the resulting
    commit should be tagged as `debian/<new-zsh-version>-1'.


** Generating packages

*** gitpkg

    `gitpkg' is a simple tool to help generating packages from debian
    packages which are maintained in the git version control system. It
    works quite well in this workflow. In fact, it works out of the box:

    % gitpkg debian/4.3.12-5 zsh-4.3.12

    The first parameter (debian/4.3.12-5) is the debian tag which
    points at the debian package version you want to build. The second
    parameter is the tag of the upstream version of the corresponding
    upstream release (zsh-4.3.12).

    Per default, `gitpkg' generates it's output in
    `../deb-packages'. This location is configurable.

    Below directories for individual packages are created and in those,
    data for individual package versions are created. For the above
    example, this would look like this:

    ../deb-packages/zsh/
    ../deb-packages/zsh/zsh-4.3.12/
    ../deb-packages/zsh/zsh_4.3.12.orig.tar.gz
    ../deb-packages/zsh/zsh_4.3.12-5.debian.tar.gz
    ../deb-packages/zsh/zsh_4.3.12-5.dsc

    You may now change to `../deb-packages/zsh/zsh-4.3.12/' and build
    binary package using `dpkg-buildpackage', `debuild' or the like.

    `gitpkg' is available from:

    <http://git.debian.org/?p=users/ron/gitpkg.git>

*** git-buildpackage

    Alternatively, `git-buildpackage' also provides ways of building
    packages from our packaging codebase. And since we are using the
    `git-dch' tool from this utility suite anyway, the tool should be
    available already.

    `git-buildpackage' allows building the package from within the
    package repository:

    % git-buildpackage --debian-branch=debian

    Make sure that the local repository is cleaned up after doing this
    before working on the package again, to avoid accidentially
    committing anything. See "Cleaning up the local repository" above
    for details.

    <https://honk.sigxcpu.org/piki/projects/git-buildpackage/>


** Git repository setup

   Getting the basic pkg-zsh git repository is quite easy. If you want
   a read only clone, use this:

   % git clone git://git.debian.org/collab-maint/zsh.git pkg-zsh

   If you are reading this, though, you probably want write access. To
   get a thusly cloned repository, first get an alioth login and
   upload an ssh-public key. As soon as the key made it to all
   involved machines, use this:

   % git clone ssh://<user>@git.debian.org/git/collab-maint/zsh.git pkg-zsh

   Where `<user>' is your alioth login. (Note, that this may be
   something with a `-guest' suffix, in case you're not a debian
   developer.)

*** Branches

    Like described earlier, pkg-zsh development involves two branches;
    `debian' and `upstream'. The former is checked out by default for
    freshly cloned repositories. To get a local version of the
    `upstream' branch, use:

    % git checkout -b upstream origin/upstream

    This is useful to update the remote upstream branch with ongoing
    development from the zsh project.

*** Remotes

    There is one remote repository with direct interest for pkg-zsh,
    and that is the zsh project's git repository. Currently, this is
    only a mirror of the project's cvs repository. But it is updated
    every ten minutes by one of zsh's developers. (Also note, that
    there has been a brief discussion about whether git may become the
    official VCS for git after a bigger future release.)

    In order to have zsh's ongoing development available from within
    your pkg-zsh repository, do this:

    % git remote add zsh.git git://zsh.git.sf.net/gitroot/zsh/zsh -t master
    % git fetch zsh.git

*** Merging and pushing upstream changes

    To get updates back into origin/upstream, do this:

    ## Get the latest updates.
    % git fetch zsh.git
    ## Switch to the local `upstream' branch for integration.
    % git checkout upstream
    ## Merge upstream's changes (*).
    % git merge zsh.git/master
    ## Push the code into pkg-zsh's central repository.
    % git push origin
    ## Make sure the central repository also has all tags.
    % git push --tags origin

    (*) This step should *always* result in a fast-forward merge. If
        it does not, something went terribly wrong. Investigate and
        fix the situation *before* pushing to origin.