New build system (dhmk based) for Qt/KDE packages (qt-kde-team/2/*)

[Modestas Vainius]

Hello,

In my opinion, it is time to move away from pkg-kde-tools/qt-kde-team/1/* 
which is based on CDBS. As most of you know, I wholeheartedly dislike CDBS for 
a couple reasons. In short:

1) The way it sequences debhelper commands (dh_command -ppackage for each 
command+package, i.e. command_count * package_count forks instead of 
command_count forks) prolongs build time significantly for huge source 
packages. As you know we do maintain such packages, actually quite a few of 
them. What's more, build logs are unnecessary cluttered due to this.

2) It's insanely complex. And I'm not referring to its internals (which are 
mind blowing by themselves) but to its "user" interface:

2a) first of all, it's controlled by a bunch of variables which names are not 
always consistent (e.g. DEB_INSTALL_DOCS_ALL vs. DH_MAKESHLIBS_ARGS_ALL 
despite both passing arguments to the respective debhelper programs);

2b) secondly, each time you want to hook something up to it you have to read 
cdbs internals and copy&paste targets. Hook names are not obvious but even 
then you can't hook to all places if you really need to.

3) So finally, I would rather spend my time doing actual work (packaging) than 
reading cdbs internals or waiting until it wastes time while I'm doing test 
builds.

Now there is dh(1) sequencer. It's pretty nice because:

1) User interface is pretty straightforward. In particular override_dh_command 
targets are nice and `dh $@ --extra-option1 --extra-option2` command line 
interface for options is fine too.

2) Sequencer is written in perl. So are dh addons which push you to write a 
new debhelper program (also perl based) when you want to extend functionally 
for general use (i.e. outside debian/rules). As a result, "templating" is 
pretty limited to perl which is not optimal due to many reasons.

Therefore I wrote dhmk which qt-kde-team/2/* is based upon (since pkg-kde-
tools 0.10). dhmk.* is a basic sequencer that is driven by make and can be 
*templated* with make snippets. debian-qt-kde.mk and friends hook a couple of 
custom stuff to general sequences.

dhmk design goals:

1) dhmk is make-based. Unlike dh(1), it does not track progress per command, 
but per action (configure, build, install etc.). It has sense of stamped 
(configure, build) and dynamic (install, binary) targets like traditional 
debian/rules would.

2) dhmk is compatible with dh as much as possible, e.g. it runs the same 
debhelper commands dh(1) would (by default) and it's possible to reuse dh(1) 
addons to extend the sequencer. Like with dh(1), source code is built by 
dh_auto_* commands. Sune expressed some criticism towards them, but I 
personally don't see why we should not use official, well supported and widely 
tested debhelper helpers given that they already do some important work behind 
the scenes (parallel, verbose output etc.).

3) There is the dhmk.pl part which is responsible for generating a dynamic 
debian/dhmk_rules.mk file. It's used for:

	a) parsing commands file (static) and converting it to a bunch of variable 
definitions which dhmk.mk would understand;
	b) executing dh addons (specified using --with option to dh variable) and 
integrating their sequence changes to debian/dhmk_rules.mk file;
    c) parsing the rest of options in the dh variable (i.e. like dh(1) would)
and passing them to all dh commands being run (and also exporting them in 
DHMK_OPTIONS environment variables for use in overrides);
	d) detecting which debhelper commands have overrides in debian/rules and 
specifying that info in debian/dhmk_rules.mk for later use by dhmk.mk.

Now yeah, dhmk.pl is still perl. But if there is ever a need to drop perl, 
debian/dhmk_rules.mk could be turned static and d) replaced with sed script 
(8de511c5 commit [1] has an example which need to be tweaked). However, 
support for b) and c) would be lost in that case. Auto-generated dhmk_rules.mk 
is a bunch of variable definitions which are not very handy to edit by hand.

4) dhmk.mk is a core sequencer and it's 100% extensible with make snippets. 
Snippets may hook to run the code before/after *each* debhelper command or 
after/before each standard target. There is no need to know perl to extend 
general functionality. See library-packages.mk, policy.mk and debian-qt-kde.mk 
for examples of such make-based dhmk extensions.

5) While make snippets use hooks, debian/rules should use override_dh_command 
targets. So here dhmk is not reinventing the wheel and borrowing the best what 
dh(1) has to offer. Contrary to dh(1), dhmk does not repass extra options 
(defined in the dh variable) to the overriden dh_command's. Those are made 
available in the DHMK_OPTIONS envvar for later use if needed. Only arch/indep 
options (-i/-a) are handled implicitly.

6) dhmk.mk code itself, while pretty short, is not very straightforward (a 
bunch of advanced make stuff like $(foreach), $(if), $(filter) etc. and 
extensive use of internal variables all over the place. In order to understand 
what's going on, have a look at what variables get defined in dhmk_rules.mk. 
Unfortunately, extensive templating options make dhmk UI a bit more cluttered 
than the one of dh(1) but it's still far far less complex than cdbs (in my 
opinion anyway). Some more information is available in qt-kde-team/2/README.

Therefore, I would like to see Qt/KDE packages getting converted to dhmk [2]. 
Unless somebody has to offer something better, they are welcome to. But 
personally, I no longer see "sticking to CDBS" as an option. While qt-kde-
team/2/* is not perfect, I tried to decouple it from tight bound with perl 
while adhering to the best successful practises established by dh(1).

[1] http://git.debian.org/?p=pkg-kde/pkg-kde-tools.git;a=commit;h=8de511c5
[2] I have recently converted phonon and its backends will follow soon.

-- 
Modestas Vainius <modestas at vainius.eu>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.alioth.debian.org/pipermail/pkg-kde-talk/attachments/20110309/e01935a4/attachment.pgp>