[Pkg-voip-commits] r3907 - snapshot

Mon Aug 6 20:17:51 UTC 2007

Faidon,

On Mon, Aug 06, 2007 at 05:10:28AM +0300, Faidon Liambotis wrote:
> kilian at alioth.debian.org wrote:
> > add new asterisk/branches/experimental to experimenal build.
> Thanks for that!
> I'm not very familiar with the whole buildserver thing, any pointers
> would be appreciated.

ok, I guess a general heads will do good, so why not take this chance.

The current setup is as follows:
1.) svn.debian.org holds the package source in revision X for path P
2.) Each project targetted at Buildserver.NET has a config file. The one
    for pkg-voip is [1].
3.) snapshot-package[2] will be run with the config file [1] as
    argument. From there reading all paths it will iterate through each
    of them doing the following:
    - checkout svn path
    - read svn revision
    - compare with last known svn revision
    - if changed, package source upload for each dist in %dist
      * if distribution in changelog is one of "stable", "testing",
        "unstable", "experimental", append a ".$dist.$revision" 
      * otherwise suffix "~$dist.$revision" to the version number
      * add changelog entry with the snapshot description from the
        config
    - upload to archive.buildserver.net into the *-build queue for each
      dist
    - cleanup and continue
3b.) If you care to check manually some certain path, a third argument
     can be passed to the snapshot-package script with the path from the
     config. In that case not all paths, but only the given one will be
     subject to the above procedure.
4.) In archive.buildserver.net it's picked up by the, installed into the
    public webspace and then caught by wanna-build. 
5.) For each of the archs a buildd will pick up the source and do a
    binary compile. Only on powerpc you will have mixed arch=all and
    arch=any uploads from the same source. Sources that are only
    existing of arch=all are handled as a separate arch and can be built
    by most buildds of Buildserver.NET.
6.) Once uploaded and installed into the build dists, our own britney
    will iterate over each new package and try to install it into the
    release webspace (e.g. http://pkg-voip.buildserver.net/). For the
    output see [3].

The overall logfile flood is represented at
http://status.buildserver.net/ which works pretty much like
http://buildd.debian.org. Additionally you can check the general
overview at
http://status.buildserver.net/packages/status.php?subdist=pkg-voip
as a whole or limit it to certain sources, dists, arches.

The config behind the email address is autogenerated from the config
file, so don't try too hard putting correct email addresses in there.

What you may find is that our buildd/sbuild/wanna-build behave somewhat
slightly improved from the ones used on real Debian buildds. First, they
work from an empty chroot always. It's synced into place upon invocation
of sbuild. If that fails to gather all required source files, it'll drop
to retry-later, which means wait 6h and put back into needs-build. 

What it so far cannot do is automatically put binNMUs for all the
rdepends of an upload. Yet it's just a matter of time till this will be
added for testing.

Soon to come is the new dpkg with experimental dpkg-gensymbols causing
decent ABI/API checks to take place during compilation. I'll drop
another mail when it's added on how to operate it.

> BTW, is there any way to generate released version besides snapshots?
> Every version is a snapshot of course, but it would be nice if we got
> tags instead of trunk with a version number of 1.4.1-1~etch.1.

For this to work properly, we'd need even more dists to sort apart the
identical source name with different versions. Right now dak doesn't
have any other option to allow two different versions in an archive.
Therefore we'd yet need some way more flexible config scheme, as with 4
project, 6*2 dists per project (build and release) and even two flavours
(trunk and experimental) for two of the projects it has become somewhat
clumsy to maintain it the old fashioned way. Moreover the britney run and
even the regular jennifer/kelly/wanna-build updates have grown to a
delay that more than asks for optimization. Naturally it we'll be able
to tear down more limits by only processing changes rather than the
whole archive - and we need multithreading rather than a single task. 
Apart from that we're slowly but steadily running out of disk space on
that box.

This being said, sure we *could* do this. The easiest way is to just add
another dist and "snapshot" tags/branches into there. It may or may not
be useful to wrap a britney around it, but sure that's possible too.

And honestly, if you ask me, people have been happy with our svn
snapshots much more than with a static and dead tag that didn't work. We
may put a rollback archive elsewhere if we find a place large enough to
host it. But tags need maintenance and that's where the whole concept
hits its roughest limit.

So, from my experience, let's rather add the missing arches if available
and do the portability checks/fixes as good as we can, i.e. concentrate
on good packaging in the first place - and that will keep trunk be a
smooth and handsome snapshot for our users to install.

> Or even 1.4.1~bpo.1 and upload them to backports.org.
> 
> My key is in the keyring of backports.org and I can do that manually
> (even though not straight from the buildserver's binaries since I'm not
> in control of it and I can't just sign the binaries not matter how much
> I trust you).

Yes, but bpo is another issue we've discussed long ago. Same problem
here, it needs someone doing it. That means, even though the script is
even there to help you with the required edit, you still need to process
it manually. Of course our buildd code can produce an upload matching
your *.dsc - no problems there (adding -A will work perfectly well). But
still it's something that's not happening automatically with a new
upload - which is exactly why we said that our internal backports by far
outperform backports.org in latency. What lacks to be implemented is a
more fine grained option to implement sources.list and apt_preference
updates for certain packs to pull external build-deps from sources like
backports.org (debhelper and dpkg-dev come to mind). This is also
planned, but not yet realized.

> Backporting trunk is of course nice but I'm not sure if it's very useful
> by itself.

It has been for a not so small community in the past. And hey, we don't
force anyone to actually install these packs. They're first and foremost
a "check-whether-svn-trunk-still-compiles" test. And this on the largest
base we could put up. So if it compiles on Buildserver.NET it's fine to
go into Debian and it should build well there, too. If it doesn't (like
wengophone - i know Mario will hate me for this), then we know there is
some corner case that isn't yet covered by the packaging.

Remains to say that of course the buildds still need to receive some
work to also shut off networking during a build. That would require
stacking linux-vserver, which so far unfortunately isn't available
upstream. Other than that the rsync needs to be replaced by a less
hardware demanding way to generate a snapshot chroot - i.e. we need to
evaluate aufs or find some other good copy-on-write daemon to help us
here. Yet the limitation is not really the buildd nodes (some more
archives will be nice), but the real bottleneck right now is the dak
with wanna-build, britney and wanna-peruse itself.

So far from the shadowy grounds of Buildserver.NET. Any questions
unanswered?

-- 
Best regards,
Kilian

[1]: svn://svn.debian.org/pkg-voip/snapshot/pkg-voip-config
[2]: svn://svn.debian.org/pkg-gnome/tools/snapshots/scripts/snapshot-package
[3]: http://status.buildserver.net/excuses/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.alioth.debian.org/pipermail/pkg-voip-maintainers/attachments/20070806/e765cc7e/attachment.pgp 