[Pkg-zfsonlinux-devel] Kernel image and OOT auto builder environment (Was: Using out of tree modules in d-i)?

[Aron Xu]

On Fri, May 24, 2013 at 11:53 PM, Turbo Fredriksson <turbo at bayour.com> wrote:
> On May 22, 2013, at 3:52 PM, Ben Hutchings wrote:
>
>> Quoting from the report of our 2009 meeting,
>> <20091015123106.GA16736 at kyllikki.org>:
>>> out of tree modules
>>> -------------------
>>>
>>> After a somewhat involved discussion taking into account the FTP
>>> masters extreme irritation about trying to match binaries to source by
>>> hand for the lenny release it was resolved to remove
>>> linux-modules-extra and -nonfree as they are an impossible to support
>>> approach.
>>
>> The Built-Using header should cover FTP masters' concerns.  However it
>> is still the case that omnibus source packages are unsustainable as many
>> OOT modules are not kept up to date with the kernel API.
>
> I haven't been keeping up with the inner workings of Debian GNU/Linux
> the last couple of years, so please take this as-is.
>
>
> Is it possible to setup an automatic build system for kernel and modules?
>
> I know that we have (had?) an auto builder to build everything automatically
> (think it was mostly (?) used for the ports), but I was thinking about a
> special build system here. Very possibly just a modification of what we
> already have...
>
>
> That is, a special place to upload the kernel source to, and once the auto
> builder have successfully built a linux-image* package(s), then it will
> look in a special list for source packages to build as well. And once
> all those have succeeded to, THEN it will upload the kernel (and the
> OOT modules) to the ftp archive...
>
> To clarify: the kernel maintainer does not build and upload the package
> to the archives, but only uploads the source package to this special
> build system and it will do the rest...
>
> That way OOT modules should be able to keep up with the kernel releases
> and uploads without much (hopefully) extra work. And the added benefit
> is that once a new kernel version is available in the archives, the
> OOT modules will as well, without any extra lagg...
>
>
> This should be reasonably easy to do, and I volunteer to do the initial
> setup and/or prof-of-consept for all the versions we currently support
> (oldstable, stable and unstable).
>

Having something like this may (or may not) generally help, but I'm
not sure this is what we want to solve the problem. If OOT modules are
added into d-i images directly, then we must set up something like
this proposal to make sure the modules are available whenever a new
kernel ABI is uploaded, and help d-i people to handle the brokenness
if a change in kernel makes the OOT module does not build, or even
function badly. This is hard to do, and I think there could be easier
way for this specific issue.

What I can think of is to do the trick in d-i, since it already has
the ability to retrieve and load udeb on the fly, and even prompt
users for missing firmware. Such functionality may be reused so that
related udebs can be fetched and loaded when selected, and if all
effort failed just generate an error message.

By doing this a small check is performed to make sure the d-i kernel
and the OOT modules are matched, so that d-i may get the ability to
include OOT modules in the image, and the worst case is wasting
several megabytes in the resulting images. debian-cd may also be
improved to check OOT modules included in the image has correct
version info to work together with the target d-i kernel to avoid such
a waste. Finally, OOT modules may be listed in a separate list so that
users are aware what they are doing, and d-i team can coordinate with
maintainers of modules they want to include whenever an "official"
image is generated (e.g. release CDs).

As for how the small check would be implemented, the question can be
discussed later in more detail if we agree that it's the preferred way to
go with.

-- 
Regards,
Aron Xu