[Soc-coordination] Weekly report - Debian GNU/Hurd Debianish initialization

Sun Jun 23 07:52:31 UTC 2013

Hi Planet Debian :)

my name is Justus Winter, I'm a computer science student at the
University of Hamburg. I'm participating in the GSOC 2013, my `task
<https://wiki.debian.org/SummerOfCode2013/StudentApplications/JustusWinter>`_
is to make Debian/Hurd systems boot using the same initialization
scripts Debian/Linux uses.

I spent the community bonding period with some gsoc related
administrative tasks, updating my development machine and getting
reacquainted with the Hurd system. It was nice to see how far
Debian/Hurd has come since my last "serious" `encounter
<https://lists.gnu.org/archive/html/bug-hurd/2010-05/msg00033.html>`_
with it.

I've setup a continuous build solution on one of my Hurd VMs and
prepared a package repository. I'll post details once I have some
packages to share.

I built the hurd, sysvinit and ifupdown packages and dived into the
interesting world of literate programming. I'm going to write about
that when I start looking into the ifupdown related issues.

But most importantly I started writing a tool to automate all the
tasks I would do during my work that could possibly be automated. As
I'm working on the boot process, it makes sense to use some kind of
virtualization software so I can work in the comfort of my usual
development environment. My virtualization solution of choice is qemu
and I actually wrote a qemu based black-box test solution for a former
employer once, so I already was familiar with qemu and wrapping my own
code around it. Without further ado, here it is:

https://bitbucket.org/teythoon/hurdtest

Provided that you have `golang` installed, you can compile it easily::

    teythoon at europa /tmp % export GOPATH=`pwd`
    teythoon at europa /tmp % go get bitbucket.org/teythoon/hurdtest
    teythoon at europa /tmp % bin/hurdtest
    Usage: hurdtest [options] <command> [<arguments...>]

Currently it has the following subcommands:

test
    Starts a VM, logs in as root, tests the interaction with the VM,
    verifies that certain VFS locations are writable, tests the
    network connectivity, the DNS resolution and installs and runs the
    `hello` package.

update
    Executes `aptitude update && aptitude full-upgrade`, reboots the
    system and runs the `test` subcommand. If successful, it commits
    the changes to the virtual disk and creates a snapshot.

overlay
    Takes as argument the path to a directory that is copied to the
    VM, reboots the machine and runs the `test` subcommand.

The `update` command is nice because it allows me to keep the VM up to
date while ensuring that the image remains usable, with respect to
what the `test` command considers a usable state of a Debian
system. I'd really like to get some suggestions on how to test the
"health" of a Debian system. You can look at how the tests are
implemented in `test.go
<https://bitbucket.org/teythoon/hurdtest/src/75408323a0f78e63aaa36bc4b950dc78de458b61/test.go?at=default>`_,
it's quite easy to add more and more interesting tests, and you don't
even have to clean up after them or worry about any side-effects as
the changes are not committed to the virtual disk.

This is `hurdtest` doing an update::

    teythoon at europa ~/repos/hurd/hurd-tests % bin/hurdtest update
    2013/06/22 19:59:45 starting kvm [-drives file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/285813369/serial -monitor unix:/tmp/285813369/monitor -nographic -m 512]
    2013/06/22 19:59:45 Waiting for grub...
    2013/06/22 19:59:46 Waiting for login prompt...
    2013/06/22 20:00:05 Updating system
    2013/06/22 20:02:52 Get: 1 http://www.local unstable InRelease [8279 B]
    Get: 2 http://http.debian.net sid InRelease [205 kB]
    Get: 3 http://ftp.debian-ports.org unreleased InRelease [23.9 kB]
    [...]
    Setting up perl (5.14.2-21) ...
    Reading package lists...
    Building dependency tree...
    Reading state information...
    Reading extended state information...
    Initializing package states...
    Reading task descriptions...
    2013/06/22 20:02:52 Update successful. Snapshotting system...
    2013/06/22 20:02:53 Rebooting system...
    2013/06/22 20:02:54 Waiting for grub...
    2013/06/22 20:02:59 Waiting for login prompt...
    2013/06/22 20:03:16 Running main.testTrue...
    2013/06/22 20:03:16 Running main.testFalse...
    2013/06/22 20:03:17 Running main.testClean...
    [...]
    2013/06/22 20:03:18 Running main.testNetwork...
    2013/06/22 20:03:18 Running main.testResolver...
    2013/06/22 20:03:20 Running main.testAptitudeInstall...
    2013/06/22 20:03:46 Test successful. Reverting system...
    2013/06/22 20:03:48 Halting system...
    2013/06/22 20:03:53 Committing changes...
    2013/06/22 20:03:57 Creating snapshot...
    2013/06/22 20:04:04 All done.
    bin/hurdtest update  157.31s user 69.71s system 87% cpu 4:18.94 total

The `overlay` command implements my work flow. I can edit scripts or
drop hacked up executables onto my VM and see the results without any
manual intervention. There's the `-interactive` flag that does this in
a loop, avoiding the VM startup time using qemu snapshots. With this I
can apply an overlay and see the results of the test suite in about a
minute on my system.

From early experiments I knew that simply running `/etc/rcS.d/S*`
prevented the system from booting correctly, so I `figured
</gsoc/roadmap>`_ my first task should be to figure out what went
wrong there. Using my tool (and developing it in that process) and
many iterations later I came up with a surprisingly small diff of
scripts in the `initscripts
</gsoc/heap/first-week-initscripts.patch>`_ and utilities shipped with
the `hurd </gsoc/heap/first-week-hurd.patch>`_ package. For your
convenience here is a `tarball
</gsoc/heap/first-week-overlay.tar.xz>`_ that you can extract to the
root of your Debian/Hurd installation to reproduce my results (of
course I invite you to use my tool for that purpose)::

    % bin/hurdtest overlay test
    2013/06/22 21:35:31 starting kvm [-drive file=image.qcow2,cache=writeback,snapshot=on -serial unix:/tmp/345188134/serial -monitor unix:/tmp/345188134/monitor -nographic -m 512]
    2013/06/22 21:35:31 Waiting for grub...
    2013/06/22 21:35:31 Waiting for login prompt...
    2013/06/22 21:35:48 Creating tarball...
    ./
    ./etc/
    ./etc/hurd/
    ./etc/hurd/rc
    ./etc/init.d/
    ./etc/init.d/checkroot.sh
    ./etc/init.d/mountall.sh
    ./runme
    ./lib/
    ./lib/init/
    ./lib/init/mount-functions.sh
    ./sbin/
    ./sbin/swapon
    ./bin/
    ./bin/mount
    2013/06/22 21:35:48 Extracting overlay...
    2013/06/22 21:35:49 Rebooting system...
    2013/06/22 21:35:49 Waiting for grub...
    2013/06/22 21:35:55 Writing log to /tmp/bootlog-802006344
    2013/06/22 21:35:55 Waiting for login prompt...
    2013/06/22 21:36:20 Booting successful.
    2013/06/22 21:36:20 Running main.testTrue...
    2013/06/22 21:36:21 Running main.testFalse...
    2013/06/22 21:36:21 Running main.testClean...
    2013/06/22 21:36:21 Running anonymous function...
    2013/06/22 21:36:21 Running anonymous function...
    2013/06/22 21:36:21 Running anonymous function...
    2013/06/22 21:36:21 Running anonymous function...
    2013/06/22 21:36:21 Running anonymous function...
    2013/06/22 21:36:22 Running anonymous function...
    2013/06/22 21:36:22 Running main.testUname...
    2013/06/22 21:36:22 Running main.testNetwork...
    2013/06/22 21:36:22 Running main.testResolver...
    2013/06/22 21:36:23 Running main.testAptitudeInstall...
    bin/hurdtest overlay ../hurd-tests-old/test  44.20s user 20.01s system 82% cpu 1:17.94 total

And here is the boot log::

    Loading GNU Mach ...
    Loading the Hurd ...
    GNU Mach 1.3.99-486
    AT386 boot: physical memory map from 0x0 to 0x9f400
    AT386 boot: physical memory map from 0x100000 to 0x1fffe000
    AT386 boot: physical memory from 0x0 to 0x1fffe000
    Enabling FXSR
    pcibios_init : BIOS32 Service Directory structure at 0xfcff0
    pcibios_init : BIOS32 Service Directory entry at 0xfc7ba
    pcibios_init : PCI BIOS revision 2.10 entry at 0xfc78c
    Probing PCI hardware.
    ide: Intel 82371 PIIX3 (dual FIFO) DMA Bus Mastering IDE 
        Controller on PCI bus 0 function 9
    ide: BM-DMA feature is not enabled (BIOS), enabling
        ide0: BM-DMA at 0xc100-0xc107
        ide1: BM-DMA at 0xc108-0xc10f
    hd0: got CHS=762/128/63 CTL=c8 from BIOS
    hd0: QEMU HARDDISK, 3001MB w/256kB Cache, CHS=762/128/63, DMA
    hd2: QEMU DVD-ROM, ATAPI CDROM drive
    ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
    ide1 at 0x170-0x177,0x376 on irq 15
    Floppy drive(s): fd0 is 1.44M
    FDC 0 is a S82078B
    probing scsi 0/22: advansys probing scsi 1/22: BusLogic probing scsi 2/22: UltraStor 14F/34F rev. 4.33.00  probing scsi 3/22: UltraStor 14F/24F/34F probing scsi 4/22: Adaptec 152x SCSI driver; $Revision: 1.1 $ probing scsi 5/22: Adaptec 1542 Invalid address for shpnt with 1542.
    Invalid address for shpnt with 1542.
    probing scsi 6/22: Adaptec 174x (EISA) probing scsi 7/22:  probing scsi 8/22: Always IN2000 probing scsi 9/22:  probing scsi 10/22:  probing scsi 11/22:  probing scsi 12/22: Trantor T128/T128F/T228 probing scsi 13/22: DTC 3180/3280  probing scsi 14/22: Tekram DC390(T) V1.11 Feb-05-1997 probing scsi 15/22:  probing scsi 16/22: ncr53c8xx - revision 2.5f.1 probing scsi 17/22: EATA (Extended Attachment) PIO driver probing scsi 18/22: Western Digital WD-7000 Failed initialization of WD-7000 SCSI card!
    probing scsi 19/22: EATA/DMA 2.0x rev. 4.33.00  probing eata on 1f0EATA0: address 0x1f0 in use, skipping probe.
    probing eata on 1c88probing eata on 2c88probing eata on 3c88probing eata on 4c88probing eata on 5c88probing eata on 6c88probing eata on 7c88probing eata on 8c88probing eata on 9c88probing eata on ac88probing eata on bc88probing eata on cc88probing eata on dc88probing eata on ec88probing eata on fc88probing eata on 170EATA0: address 0x170 in use, skipping probe.
    probing eata on 230probing eata on 330probing scsi 20/22: AM53C974 probing scsi 21/22: Iomega parport ZIP drive ppa: Version 1.42
    ppa: Probing port 03bc
    ppa: Probing port 0378
    ppa:     SPP port present
    ppa:     PS/2 bidirectional port present
    ppa: Probing port 0278

    done
    scsi : 0 hosts.
    scsi : detected total.
    Partition check (DOS partitions):
     hd0: hd0s1 hd0s2 < hd0s5 >
    lpr0: at atbus0, port = 378, spl = 6, pic = 7.
    module 0: ext2fs --readonly --multiboot-command-line=${kernel-command-line} --host-priv-port=${host-port} --device-master-port=${device-port} --exec-server-task=${exec-task} -T typed ${root} $(task-create) $(task-resume) module 1: exec /hurd/exec $(exec-task=task-create)                                                                                                                                                                           2 multiboot modules                                                                                                                                                                                                                    task loaded: ext2fs --readonly --multiboot-command-line=root=device:hd0s1 console=com0 --host-priv-port=1 --device-master-port=2 --exec-server-task=3 -T typed device:hd0s1
    task loaded: exec /hurd/exec

    start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
    Using makefile-style concurrent boot in runlevel S.
    ERROR: could not open /proc/stat: No such file or directory
    Activating swap...done.
    Checking root file system...fsck from util-linux 2.20.1
    hd2 : tray open or drive not ready
    hd2 : tray open or drive not ready
    hd2 : tray open or drive not ready
    hd2 : tray open or drive not ready
    end_request: I/O error, dev 02:00, sector 0
    ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/hd0s1 is mounted.
    /dev/hd0s1: clean, 44328/181056 files, 288266/723200 blocks
    done.
    Cleaning up temporary files... /tmp.
    mount: /run: Unknown device or filesystem
    mount: /run/lock: Unknown device or filesystem
    mount: /proc: Unknown device or filesystem
    mount: /run/shm: Unknown device or filesystem
    /etc/rcS.d/S06mtab.sh: 48: /etc/rcS.d/S06mtab.sh: cannot open /proc/mounts: No such file
    Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
    done.
    Checking file systems...fsck from util-linux 2.20.1
    hd2 : tray open or drive not ready
    hd2 : tray open or drive not ready
    end_request: I/O error, dev 02:00, sector 0
    done.
    Mounting local filesystems...mount: invalid option -- 'O'
    Try `mount --help' or `mount --usage' for more information.
    failed.
    Activating swapfile swap...(default pager): Already paging to partition hd0s5!
    done.
    df: Warning: cannot read table of mounted file systems: No such file or directory
    Cleaning up temporary files....
    Configuring network interfaces...inetutils-ifconfig: invalid arguments
    ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
    Internet Systems Consortium DHCP Client 4.2.2
    Copyright 2004-2011 Internet Systems Consortium.
    All rights reserved.
    For info, please visit https://www.isc.org/software/dhcp/

    can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
    Listening on Socket//dev/eth0
    Sending on   Socket//dev/eth0
    DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 5
    DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
    DHCPOFFER from 10.0.2.2
    DHCPACK from 10.0.2.2
    can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
    bound to 10.0.2.15 -- renewal in 36884 seconds.
    done.
    Cleaning up temporary files....
    Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
    Using makefile-style concurrent boot in runlevel 2.
    Starting enhanced syslogd: rsyslogd.
    Starting deferred execution scheduler: atd.
    Starting periodic command scheduler: cron.
    Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
    Starting OpenBSD Secure Shell server: sshd.

    GNU 0.3 (debian) (console)

    login:

I know it isn't that pretty, but it yields a working Debian/Hurd
system that is more like a Debian/Linux system. For example `/dev/shm`
and `/run{,/*}` are connected to a `tmpfs` translator. I think this is
a rather encouraging first result. I added my findings to the list of
`open issues </gsoc>`_.

Another nice thing about `hurdtest` is that it is not even Hurd
specific (only a tiny bit GNU Mach specific). I'm going to make it
work nicely with Debian/Linux and Debian/kFreeBSD, since this is a
very handy tool for anyone hacking on the Debian boot process (and
with some changes for any OS). If anyone tries it out, I'd love to get
feedback and patches.

The tool requires the guest to be configured to use the first serial
console for grub, the kernel and the userspace. The password for the
user root must be empty. The guest has to have a working network
connection setup at boot time. I will provide a suitable image as soon
as I devise a good way to zero-fill the free space on the virtual
disk. Is there any better way than to fill the disk with a huge file
containing zeros? Without doing that the image is too large even
compressed with xz.

So this was my first week working for Debian. Next week I will focus
on getting patches for the `hurd` and `initscripts` package in shape,
I need to add some functionality to the `mount` utility shipped with
the `hurd` package. It provides a `mount`-like experience on the Hurd,
on which binding a translator (e. g. `/hurd/ext2fs`) to a filesystem
node is an operation that does not involve the kernel (`GNU Mach`) at
all, and is in fact a operation that any unprivileged user may
perform. That is one of the nice features the Hurd has, it empowers
its users.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: signature
URL: <http://lists.alioth.debian.org/pipermail/soc-coordination/attachments/20130623/66e4e4a0/attachment-0001.sig>