[pkg-go] Bug#842380: gitlab-ci-multi-runner: mk-prebuilt-images.sh fails at install time

Vincent Danjean vdanjean at debian.org
Sun Oct 30 14:03:19 UTC 2016 

  Hi,

Le 30/10/2016 à 10:22, Dmitry Smirnov a écrit :
> Hi Vincent,
> 
> Sorry if my response reads a bit grumpy but I do not recognize your report as 
> bug.

So, I will try to argue a little bit more ;-)

> On Friday, 28 October 2016 6:00:49 PM AEDT Vincent Danjean wrote:
>> There is two problems:
>> 1) the cdebootstrap command is failing. I do not know why
> 
> It may be a problem in cdebootstrap, or in your APT proxy or in local mirror.
> We can't guarantee successful execution of cdebootstrap as gitlab-runner is 
> merely a user of cdebootstrap and the latter may fail due to number of 
> reasons.

ok

>>   but the current result (can be easily reproduced) is :
> 
> Not so easy apparently. I've tested gitlab-runner on x86_32 and x86_64 before 
> upload. Works for me.

Do you try the cdebootstrap command itself?
Here is, using a plain up-to-date sid chroot where I install
gitlab-ci-multi-runner to get all its dependencies:

vdanjean at atsina:~$ schroot -b -n test -c sid-amd64
vdanjean at atsina:~$ schroot -u root  -r -c test
atsina:/home/vdanjean# apt update
[...]
atsina:/home/vdanjean# apt full-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
atsina:/home/vdanjean# apt install gitlab-ci-multi-runner
[...]
0 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.
[...]
Setting up gitlab-ci-multi-runner (1.7.1+dfsg-1) ...
W: Docker is not available.
W: please run 'sudo /usr/lib/gitlab-runner/mk-prebuilt-images.sh' to generate Docker image.
invoke-rc.d: could not determine current runlevel
Setting up cdebootstrap (0.7.5) ...
[...]
atsina:/home/vdanjean# mkdir /tmp/test
atsina:/home/vdanjean# cd /tmp/test
atsina:/tmp/test# cdebootstrap \
    --flavour=minimal \
    --exclude="dmsetup,e2fsprogs,init,systemd-sysv,systemd,udev" \
    --include="bash,ca-certificates,git,netcat-traditional" \
    stable ./debian-minbase |& tee /tmp/log
P: Retrieving Release
P: Retrieving Release.gpg
[...]
P: Configuring helper cdebootstrap-helper-apt
E: Internal error: install
atsina:/tmp/test#

Do the cdebootstrap command really works on your systems?

=> that said, I think there is a bug in cdebootstrap here.
Nevermind, coming back to gitlab-ci-multi-runner.

>>   The mk-prebuilt-images.sh script continues but produce a image that does
>> not have the required tools (git in particular)
>>
>>   => docker is unusable.
> 
> Are you sure this is what happens? More likely image fails to build and 
> that's why docker executor can't work. You may still have older version of 
> gitlab-runner image which new version will not use.

I just installed gitlab-ci-multi-runner as my institution just provided me
a working gitlab server. So, I did not have older/previous version of
the /var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz.
  I also tried to remove this file. mk-prebuilt-images.sh recreates
a (not working) one.


>> 2) mk-prebuilt-images.sh does not report this problem and the postinst
>>   does not fail (hence the 'important' severity)
> 
> This is intentional! And for a good reason. Not everyone use Docker executor, 
> you know. Other executors are also useful and they do not require docker.
> 
> Not so long ago mk-prebuilt-images.sh used to print log to the console 
> instead of saving it and failure to configure package is not as useful as you 
> may think...

  In this case, please remove the bad gitlab-runner-prebuilt.tar.xz
Or ask the user to re-create it with a --force parameter to
mk-prebuilt-images.sh if he wants to keep a probably problematic
image.

>>   It tooks me lot of time to discover that

In fact, the only info I get back was from the gitlab server.
If I look at the details of the failed run, I see:
======
Running with gitlab-ci-multi-runner dev (1.7.1)
Using Docker executor with image vdanjean/jessie-git:v1 ...
Pulling docker image vdanjean/jessie-git:v1 ...
Running on runner-61fe6005-project-69-concurrent-0 via eyak...
/bin/bash: line 60: git: command not found
/bin/bash: line 41: cd: /builds/danjean/ocl-icd: No such file or directory
ERROR: Build failed: exit code 1
======

You can see the reported error message from gitlab is minimal.
It took me a long time (and a certain level of expertise) to understand
that the problem does not come from my 'debian' docker image, but
from a special docker image badly created at install time.

If the gitlab-runner-prebuilt.tar.xz file is not present, gitlab gives
this feedback:
======
Running with gitlab-ci-multi-runner dev (1.7.1)
Using Docker executor with image vdanjean/jessie-git:v1 ...
ERROR: Preparation failed: open /var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or directory
Will be retried in 3s ...
Using Docker executor with image vdanjean/jessie-git:v1 ...
ERROR: Preparation failed: open /var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or directory
Will be retried in 3s ...
Using Docker executor with image vdanjean/jessie-git:v1 ...
ERROR: Preparation failed: open /var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or directory
Will be retried in 3s ...
ERROR: Build failed (system failure): open /var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or directory
======

=> it is way more easy to understand there is a problem with the
gitlab-runner-prebuilt.tar.xz file.

As this file (or its building) is Debian specific, a README.Debian
in the doc of this package can talk about it and about the way to
recreate it. Something like (to be checked and/or improved):
=====
When used with the docker runner, gitlab-ci-multi-runner needs a docker
image that is used to setup the build environment (ie this image is *not*
the one that will be used to run the build).
The Debian package do not install a pre-built image but, instead, provides
a script to regenerate it. Just run '/usr/lib/gitlab-runner/mk-prebuilt-images.sh'
as root to rebuild it (or reconfigure the package with dpkg-reconfigure)
=====

>>   1) a special docker image is used to clone the repo
> 
> Well it took me a while to learn gitlab internals too. Though our learning 
> curve does not warrant bug like this.
> 
> 
>>   2) this special image is corrupted (the 'E: Internal error: install' can
>>     be seen in the log file whose name is displayed during the package
>>     configuration).
> 
> As I recall those errors may be harmless as cdebootstrap output is not 
> particularly pleasant even when everything is fine...

In this case, it should be reported back to cdebootstrap.
I really think that the default behavior should be to remove the
gitlab-runner-prebuilt.tar.xz if cdebbootstap (or another tool)
reports an error when building it.

>>     But, as the package installs fine, I do not suspect
>>     a problem here until I really llok in depth.
> 
> So far nobody was able to suggest a better way to build an image. It would be 
> great to ship gitlab-runner docker image in separate package but it needs 
> root privileges to build (as well as pre-configured build environment with 
> Docker) so we can't do that... In theory it may be possible to do with some 
> alternative tools... Improvement ideas are welcome...

Interesting question. I will think about it.

>> https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/1669
>>
>>   Perhaps, this bug can be duplicated and the first part reassigned
>> to cdeboostrap (the second part is really about this package)
>> if the cdeboostrap invocation is correct (I do not know cdeboostrap).
> 
> I've seen few sporadic errors from cdebootstrap due to outdated mirror.
> Most of the time cdebootstrap works for us through apt-cacher-ng over 
> httpredir.debian.org as well as directly (without proxy). IMHO you really 
> need to try another mirror or file a bug against cdebootstrap if you have a 
> reliable reproducer...

I'm using a 'standard' debian mirror (ftp.fr.debian.org), and the
error is still here after a few days. So it would be very strange that
the cdebootstrap bug was due to the mirror.

  But the gitlab-ci-multi-runner bug is to ignore the error and
to let a bad gitlab-runner-prebuilt.tar.xz file.


  Would you agree to remove/not install the gitlab-runner-prebuilt.tar.xz
file if an error occurs and if no --force parameter is passed to
mk-prebuilt-images.sh?
  And would you agree to add a README.Debian with a text similar to the
one I proposed, so that Debian users find easily where to look if
something goes wrong with the gitlab-runner-prebuilt.tar.xz file?

  My main point (for gitlab-ci-multi-runner) is to avoid to hide any
error and to provide enough feedback/documentation so that the user
knows where to look at if a problem occurs.

  Regards,
    Vincent

  Last side-note: cdebootstrat is currently using 'stable' as target
distribution. When the stretch will be released, your package will
start to use it without any previous testing... I cannot imagine a
good way to get a good behavior here :-(


-- 
Vincent Danjean       GPG key ID 0xD17897FA         vdanjean at debian.org
GPG key fingerprint: 621E 3509 654D D77C 43F5  CA4A F6AE F2AF D178 97FA
Unofficial pkgs: http://moais.imag.fr/membres/vincent.danjean/deb.html
APT repo:  deb http://people.debian.org/~vdanjean/debian unstable main

More information about the Pkg-go-maintainers mailing list