[PKG-Openstack-devel] [MoM] Packaging manila

[Thomas Goirand]

On 04/24/2015 04:12 PM, Andreas Tille wrote:
> Hi,
>
> sorry for the delay - I was a bit busy.

And so was I, with the release of OpenStack Kilo last week...

>> I installed the openstack-pkg-tools package on my system but could not find
>> a command which its name is pkgos-alioth-new-git. Also, I could  not find
>> the script on source code. finally, i found it out that there is a link to
>> the script on https://openstack.alioth.debian.org/.
>
> I admit I do not see this link myself.  Thomas, could you perhaps put
> this more prominently.  Newcomers seem to have trouble finding this
> script.  (I can confirm that it is not part of openstack-pkg-tools
> package.)

I have uploaded version 24 of openstack-pkg-tools in Sid. Please use 
that one from now on, it includes pkgos-alioth-new-git.

>> Then, I tried to create a Test rgit reo, using the script. The created git
>> repository is not observable on http://git.debian.org/openstack.  Is this
>> the issue that Thamos has mentioned it, before?
>>     "This should be slightly adapted if you're not add (to handle a
>> different username with "-     guest" at the end). "

I'm not sure, really. The -guest thing was reported by another 
contributor, and I haven't experienced it myself. Maybe just try again, 
and check the script output?

>> If they are not related to the same issue, would you guide me why may repo
>> wont be observable if i use the script? Moreover would you please explain a
>> little about what is the mentioned problem by Thomas?
>
> I think it might be a good idea to use the script with "set -x" to see
> where it fails and in case of problems do all the steps manually.

Exactly. And contribute the fix if there's one.

>  If
> you send me the link where you downloaded the script I could check
> myself but I guess it will be no magic and contains some "git init" etc.
> stuff that could be done manually for sure.

Vcs-Browser: 
http://anonscm.debian.org/gitweb/?p=openstack/openstack-pkg-tools.git;a=summary
Vcs-Git: git://anonscm.debian.org/openstack/openstack-pkg-tools.git

Please use the debian/kilo branch. The script is in the misc folder.

>> The script uploads the local git repository on Alioth.  So I needed to
>> create my local Manila, first.
>> Therefore, I tried different ways for creating initial commit on Manila
>> repo.
>> I could choose between 1) last version of Manila on github (tag:
>> 2015.1.0rc1)

Yes, this is what you should do.

> 2) Manila source code on Vivid, and 3) the last uploaded
>> version on Manila (which James had mentioned it on the other e-mail).

Don't use Ubuntu as Debian's upstream. Please use upstream code from 
upstream repository:

git clone git://github.com/openstack/manila.git
cd manila
git reset --hard 2015.1.0
git checkout -b debian/kilo
[ add your debian folder here ...]
git add debian/*
git commit -a -m "Initial debian folder"

Something like this...

> Please always keep in mind that I do not know the relation between the
> launchpad and the github, but *I* consider it good practice to use the
> version that is released by upstream and as far as I can see upstream is
> at github.

Nearly! Github is only a place where there's a copy from 
git.openstack.org. I often use github as it's faster to download form 
there, but using git.openstack.org is also ok.

> To fetch the latest version from github you can use a
> debian/watch file with the following content:
>
> version=3
> opts="uversionmangle=s/([\d.]*)([a-zA-Z])/$1~$2/;s/\.0b/~b/" \
>    https://github.com/openstack/manila/releases .*/archive/(\d[\d.-brc]+)\.(?:tar(?:\.gz|\.bz2)?|tgz)
>
> If you do this and go to the main packaging dir (== the dir where you
> see the content above if you do `cat debian/watch`) than you can do
>
>   $ uscan --verbose --report
> -- Scanning for watchfiles in .
> -- Found watchfile in ./debian
> -- In debian/watch, processing watchfile line:
>     opts="uversionmangle=s/([\d.]*)([a-zA-Z])/$1~$2/;s/\.0b/~b/"   https://github.com/openstack/manila/releases .*/archive/(\d[\d.-brc]+)\.(?:tar(?:\.gz|\.bz2)?|tgz)
> -- Found the following matching hrefs:
>       /openstack/manila/archive/2015.1.0rc1.tar.gz (2015.1.0~rc1)
>       /openstack/manila/archive/2015.1.0b3.tar.gz (2015.1.0~b3)
>       /openstack/manila/archive/2015.1.0b2.tar.gz (2015.1.0~b2)
>       /openstack/manila/archive/2015.1.0b1.tar.gz (2015.1.0~b1)
>       /openstack/manila/archive/2014.2.tar.gz (2014.2)
>       /openstack/manila/archive/2014.2.rc2.tar.gz (2014.2.~rc2)
>       /openstack/manila/archive/2014.2.rc1.tar.gz (2014.2.~rc1)
>       /openstack/manila/archive/2014.2.b3.tar.gz (2014.2.~b3)
> Newest version on remote site is 2015.1.0~rc1, local version is 2015.1~b2
>   => Newer version available from
>      https://github.com/openstack/manila/archive/2015.1.0rc1.tar.gz
> -- Scan finished
>
> If you do
>
>   $ uscan --verbose --force-download

PLEASE NO !!!!!!! We're using git tag packaging. So don't get any 
tarball involved here. Just git clone from github, and do:

./debian/rules gen-orig-xz

If you have a correct debian/changelog and debian/rules, then 
openstack-pkg-tools will generate the orig.tar.xz for you correctly.

> I *personally* hate dh-make since it leaves you alone with a lot of
> useless templates you need to delete afterwards.

Same over here. Using what's already done in the cinder package seems a 
much better idea, since Manila is based on Cinder if I'm not mistaking.

> I'd (strongly) recommend to start from this since this is most probably
> the best way to start from.

I'd very much prefer using Cinder as a base.

>> if iI copy ubuntu-Manila debian dir,
>> is it needed to change some thing like control file, changelog, ...?

Yes. We're not packaging the same way. The debian/control dependencies 
may be adapted for Trusty, but not for Sid / Jessie. The debian/rules 
needs to be adapted for using the tag-based packaging style too (see 
what's on top of Cinder's debian/rules for example).

>> Or I
>> should just do the first commit, and then we will try to adapt it for
>> Debain Repositories?
>
> You should have a close look into some other package done by the
> openstack team and adapt the "Maintainer" and "Uploaders" field
> accordingly.

Yes, as well.

>> I tried to find some thing about tag-based packaging or guess how may it
>> works, but I could not find any thing unfortunately.
>
> Unfortunately I also have no idea about this.  Thomas, could you clarify
> please what exactly do you mean.  I personally have no idea what to do
> once the tarball is downloaded via
>
>      uscan

Don't use the tarball downloaded via uscan. Just generate one using:

./debian/rules gen-orig-xz

once you have a correct debian/changelog for which the version matches 
the tag which you wish to package.

> While I'd like to adapt to the openstack workflow I think
> if we do not get any enlightenment here we stick to the Git workflow I'm
> used to. I'm afraid we might simply loose our goal to deliver a package
> out of sight if we are fighting with different workflows.  As I
> understood the Ubuntu folks correctly they are only interested in a
> Debian source package.  We will get this with any workflow.  So I have
> the following suggestion:  If we do not find out how to do tag based
> packaging I will create a Git repository using pristine-tar as many
> other teams are successfully using and inject the debian/ dir up to the
> point I described above. I will do this tomorrow morning in case nobody
> insists.

I insist: please don't do this. All my tooling, including the automated 
build using jenkins, are using this git tag thing. It's also well 
explained on http://openstack.alioth.debian.org/ (there's even a HUGE 
snapshot of the pkgos.make which is included in each package, with the 
command for generating the orig.tar.xz... you can't miss that one!). If 
you miss some info there, let me know, and I'll try to explain better.

> Hmm, reading this again this is possibly what Thomas means with "tag
> based packaging".

By "tag based packaging", I mean using a git tag to generate the 
orig.tar.xz file, instead of using pristine-tar. So this way, a single 
git branch is used (no need to use pristine-tar or upstream branches). 
Bonus point: we can check the upstream tags with gpg, and there's no 
risk to do any mistake with what's in an upstream tag. It's also fully 
reproduceable and the integrity of the orig.tar.xz can be checked (I 
can't say the same thing with a tarball stored on Alioth...).

> So please if there are remaining problems:  Do *every* single step and
> post the output (may be shortened) to enable me reproducing what you did
> and comment ond potential problems.  If you confirm that all above went
> smoothly you might just hang on the pkgos-alioth-new-git script which as
> I wrote above could be separated into simple steps most probably.  Just
> let me know about any single step and any single failure that might
> occure.  I'll try to reproduce this here and will take over things that
> might fail on your side (for whatever reason).

On http://openstack.alioth.debian.org/, there's a section with as title:

"Once your done: uploading to Alioth with correct rights"

which explains the steps. Basically, the pkgos-alioth-new-git creates a 
bare clone of the git repository on your laptop, it makes a tarball out 
of it, uploads that to Alioth with scp, extract it there, then fixes the 
perms as explained in the page above...

> I guess the retagging is done since in Debian we are using the tilde (~)
> to make sure we can easily increase the version number when there is an
> official release (without the attached "b3").  You might like to have a
> look at the following comparison:
>
>    if dpkg --compare-versions 2015.1~b3 lt 2015.1 ; then echo "2015.1~b3 is lower than 2015.1" ; fi
>
> That's the reason why in debian/changelog you find the '~' before the
> 'b' in the version string and this in turn is the reason why in
> debian/watch the '~' is injected in the uversionmangle expression (see
> above).  However, (as far as I know) the tilde is no valid character for
> Git tags and thus in Git tags it is emulated with an underscore ('_').
>
> On the other hand you are correct that Thomas was not refering to the
> latest upstream tag (as we know from the uscan --report above) and thus
> I think it is rather
>
>     git tag 2015.1.0_rc1 2015.1.0rc1
 >
> what we want to write here.  Please note that I have currently no clone
> on my machine - may be the upstream tag is different - adapt if needed.

All correct. Now we don't need retagging, as 2015.1.0 is out anyway.

Cheers,

Thomas