[Debichem-devel] Git migration plan

[Daniel Leidert]

Am Donnerstag, den 25.01.2018, 14:40 +1100 schrieb Stuart Prescott:
[..]
> > Now many of our packages have a clean subversion history and the
> > resulting GIT repositories look fine to me. Others had a more
> > complicated life, especially when branched off or tagged several times.
> > For these I did my best to make sensible decision writing the ruleset.
> > I checked the results by looking at the graphical log version (git log
> > --all --graph) and the references (git show-ref). I also imported the
> > results several times to salsa.d.o.
> 
> A common problem I've seen in the conversion is that the tagged commit made by 
> svn-bp is actually a separate commit that is off the history rather than 
> simply a tag on the final commit. This is only seen in some situations but 
> I've not worked out which combination of svn, svn-bp or workflow leads to it. 

That's due to so called mixed-revision-tags, created by running "svn
cp" without calling "svn update" first. Anders Kaseorg explained it in
#887881. If you take a look at the subversion log, you'll see, that
many of our tags have several parent revisions instead of just one.

> Programmatically fixing it is not impossible but is difficult.

What about this: After creating the Git repositories, get the parent
commit IDs of a tag, determine the youngest one, compare its content
tree to the current tag content tree and if both match (safety net),
recreate the tag. I'm not that familiar with using `git log` in
scripts, but it might be possible to do this. Help is appreciated.

> Personally, I've not worried about such things since the tag has the correct 
> content, it just doesn't appear within the linearised history.

ACK.

> > - Packages ignored are those
> 
> [...]
> >   Their files go into a fake repository called "junk" (based on Stuart
> > Prescotts ruleset).
> 
> FAOD, it was never my intention to do anything with junk.git; it only existed 
> in my rules to stop svn-all-fast-export from dying on the import and would be 
> never be uploaded to salsa.

Well, my interpretation was different, because you could have simply
ignored it by removing the repository-line in this match-rule. 

>  I don't think it is of any interest to keep, but 
> have I missed something there?

I'd like to keep it for the moment. Because subversion will completely
vanish, everything inside the repository, that is not saved now, will
not be recoverable. So I thought, that we leave it in junk.git and
delete the repository it in a year or so.

> > So if you would like to help, these are the tasks and open questions
> > left before the conversion:
> 
> [...]
> > - Is it ok to use a branch wnpp or shall it be named differently?
> 
> For old packages, sure, let history be history. Or squash that branch into 
> master.

The latter is not that easy. I tried that with several packages, but I
often ended up with gaps in the git log for the moment, where the
package files were moved from wnpp to unstable. IIRC it only worked for
travis. So I decided to not try any further. 

[..]
> For future packages, I don't think there's a need for a different branch name 
> for not-yet-uploaded packages. (It made sense for the svn workflow where 
> everything was in one repository but less so with separate git repo per 
> package. I guess we would be reliant upon blends or PET or similar to keep 
> track of never-uploaded packages.)

Well, I'd suggest then, that we make wnpp the master branch after the
conversion, because of the issue described above. IMO only a few
packages are affected.

> > - Are there hooks, templates, ... etc. that should be created for every
> > repository? Do we need to setup a repository template at
> > salsa.debian.org for every repository? Is there anybody already
> > familiar with the latter?
> 
> There are two options here -- straight to salsa or park on alioth for a short 
> period. I suspect going straight to salsa is a better idea.

ACK.

> Hooks can be enabled on salsa using its http API (as can creation of the repo 
> itself within a team). I think the current state of the art is described at
> 
> http://blog.dogguy.org/2018/01/salsa-webhooks-and-integrated-services.html
> 
> although all of this is sufficiently new and volatile that we may have already 
> moved on from there.

>From my understanding, hooks are also not allowed at salsa.d.o for
safety reasons except for the webhooks you mentioned. So to my
understanding there is currently nothing to do for the created Git
repositories in regard to hooks/templates.

> > - Tags called backup/* are used by svn-all-fast-export to mark the last
> > commit, before a branch (sometimes tags, but these should all be fixed
> > already) was deleted. These can be removed if you feel no need to keep
> > them. Please check the graphical log and/or the subversion history if
> > necessary.
> 
> I notice in your script that you are copying the bare repo created via svn-
> all-fast-export. Instead, it is best to immediately clone that repo (either a 
> bare clone or otherwise). There tends to be a lot of mess left in the repo 
> that svn-all-fast-export creates (including all these backup tags) and from 
> memory that initial clone lets git get rid of them. The call to git-repack 
> does some of this but the recommendation is to clone. (Yes, I am painfully 
> aware that svn-all-fast-export is woefully underdocumented; unfortunately, 
> it's a tool that by the time you've become expert enough in using it, you 
> probably stop using it because you've done all the conversions you need.)

Thanks for the advice. Didn't know that. I'll try to do a clone, then
fix the tags as mentioned at the top, then create a bare repo and
upload it to salsa.d.o.

> I agree that these tags should be removed prior to pushing to salsa.
> 
> 
> Final steps in the conversion that can be considered:
> 
> - make an index for mr (aka myrepos) to automate fetching all repos within the 
> team. (Not my cup of tea but those used to doing mass changes to all packages 
> via the omnibus svn repo tend to like it.)

No idea, what you are talking about :) I'll check it out.

> - updating debian/control to add Vcs-Git, remove Vcs-Svn and update Vcs-
> Browser; a simple sed is probably enough but I can help with python-debian's 
> deb822 if not. I assume it's worth doing that on all packages immediately even 
> if they are not yet uploaded.

ACK. sed and dch might be enough. The commit could be done locally
before creating the bare repo and uploading it. Help is appreciated.

> BTW I started looking at git-buildpackage to see if those hooks could be 
> extended to have further substitutions for you. It looks possible but I've not 
> had time to do it and test it.

Also thanks for the advice. I'll take a look at it too.

Regards, Daniel
>