patches-applied historical imports (usd import)

[Nish Aravamudan]

On 09.01.2017 [13:40:16 +0000], Ian Jackson wrote:
> Nish Aravamudan writes ("patches-applied historical imports (usd import)"):
> > 1) Some source packages (bouncycastle, php7.0 are the ones I can think
> > of off the top of my head) upstream tarballs contain .gitattributes
> > files, which will change the behavior of git itself when checking out a
> > branch.
> 
> I need to do some tests to see exactly how to work around this.
> Can you point me at an affected source package version ?

I believe all php7.0 source packages will have a .gitattributes entry
containing 'ident' attributes amongst others.

> > 2) How do we determine if a source package is 1.0 vs. 3.0? I am
> > currently using `dpkg-source --print-format`, but have found one source
> > package (util-linux 2.13~rc3-5), where dpkg-source emits:
> > 
> > syntax error in /tmp/tmp3y515osf/util-linux-2.13~rc3/debian/control at
> > line 14: duplicate field Depends found
> > 
> > and thus we error out.
> 
> How exciting.  dgit looks at debian/source/format, which is what
> dpkg-source reads when generating the package.

Yep, based upon Sean's advice, I will switch to using that as well.

> > 3) Imagine the following graph in the git repository:
> > 
> >    A           D
> >  ->o.....f....>o->
> >    ^ .       . ^
> >    |  c     e  |
> >    o   .   .   o
> >    ^    . .    ^
> >    |     B     |
> >    o     o     o
> >    ^     ^     ^
> >    |     |     |
> >  ->o---->o---->o-> 
> >    a     b     d
> ...
> > Let's assert that there is a problem with obtaining the patches-applied
> > version of b. This can occur for (at least) the following reasons:
> > 
> > i) as in 2), we might not be able to determine the source package format
> > (implementation detail, to some degree), so are unable to correctly
> > derive if there is a patches-applied state that is distinct from b.
> 
> You can use dpkg-source -x to extract the patches-applied state
> corresponding to any .dsc, I think.  Any case where you can't is a bug
> in dpkg-source.

Right, that will get me to the fully patches-applied state, but what
about the intervening ones?

> > ii) some patches may fail to apply with a trivial `quilt push`. This
> > occurs with, at least, a historical publish of samba.
> 
> Do you have an example source package ?

I will re-run the import today and get that info for you.

> > In theory, there are other reasons/cases where this might happen and the
> > importer needs to never fail (so it is of some use to run automatically
> > :)
> 
> I haven't run dgit's dsc importer on a whole historical archive but
> Peter Green of Raspian has been running it and filing bugs.  I haven't
> seen such a bug recently so I hope it has been working for all the
> packages he's seen.

And it's also very possible that it may not be a Debian publish, but an
Ubuntu publish where this occurs :)

> > The questions I have relate to what to do when we encounter this
> > situation, which in turn is divided into two parts:
> 
> I think this situation must be excluded.
> 
> This kind of difficulty is one of the main reasons for the failure of
> previous efforts to transition universally to git.  It is why dgit
> does not currently work, in Debian, with a full history of each
> package.  Instead there's a bit of a bodge, with a locally generated
> history.  Joey Hess and I came up with this approach at the Vaumarcus
> Debconf.
> 
> Eventually I intend to import the whole history but not yet.
> 
> If you intend to go forward without fully solving this problem, your
> choices are not good.

Agreed.

> > Let's presume that this failure is not persistent and that D is able to
> > be imported successfully, we again have to make decisions about
> > parenting. I think it only makes sense for one of c or f to exist, based
> > upon what we decide is the right policy above.
> 
> This is particularly troublesome.  When you can generate the missing
> package, you will not be able to make it an ancestor of your existing
> history.
> 
> You could rewrite the existing history and then pseudomerge the old
> and new histories together, but that means duplicating most of the
> history metadata.
> 
> 
> If you wait with making an official import until it is suitable for
> use with dgit, then I might consider using the relevant parts of your
> imports history as the official Debian dgit history.
> 
> That would save you from the problem which I foresee in our future,
> where Debian and Ubuntu have disjoint imports of the same history.

Right, ideally, we'd be able (at some point in the future) perhaps stop
importing Debian manually (from Launchpad) and trust dgit or some other
source of information for the Debian history (with some translation
layer to go from a version string to a commitish in that source).

-Nish

-- 
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd