[DRE-maint] Proposed new requirements for emacsen add-on packages

[Agustin Martin]

On Wed, Jan 22, 2014 at 12:39:27PM -0600, Rob Browning wrote:
> Agustin Martin <agustin6martin at gmail.com> writes:
> 
> > Did not test in depth, but I think the add-on state files state files
> > could be recreated from /usr/lib/emacsen-common/packages/install in
> > case emacsen-common is installed for the first time (i.e., not
> > upgraded). Since emacs flavours depend on emacsen-common they should
> > byte compile things when configured.
> 
> I may be misunderstanding, but if not, I think the problem there is
> simultaneous installs (which are our primary complication in general),
> 
> i.e. most of the complexity we have comes from the fact that we're
> operating outside dpkg's dependency system (to avoid add-on deps), and
> so we have no ordering guarantees.
> 
> That's why I added the state/package/installed infrastructure, but I'm
> beginning to think that this approach may just be insufficient.  The
> original idea was that the state files would allow add-ons to signal
> that they're "ready to go" with respect to emacsen-common install/remove
> calls.
> 
> For example, consider the case where no emacsen-related packages are
> installed and someone runs this:
> 
>   apt-get install emacs24 add-on-1
> 
> Because there are no relevant dpkg deps, the packages' maintainer
> scripts can fire in any relative order, and if emacs24's postinst fires
> before add-on-1's, it can know to skip installing add-on-1 because
> package/installed/add-on-1 doesn't exist.  But emacs24 *will* create
> flavor/installed/emacs24, so that when add-on-1's postinst fires, its
> install script will be run for emacs24.  Of course, something reasonable
> should also happen if the order is reversed.
> 
> However, to demonstrate why I'm beginning to think the current approach
> may be unworkable, consider the case where no emacsen-related packages
> are installed and someone runs this:
> 
>   apt-get install emacs24 add-on-depending-on-add-on-1 add-on-1

Hi,

I may be wrong, but if add-on-depending-on-add-on-1 depends on add-on-1 dpkg
should care of configuring add-on-1 first. However, there is no warranty
about unpack ordering.

> I begin to suspect that up to now, we may have just been fairly lucky,
> and that the current approach is may just be broken for any number of
> cases like this.  And unfortunately, adding the emacsen-common
> dependency won't help.
> 
> Another reason why I keep wondering about the possibility of using
> triggers is that even if we addressed some of the above issues, the
> current approach will still probably result in redundant installs from
> time to time.  Even if you want to be conservative and expect to
> reinstall an add-on whenever it's upgraded, or any flavor is upgraded,
> consider the previous example, but assume that all of the packages are
> already installed (just being upgraded):
> 
>   apt-get install emacs24 add-on-depending-on-add-on-1 add-on-1
> 
> Currently, the emacs24 postinst will install both of the add-ons, and
> then each add-on will install itself, and add-on-depending-on-add-on-1
> will also reinstall add-on-1 again.  With a triggers-based approach
> you'd hope that we could coordinate better, and only run each install
> once.

Some add-ons do a check to avoid that, check that .elc for last
byte-compiled file is present. If so, it is already done, otherwise
byte-compile. But is way better if is emacsen-common who cares about that.

> So, generally speaking, unless we come up with alternatives, I'm
> beginning to wonder if triggers may be our best option, assuming we can
> shoehorn all add-ons into accommodating the attendant restrictions
> (whatever they end up being).

I think you are right, they may be the best option, but some ellaboration is
needed to properly evaluate that possibility, before going further into it.

In a quick proposal, I'd think about emacs-install and emacs-package-install
just touching action flags like

$flavour___$package

when run under dpkg control and not triggered and also enable the common
emacsen-common trigger. If run from the command line or from
dpkg-reconfigure (that is, not under dpkg control) they should do 
their usual current work and remove/update flags as needed in success.

When the trigger is run, some script runs emacs-package-install as
--triggered, old .elc files removed, byte-compilation done after flags
and flag removed on success.

I wrote some code to deal with aspell and ispell dictionaries automatic
hash rebuild from dictionaries-common triggers that may be useful here (perl
is also used).

> I'm not sure if I already mentioned it here, but I'm not opposed to
> triggers.  Though if we made a change that substantial, we probably
> *would* want to move to emacsen-common 3.*.

In package version yes, but there is no need to change compatibility level
unless maintainers need to do something in their packages to accomodate for
the new system.

> The problem I have with triggers to date is that I haven't yet been able
> to convince myself that they're flexible enough to handle all of our
> (potential) cases.  For example, with triggers, can we arrange it so
> that every add-on (or flavor) gets a chance to "remove" itself while
> still fully configured?  And if not, do we think that we can change
> policy in a way that will still accommodate the needs of all add-ons
> (i.e. would some kind of generic removal be sufficient in all cases)?

There may be some discrepancies between what dpkg considers as configured
and what emacsen-common considers as fully installed, not sure how to handle
this or even if its important enough as long as problems are properly
signalled.

> Another question -- assuming triggers run "later", can all all-on
> packages be adjusted to behave "sanely enough" in the window between
> when they're postinst fires, and when the triggers eventually run?

We already have a time gap between .elc removal in preinst and re-creation
during byte-compilation from postinst. If .elc files are removed in preinst
this would be larger, but the gap is already there. If we go for removal
just before rebuild we may end up in some corner cases with incompatible
byte-compiled files (I am unsure if this can trigger unpredictable behavior
and so should be avoided at any cost), but in the majority of cases, this
would not happen.

> > If something like this does not work adding the dependency is the
> > lesser evil as long as emacsen-common NEVER tries to pull emacsen nor
> > anything that should not be present in a very minimal system. But I
> > still prefer a fix into emacsen-common itself if possible.
> 
> Right -- if we go this way, I'd plan to keep emacsen-common very small,
> and as independent as possible.

Fine, that is what I expected. Note however that if depends is the way to
go, you will have to handle this depends with a lot of maintainers with very
different responsiveness maintaining add-ons with very different complexity
and userbase. This may become a hard work. That is why I'd suggest to try
fixing things from emacsen-common if possible.

> > Other interesting things were proposed in this thread like having
> > support for things like
> >
> >  elfiles=foo.el bar.el autoload.el
> >  . /usr/lib/emacsen-common/emacs-install-standard.sh
> >
> > but while highly desirable this is something different from the
> > original problem this thread is about.
> 
> Agreed, and there are a number of ways we might manage something like
> that, which might or might not involve the code going directly into
> emacsen-common -- i.e. we might want a separate binary package that the
> relevant add-ons could depend on.

If there is a way without explicit depends, they could just conflict with
older emacsen-common. Of course, if there is depends just use it.

Regards,

-- 
Agustin