[Pkg-crosswire-devel] Module packaging, distribution, and management

[Matthew Talbert]

> Eeli Kaikkonen wrote:
>
>> http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA
>>
>> It's possible that the Sword library has a bug in this respect. However,
>> it doesn't mean that the package manager is the correct way to handle
>> the modules, except in some special cases.
>
> Thanks for this link.  Now we have an official reference document to use
> as a basis for our discussion.
>
> It says in part:
>
>     Any program or package which contains or requires data that doesn't
>     need to be modified should store that data in /usr/share (or
>     /usr/local/share, if installed locally). It is recommended that a
>     subdirectory be used in /usr/share for this purpose.
>
> Is the libsword package "a program or package which contains or requires
> data that doesn't need to be modified"?  I would say yes, it is.  (It
> may well contain or require other kinds of data that *does* require
> modification, which would need to be installed in a user-writeable
> location, of course!).
>
> Are the sword-text-* packages "a program or package which contains or
> requires data that doesn't need to be modified"?  I would say yes, they are.

Unfortunately the matter of indices rises again here. I just want to
explain how these work without defending it, because I don't entirely
agree myself. Indices are stored in the same directory as the module
in a directory called "lucene". Because these indices are created by
the user typically, this causes a fundamental incompatibility with
installing the modules in /usr/share/sword.

> Therefore, libsword and its related data packages "should" store that
> kind of data in a subdirectory of /usr/share/.  Indeed, the sword
> tarball docs are 100% consistent with this, recommending use of
> /usr/share/sword/ for module data storage.  So far, so good --
> everything is in line with the standard.
>
> It seems, then, that if a "module manager" is incompatible with this
> standard and recommended approach, or how it presents things to the user
> causes them difficulties that lead to support requests, then that module
> manager has an issue that probably should be addressed.  The primary
> disagreement is apparently over how to address this issue.

Please note again here that the vast majority of the support requests
have been because of the incorrectly packaged dependencies (installing
ArabSV, etc) which we all agree is wrong and is going to be taken care
of. In addition, forcing the user to use the package manager to
install a module is also going to cause problems, but I think we're
all in agreement to use "Suggests" which would take care of that.

> Possibility #1:
>
> (1) Maybe the module manger UI(s) currently used do not make clear to
> end users which of the modules they can see are in shared, read-only
> locations and which are in per-user, read/write locations?  Making that
> clearer could help reduce user confusion (users would be less likely to
> try and remove a module using the Remove button (or whatever) if that
> button were greyed out when a module in a read-only location is selected
> (for example), or (better) it pops up a message that explains that this
> module was installed by a system admin and is read-only, so it should be
> uninstalled (or updated, or whatever) the same way it was installed.

GnomeSword does indeed show the user that he can't *install* to a
write-protected area. Unfortunately, it isn't as smart about removing.
This is indeed something that should be fixed. In addition, the
indices are again a problem.

> If the module managers need API support from libsword to do this, which
> they probably do, then of course this is firstly a libsword development
> issue.  If not, then it is a front-end development issue.  Doing this
> adds utility on systems where there is no package management but there
> are read-only network shared module data locations, for example -- this
> kind of enhancement is clearly not only of benefit to Debian/Ubuntu.

Two things should, imo, be done to libsword to help with this. First
of all, it should allow creating indices somewhere other than the
module location. This would help tremendously. The second thing (which
has already been proposed) is for it to give a clear failure message
when trying to uninstall from a write-protected area. These two things
would solve a lot of the issues here.

> Most likely, all that would be needed is a way to ask any given module
> (or maybe, any given module data area?) "Are you writeable by me?".
> That should be a very minimal API change, I would think.  It is just a
> single bit of info about a module the API can provide to the calling
> application.  There is no package management specific work here at all.
>
> On top of this, we have package managers; many current operating system
> distributions use some form of package management system to manage files
> in system space, of course some implement them more consistently that
> others or more thoroughly that others.  While Possibility #1 improves
> things somewhat for package-managed modules, it is still a less than
> complete solution.  This leads to:
>
> Possibility #2:
>
> If, in addition to #1, a "module manager" were able to recognize which
> module storage areas were under package management control (whether the
> manager is apt or RPM or PBI or some other management setup!) and (at
> minimum) let the end user know, clearly, whenever they try to perform a
> module management action that "undermines" the package management
> system, then users would be even better informed, and so enabled to do
> the right thing (use a package management tool to manage files that are
> under package management control, basically!).  This is a pretty low
> level of package manager integration, but it adds considerably to module
> manager end user understanding of what is going on.
>
> At the API level this probably just needs #1, plus a "is this module
> under system-wide package management control?" call (returning an
> identifier for the package management system involved, and perhaps the
> package name concerned, might a nice little enhancement, allowing for
> clearer and more specific feedback to the end user -- but is not 100%
> required).

This is probably quite unlikely as it is a rather platform-specific
approach. Norbert's recommendation seemed quite a bit more reasonable.

> Possibility #3:
>
> A step up from #2 would be for the "module manager" to actually
> integrate more fully with the system package management tools, and so
> seamlessly allow end users to manage such packaged modules (prompting
> for enhanced perms as needed, of course).
>
> This is, of course, quite a lot of work!

Even more so.

> Possibility #4 (sort of):
>
> An alternative would be for a "module manager" in front ends to totally
> ignore all shared read-only modules, which most likely would include all
> package-managed modules; this is very imperfect indeed, but at least the
> two systems do not both try to manage the same set of data this way.
>
> It feels ugly, but I think this needs relatively little implementation
> work upstream, and no API changes at all (module managers need no
> knowledge whatever of package management systems).  We could even patch
> this approach into the packaged GnomeSword and BibleTime at package
> build time ourselves, and even into installmgr -- but that's a
> divergence from upstream behaviour we should strive to avoid.

Please don't do this. There is already an unfortunate discussion on
gnomesword-devel where some have alleged you are going to disable the
module manager.

>>> However I also believe that from the admin's perspective, the various
>>> programs should adhere to well-established principles like "the
>>> program is not fundamentally incompatible with my distro's package
>>> management system", with as few app-specific quirks as possible.
>
> Amen!
>
>> You have good points, but still I think the main problem has been that a
>> random module has been installed without asking the user and it wasn't
>> possible to uninstall it easily (because the user didn't know how it was
>> installed).
>
> Right; this aspect of the issue would be fully addressed by either of
> Possibilities #2 and #3 above, and very partially addressed by #1 and #4.
>
> Suggestion: #2 sounds like a doable enhancement to the module manager(s)
> to me; make the module manager aware of system package management, but
> not actually able to work with such package management.  Is an
> enhancement request to that effect likely to be acted on by the upstream
> (CrossWire) development team?
>
>> If the modules are only suggested everything works fine:
>
> Only up to a point; on a multiuser system or a networked setup with a
> shared readonly module data location, if the module manager as run by an
> end user remains blissfully unaware of the fact that some modules are in
> read-only space,  users will try to "manage" these modules, because they
> look just like all other personal-only modules in their module manager
> -- then the user will be confused at the errors which they will get, I
> suspect.
>
>> if someone installs a module with the package manager he knows how to
>> uninstall it properly.
>
> Hopefully!  But his users may well not, if the UI they see presented by
> a "module manager" in the app they use does not clearly distinguish
> these modules from others a user may have installed personally and
> per-user.  Hence the need for (at least) Possibility #1 above, IMO.
>
> Finally (for now): do the *.conf files for current modules provide an
> indicator of whether a module will work if installed on a read-only
> module location?  If not, should they perhaps include such info in
> future?  Something like NeedsUserWriteableStorage=1 maybe?
>

This has actually been proposed, and may get implemented as we are
moving towards enabling a lot more user-editable content. However
there is nothing in place now. Note again, that *all* modules need to
be writable for indices to be created.

It seems that one of the biggest issues is the index situation. Feel
free to complain about this all you want, but the reality is that we
need to deal with this some way for now. I would like to second
Norbert's suggestion that we remove the modules for this release while
we work out some solutions. A patch to libsword that would allow
symlinking as he suggested, plus allow indices in a different location
would have a better chance of getting accepted into libsword than
questions and requests on the mailing lists.

Matthew