[Calendarserver-discuss] imip-agent 0.2 (iMIP capabilities for mail servers)

Fri Jan 27 15:36:43 UTC 2017

Quoting Paul Boddie (2017-01-27 13:25:38)
> On Friday 27. January 2017 11.11.46 Jonas Smedegaard wrote:
> > Quoting Paul Boddie (2017-01-26 16:19:42)
> > > 
> > > I'll take a look at that. The database functionality is supposed 
> > > to be optional, of course, so it makes sense to explore ways of 
> > > decoupling it from the basic functionality. One thing I've done 
> > > for the scheduling extensions is to have a tool that explicitly 
> > > configures them...
> > > 
> > > tools/update_scheduling_modules.py
> > > 
> > > ...which builds the manifest for those extensions:
> > > 
> > > imiptools/handlers/scheduling/manifest.py
> > 
> > That looks like a bad approach to me: I believe Debian (and any 
> > system following the FHS - File Hierarchy Standard) must treat /usr 
> > as write-protected except during package install or removal.
> 
> I've sort of avoided that myself by having the software installed in 
> /var/lib/imip-agent, as you probably noticed, because I didn't regard 
> the libraries as being more widely applicable. Thus, it was treated as 
> an application. I'm not sure whether that's the right way in Debian, 
> either. ;-)

Oh - I might have noticed when packaging initially, but then happily 
forgot again since :-P

Debian supports "private modules" below /usr but outside default lookup 
paths - but same applies there.

Debian possibly also supports dynamically generated/adapted python code 
below /var, but I fear I then need to implement and maintain custom code 
to ensure compliance with e.g. Debian Pyhon Policy §3.7: 
https://www.debian.org/doc/packaging-manuals/python-policy/ch-module_packages.html#s-byte_compilation

> But I accept what you're saying: mutable files in the software regions 
> of an installation are a bad thing for various reasons.

Sounds like that's exactly my concern (just not fluent in the terms).

> > I can add a package install hook which generates that manifest below 
> > /var and symlink it, but that means the runtime package must depend 
> > on _development_ packages for Python.  I am not a Python programmer, 
> > but seems more elegant to me if your code would gracefully handle 
> > missing "plugin" code dynamically instead (or in addition - if you 
> > believe your approach has some use for deployments not caring about 
> > the FHS).
> 
> I'll admit that avoiding conditional imports is something I don't 
> usually do, but they become something that a lot of Python programs do 
> (usually for "cross-platform" reasons), and then the configuration and 
> execution concerns get confused. I also have an aversion to programs 
> that look around and see what modules are there, even though such 
> things enable plug-in systems. Python's import system provides plenty 
> of opportunities for things to get messed up, modules to conflict with 
> each other, and so on.
> 
> But the point about needing python-dev or equivalent is a good one, 
> although I'm not quite sure I understand why you would need the 
> development package to run a Python script that generates a manifest. 
> Could you spell it out to me?

Uhm, seems I confused stuff: For pure Python code only the Python 
interpreter is needed.

What distracted me was a) the Perl-based spamassassin optionally doing 
byte-compiling to /var - but since perl byte-compilation is unusual, no 
Policy has evolved governing routines around that.  And b) Debian Python 
Policy mandating byte-compilation during install - which requires python 
development tools when the module links with C.

> > > This might seem inelegant when Python allows conditional importing 
> > > and all sorts of other tricks, but I wanted a separation between 
> > > the configuration of the software and what it does when it runs. 
> > > When the packaging system is supposed to take care of such things, 
> > > you don't then want the software failing because it tries to take 
> > > on the same job at run-time.
> > > 
> > > The quick fix here is to do a conditional import, but it does make 
> > > me think that I could provide a similar configuration mechanism 
> > > for storage extensions that also has a manifest file. New storage 
> > > extension packages would invoke a tool to update the manifest.
> > 
> > I would prefer a method that required python development tools 
> > available only at the build environment (not the target 
> > environment).
> > 
> > Is it relevant for speed, or just for coding style?  How do others 
> > in the Python community handle plugins?
> 
> It's mostly for clarity: you know what gets imported because there's a 
> module that tells you. When people provide plug-in features they often 
> allow the plug-ins to reside in various convenient places in the 
> filesystem (some of them appearing later in various security 
> advisories ;-) ), but I envisaged additional plug-ins being installed 
> into the imiptools package in a controlled way.
> 
> There are mechanisms to have union packages, I think, meaning that 
> plug-ins could be installed separately, although I imagine that the 
> paths would need updating to alert Python to the presence of the 
> different package locations. I don't know how much you use Python 
> personally, but one of the complaints about Python is that it opens 
> lots of files when starting up: the various features supporting some 
> of the things I mention above is definitely one of the culprits here.

Regarding how much I use Python: I used it as much as it is imposed on 
me - if I had a choice (read: If you could convince you and other coders 
to switch) then I would uninstall Python from my system, instead using 
Perl (and look forward to Perl6 with its native byte-compilation 
maturing). ;-)

> > [Python modules symlinked from /etc] is unusual but not sure it is 
> > outright wrong in debian: It triggers a warning about scripts below 
> > /etc due to the hashbang of the file...
> 
> I would be interested to know about alternatives that work. My only 
> previous experience with Debian packaging and things in /etc was for 
> Kolab, where various PHP settings files lived in /etc/kolab (and maybe 
> a directory for Roundcube as well).

Please - I am allergic to PHP... ;-)

> Given that my configuration files are just Python syntax but need not 
> be Python modules, they could actually be loaded and evaluated in a 
> safe way, never being considered as executable either by the system or 
> by Python. I might even have code lying around to perform this because 
> it's something I did for another project of mine a few years ago.

Perhaps look at how Radicale handles plugins: It seems to me that it 
supports _either_ probing (touching multiple files - although here with 
only "files" and "SQL" I cannot imagine it being very expensive) or 
explicitly declared choice.

 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: signature
URL: <http://lists.alioth.debian.org/pipermail/calendarserver-discuss/attachments/20170127/ac4a462f/attachment.sig>