[Calendarserver-discuss] imip-agent 0.2 (iMIP capabilities for mail servers)

[Paul Boddie]

On Friday 27. January 2017 11.11.46 Jonas Smedegaard wrote:
> Quoting Paul Boddie (2017-01-26 16:19:42)
> > 
> > I'll take a look at that. The database functionality is supposed to be
> > optional, of course, so it makes sense to explore ways of decoupling
> > it from the basic functionality. One thing I've done for the
> > scheduling extensions is to have a tool that explicitly configures
> > them...
> > 
> > tools/update_scheduling_modules.py
> > 
> > ...which builds the manifest for those extensions:
> > 
> > imiptools/handlers/scheduling/manifest.py
> 
> That looks like a bad approach to me: I believe Debian (and any system
> following the FHS - File Hierarchy Standard) must treat /usr as
> write-protected except during package install or removal.

I've sort of avoided that myself by having the software installed in 
/var/lib/imip-agent, as you probably noticed, because I didn't regard the 
libraries as being more widely applicable. Thus, it was treated as an 
application. I'm not sure whether that's the right way in Debian, either. ;-)

But I accept what you're saying: mutable files in the software regions of an 
installation are a bad thing for various reasons.

> I can add a package install hook which generates that manifest below
> /var and symlink it, but that means the runtime package must depend on
> _development_ packages for Python.  I am not a Python programmer, but
> seems more elegant to me if your code would gracefully handle missing
> "plugin" code dynamically instead (or in addition - if you believe your
> approach has some use for deployments not caring about the FHS).

I'll admit that avoiding conditional imports is something I don't usually do, 
but they become something that a lot of Python programs do (usually for 
"cross-platform" reasons), and then the configuration and execution concerns 
get confused. I also have an aversion to programs that look around and see 
what modules are there, even though such things enable plug-in systems. 
Python's import system provides plenty of opportunities for things to get 
messed up, modules to conflict with each other, and so on.

But the point about needing python-dev or equivalent is a good one, although 
I'm not quite sure I understand why you would need the development package to 
run a Python script that generates a manifest. Could you spell it out to me?

> > This might seem inelegant when Python allows conditional importing and
> > all sorts of other tricks, but I wanted a separation between the
> > configuration of the software and what it does when it runs. When the
> > packaging system is supposed to take care of such things, you don't
> > then want the software failing because it tries to take on the same
> > job at run-time.
> > 
> > The quick fix here is to do a conditional import, but it does make me
> > think that I could provide a similar configuration mechanism for
> > storage extensions that also has a manifest file. New storage
> > extension packages would invoke a tool to update the manifest.
> 
> I would prefer a method that required python development tools available
> only at the build environment (not the target environment).
> 
> Is it relevant for speed, or just for coding style?  How do others in
> the Python community handle plugins?

It's mostly for clarity: you know what gets imported because there's a module 
that tells you. When people provide plug-in features they often allow the 
plug-ins to reside in various convenient places in the filesystem (some of 
them appearing later in various security advisories ;-) ), but I envisaged 
additional plug-ins being installed into the imiptools package in a controlled 
way.

There are mechanisms to have union packages, I think, meaning that plug-ins 
could be installed separately, although I imagine that the paths would need 
updating to alert Python to the presence of the different package locations. I 
don't know how much you use Python personally, but one of the complaints about 
Python is that it opens lots of files when starting up: the various features 
supporting some of the things I mention above is definitely one of the 
culprits here.

> >> Question: is imiptools/config.py meant to be only internal defaults,
> >> or user configurable?
> > 
> > It is supposed to be user-configurable and maybe the documentation
> > should be clearer about that. The tools/install.sh script copies it to
> > /etc/imip-agent (or equivalent) and then makes a symbolic link to it
> > from the normal library location.
> 
> You write more documentation than most of the projects that I package.
> Please don't take my question as an indicator that you ought to write
> even more: Sounds sensible for that detail to be implicitly handled by
> your install script.

Sure, but people might need to know about it anyway. :-)

> In fact, the reason I ask is that I noticed and wondered why I my
> packaging already does similar to how you describe that your install
> script does - most likely because when initially packaging I examined
> and mimicked your install script where sensible :-)
> 
> > I'm not sure whether that's the right thing to do in Debian, though.
> 
> It is unusual but not sure it is outright wrong in debian: It triggers a
> warning about scripts below /etc due to the hashbang of the file...

I would be interested to know about alternatives that work. My only previous 
experience with Debian packaging and things in /etc was for Kolab, where 
various PHP settings files lived in /etc/kolab (and maybe a directory for 
Roundcube as well).

Given that my configuration files are just Python syntax but need not be 
Python modules, they could actually be loaded and evaluated in a safe way, 
never being considered as executable either by the system or by Python. I 
might even have code lying around to perform this because it's something I did 
for another project of mine a few years ago.

Thanks once again for your advice on these matters!

Paul