[Debian-ha-maintainers] Split of HA agents into multiple binary packages

[Valentin Vidic]

On Mon, Mar 20, 2023 at 06:16:58PM -0300, Lucas Kanashiro wrote:
> I am Lucas and I am a Debian and Ubuntu developer. I have been taking care
> of the HA stack in Ubuntu for a while. Since I started to work on it I have
> noticed some issues with HA agents (a.k.a fence-agents and resource-agents),
> like missing dependencies and also installing some packages which might not
> be needed by the agent I want to use. I believe the way we ship those agents
> right now is not ideal for our users, and since we are approaching a new
> development cycle for Debian I decided to propose some changes here and see
> your thoughts.

Hi Lucas and thank you for the detailed proposal. I'm not sure I agree
with all the points, but let's try to discuss it in more detail.

> Once I identified this issue, the first thing I did was checking what
> Fedora/Red Hat was doing since they are also the upstream of those projects.
> There they have one binary package per agent and also some binary packages
> shipping related agents useful in specific scenarios. This seems to be the
> ideal scenario thinking about user experience, so one can install targeted
> packages with all the needed dependencies instead of pulling all the agents
> in a single package with non-optimal dependency management (what we have in
> Debian right now IMHO).

Yes, I agree the current setup for agents is a simplistic one, but it
has also quite flexible and has worked for many years. I don't remember
users opening bugs about this before, maybe there are requests for this
from Ubuntu users? As for RedHat, I think they have more resources to
work on this, so maybe a more complex setup is acceptable there.

In general, my thinking is that a production cluster setup is not something
you can do in an hour just installing a bunch of packages on a few
machines. It requires a lot of time for testing and tuning the setup.
Also since many machines are required, you probably need some form of
automation like Ansible to set it up and there is not much benefit from
helping these advanced users here as they would just spend more time
fighting the more rigid packaging setup.

> In Ubuntu, we started an initiative to curate some agents, which means that
> we have automated tests in place for them [1]. Those curated agents are now
> shipped in a binary package called {fence,resource}-agents-base, and all the
> non-curated agents are shipped in a binary package called
> {fence,resource}-agents-extra. This is just an intermediary step and we
> already know those changes have their own issues. We are still shipping
> multiple agents in a single binary package, this time we have all the needed
> dependencies in place, but the user experience is not that good because if a
> user wants a single agent in the curated set they will get some dependencies
> installed which are not needed for their use case.

I would think it will be very difficult to test and curate fence-agents
because most of them require special hardware and can't be automated
easily. Debian definitely does not have resources to do this.

Also, I don't see that resource-agents are split in the upstream spec
file and not sure how this would work at all? For example if there is
resource-agents-nginx and resource-agents-apache they will probably
fail to install at the same time because of the port conflict. Maybe
the proposal here is to remove resource-agents package and only go
with specific (more than 100) packages? Another approach would be
to have a resource-agents-nginx package that depends on resource-agents
and nginx. But the question again is, how do users find out this package
exists at all and is it really useful if you can do the same thing in
few lines of Ansible (or some other automation).

One advantage of the current resource-agents package is also that you
don't need to use the Debian package for e.g. nginx if you need a newer
version for some reason. With a strict dependency this is not possible
anymore.

> My proposal is to move to what the upstream of those projects are doing, and
> create a separate binary package for each agent, and the agent would be
> useful out-of-the-box (with everything needed installed). Since those
> projects are quite stable, without big changes, I believe it is OK to
> maintain them all as separate binary packages. I understand it will
> introduce many new binary packages in the archive but I really believe this
> will help our users' experience using the HA stack in Debian.

For fence-agents this introduces around 70 new packages. I don't think
this will help with discoverability and most users will just install
fence-agents that will pull in everything again. Not to mention that
pacemaker recommends fence-agents, again causing everything to be
installed. Also since the size of fence-agents is really small (250KB),
even with dependencies I don't expect more than few MB to be saved, so
the disk usage argument also does not hold.

Additionally, every time a new agent is introduced we need to go through
the NEW queue for approval, further complicating the package updates for
us. Same holds for agent removal since we need to file a bug to remove a
binary package from the pool.

> As a starting point I already have a salsa MR up for review for fence-agents
> [2]. If the maintainers agree on the approach I will do those changes in
> resource-agents as well. My idea would be to get those changes in
> experimental until the end of the freeze for the next release.

The change looks ok to me, but as I explained above, I see a lot of
complexity and downsides with this going forward and not that much
benefit. So for Debian I would choose a setup that is less complex and
easier to maintain, but still works for the majority of users.

Instead, what I would like to have more is automated testing with
autopkgtests, Ansible or some other tool that we could use to test the
whole stack in more complex scenarios. For example, I only recently
discovered through manual testing that ocfs2 does not work anymore.

-- 
Valentin