[Debian-ha-maintainers] Bug#553166: redhat-cluster: services are not relocated when a node fails

[Martin Waite]

Package: redhat-cluster
Severity: important

*** Please type your report below this line ***

An unresolved upstream bug (which includes a patch) affects the
usability of the cluster suite on Debian Lenny.

https://bugzilla.redhat.com/show_bug.cgi?id=512512

This bug prevents the cluster from correctly handling the relocation
of services that were running on a failed node.
The problem is that fenced correctly handles the fencing of the failed
node, but is unable to correctly communicate
this to the cman module - which in turn fails to notify rgmanager that
the node has been fenced and any services
it was running are safe to be relocated to other nodes.

This bug makes the cluster suite on Debian Lenny unusable for
providing HA services.

A symptom of the notification failure can be seen in syslog when a node fails:

Oct 28 16:15:51 clusternode27 clurgmgrd[8602]: <debug> Membership Change Event
Oct 28 16:15:51 clusternode27 fenced[2760]: clusternode30 not a
cluster member after 0 sec post_fail_delay
Oct 28 16:15:51 clusternode27 fenced[2760]: fencing node "clusternode30"
Oct 28 16:15:51 clusternode27 fenced[2760]: can't get node number for
node p<C9>@#001
Oct 28 16:15:51 clusternode27 fenced[2760]: fence "clusternode30" success
Oct 28 16:15:56 clusternode27 clurgmgrd[8602]: <info> Waiting for node
#30 to be fenced

On line 4, "Can't get node number..." should report the name
"clusternode30", but the bug involves incorrectly
freeing the memory holding this name before using it in this message.
While fenced is satisfied that the
node has been successfully fenced, in the final line we can see that
the resource manager clurgmgrd
is still awaiting notification that the fencing has taken place.

This bug does not affect the RHEL5 version of the package, which is
based on a different source tree.
However, all distributions based on the STABLE2 branch are likely to
be affected.

The cluster used to expose the bug consists of 3 VM guests with the
following cluster configuration:

<?xml version="1.0"?>
<cluster name="testcluster" config_version="28">
<cman port="6809">
  <multicast addr="224.0.0.1"/>
</cman>

<fencedevices>
  <fencedevice agent="fence_ack_null" name="fan01"/>
</fencedevices>

<clusternodes>
<clusternode name="clusternode27" nodeid="27">
  <multicast addr="224.0.0.1" interface="eth0:1"/>
  <fence>
   <method name="1">
     <device name="fan01"/>
   </method>
  </fence>
</clusternode>
<clusternode name="clusternode28" nodeid="28">
  <multicast addr="224.0.0.1" interface="eth0:1"/>
  <fence>
   <method name="1">
     <device name="fan01"/>
   </method>
  </fence>
</clusternode>
<clusternode name="clusternode30" nodeid="30">
  <multicast addr="224.0.0.1" interface="eth0:1"/>
  <fence>
   <method name="1">
     <device name="fan01"/>
   </method>
  </fence>
</clusternode>
</clusternodes>

<rm log_level="7">
  <failoverdomains>
    <failoverdomain name="new_cluster_failover" nofailback="1"
ordered="0" restricted="1">
      <failoverdomainnode name="clusternode27" priority="1"/>
      <failoverdomainnode name="clusternode28" priority="1"/>
      <failoverdomainnode name="clusternode30" priority="1"/>
    </failoverdomain>
  </failoverdomains>
  <resources>
    <script name="sentinel" file="/bin/true"/>
  </resources>

  <service autostart="0" exclusive="0" name="SENTINEL" recovery="disable">
    <script ref="sentinel"/>
  </service>

</rm>

</cluster>

Where clusternode27, clusternode28, and clusternode30 are the 3 node names.

The only non-standard component is a dummy fencing agent installed in
/usr/sbin/fence_ack_null:
#!/bin/bash

#
# Fencing agent that always succeeds
#

echo Done
# eof

The cluster can now be started.  On each node:

sudo /etc/init.d/cman start
sudo /etc/init.d/rgmanager start

Once all nodes are running, start the SENTINEL service on clusternode30:

sudo /usr/sbin/clusvcadm -e SENTINEL -m clusternode30

On clusternode27, view the cluster status.  Service SENTINEL should be
in state "started" on clusternode30.

sudo /usr/sbin/clustat

At this point, tail syslog on clusternode27 and clusternode28.
Power-off clusternode30 (as rudely as possible).

On clusternode27, view the status of the cluster again:

sudo /usr/sbin/clustat

This will show that clusternode30 if "Offline", but that
service:SENTINEL is still in state "started" on clusternode30.

Again on clusternode27, view the node status:

sudo /usr/sbin/cman_tool -f nodes

Clusternode30 will show as status "X", with a note saying "Node has
not been fenced since it went down".


-- System Information:
Debian Release: 5.0.3
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26-2-amd64 (SMP w/1 CPU core)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash