Troubleshooting NVIDIA UFM HA Single-Node Recovery

When running NVIDIA Unified Fabric Manager (UFM) in a High Availability (HA) configuration powered by Pacemaker, Corosync, and DRBD, a single-node outage during planned maintenance can sometimes trigger unexpected dependency cascades.

This post walks through a real-world scenario where a secondary node was powered down for maintenance, causing the surviving primary node to block UFM startup due to STONITH fencing state, systemd service unit dependencies, and stale health flag files.

Situation Summary & Symptoms

During scheduled maintenance, ufm-node-01 was powered off. The expected behavior was for Pacemaker to maintain UFM services seamlessly on ufm-node-02. Instead, UFM failed to start due to several stacked issues:

#	Issue Identified	Operational Impact
1	STONITH/fencing resource misconfiguration	DRBD resource assignment blocked
2	Hard dependency on `openibd.service`	`ufm-enterprise.service` failed to start
3	Unit file syntax typo	systemd invalid argument warning
4	Missing `/dev/infiniband/umad0` device inside Docker	`opensm` failed inside container
5	Stale `failover.flag` indicator	`ufm-ha-watcher` repeatedly triggering failover

Step 1 — Clearing STONITH Fencing Constraints

Pacemaker’s fencing resources were configured with location constraints tied to the offline peer node. This caused Pacemaker to halt resource promotion:

pcs stonith status
pcs constraint location config

Fix

Temporarily clear fencing constraints while the peer node is offline:

pcs stonith delete fence_node1
pcs stonith delete fence_node2
pcs resource cleanup

Check DRBD status to confirm Primary role promotion:

drbdadm status

Expected output:

ha_data role:Primary
  disk:UpToDate
  peer connection:Connecting

Step 2 — Adjusting Systemd Unit File Dependencies

The ufm-enterprise.service unit file contained Requires=openibd.service. However, because kernel InfiniBand drivers (mlx5_ib, ib_umad) were already active, openibd.service returned a non-zero exit code on start, causing systemd to block ufm-enterprise.

Since the InfiniBand stack was already operational (ibstat showing LinkUp), we relaxed the dependency from Requires= to Wants=:

# /etc/systemd/system/ufm-enterprise.service
Wants=docker.socket openibd.service
Requires=docker.socket

Reload systemd daemon:

systemctl daemon-reload

Step 3 — Container Device Mapping & Subnet Manager (`opensm`)

opensm (InfiniBand Subnet Manager) running inside the UFM Docker container requires access to /dev/infiniband/umad0. If the container starts before the host device node registers, Docker does not dynamically bind the new character device.

Diagnosis & Fix

Verify device availability on host vs container:

# On host
ls -la /dev/infiniband/

# Inside container
docker exec ufm ls -la /dev/infiniband/

Restart ufm-enterprise once host drivers settle so Docker re-binds the device nodes:

systemctl restart ufm-enterprise

Step 4 — Clearing Stale Failover Flag

Even after opensm recovered, ufm-ha-watcher repeatedly triggered failover alerts because UFM’s internal health monitor had written a persistent flag file:

find /opt/ufm -iname "*failover*"
# Output: /opt/ufm/files/conf/failover.flag

Remove the stale flag to allow ufm-ha-watcher to resume normal operation:

rm -f /opt/ufm/files/conf/failover.flag
systemctl start ufm-ha-watcher

Summary of Key Takeaways

Service Dependencies: Use Wants= rather than Requires= for driver startup scripts like openibd when drivers are already initialized by the kernel.
Container Device Propagation: Containers initialized prior to InfiniBand character device creation (umad0) require a container restart to pick up host device nodes.
Health State Cleanup: UFM health monitors generate /opt/ufm/files/conf/failover.flag during outages; manual removal is required after resolving root causes.