A Power Cut at the Rack: An OpenStack Start/Stop Runbook Built from the Aftermath

An unplanned power cut took my OpenStack lab down. When power returned, everything powered on at once, and the cloud came back inconsistent: services reported healthy while VMs couldn’t be started or couldn’t send traffic.

This post is the failure list and the ordered shutdown/startup procedure I wrote afterwards.

Treat this as a draft. The commands assume a Kolla-Ansible deployment (containers named nova_compute, neutron_openvswitch_agent, …), which I inferred from the restart commands I used during the incident. Container names, interface names and inventory paths marked <...> are placeholders. Confirm them against your environment and rehearse in a maintenance window before relying on any of it.

What actually broke

The lesson hiding in the table: “service is up” is not the same as “service works”. Two of the failures (queue consumers, port tags) showed no sign in service list or agent list.

Before shutdown

1. Announce the maintenance window to tenants.

2. Confirm a recent backup of the OpenStack database and any critical VM data.

3. Snapshot the current state so you can diff after startup:

   openstack server list --all-projects --long -f csv > /root/pre-shutdown-servers.csv
   openstack compute service list  > /root/pre-shutdown-compute.txt
   openstack network agent list    > /root/pre-shutdown-agents.txt
   openstack volume service list   > /root/pre-shutdown-volumes.txt
   

4. Confirm the cluster is healthy first. Don’t shut down a degraded cluster without noting it.

5. Confirm out-of-band access (iDRAC/iLO/IPMI) works on every node. One of mine needed a manual disk boot, so console access matters.

Graceful shutdown

Reverse order of dependencies: tenants, then compute, then control plane, then storage, then power.

1. Stop scheduling and tenant workloads

for h in compute01 compute02 compute03; do
  openstack compute service set --disable --disable-reason "planned shutdown" $h nova-compute
done
openstack server list --all-projects --status ACTIVE -f value -c ID | xargs -n1 openstack server stop

Wait until nothing is ACTIVE; anything left running is hard-stopped at power-off.

2. Storage pre-steps. If the backend is Ceph, stop it rebalancing while nodes are down. (Confirm this applies to your backend first.)

ceph osd set noout
ceph osd set norecover
ceph osd set norebalance

3. Compute nodes, one at a time:

docker stop nova_compute nova_libvirt neutron_openvswitch_agent
sudo systemctl poweroff

Confirm each is fully off via the console or BMC before continuing.

4. Controllers. Stop services in the order your deployment tool supports (kolla-ansible -i <inventory> stop; check flags for your version), then power off one controller at a time. If the database is clustered (Galera), make the node with the latest state the last one stopped.

5. Storage nodes last, once nothing consumes them. 6. Switches and other infrastructure only after every node is confirmed off.

Startup

Power on in dependency order and verify each tier before the next. Don’t power everything on at once — that is exactly what produced the inconsistent state.

1. Network. Power on switches, wait until fully up, verify uplinks.

2. Storage. Power on, wait for the service to report healthy, and check log and disk behaviour immediately (a log flood on a shared disk caused the slowdown). If you set Ceph flags earlier:

ceph osd unset noout; ceph osd unset norecover; ceph osd unset norebalance

3. Controllers. Start the node with the latest database state first if clustered, then:

docker ps -a --format 'table {{.Names}}\t{{.Status}}'
docker exec mariadb mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
docker exec rabbitmq rabbitmqctl cluster_status
openstack token issue          # does the API answer?

4. Compute nodes — one at a time, verifying each.

a. Boot. If it doesn’t boot unattended, boot the disk from the console and fix the boot order afterwards.

b. Check NIC link speed before anything else:

ethtool <nic> | grep -E 'Speed|Duplex|Auto-negotiation|Link detected'

If the link is up but traffic fails, pin the speed, then make it persistent in your network config (netplan, NetworkManager or udev) and test that it survives a reboot:

sudo ethtool -s <nic> speed 10000 duplex full autoneg off

c. Confirm containers are running:

docker ps --format 'table {{.Names}}\t{{.Status}}' | grep -E 'nova|neutron|openvswitch|libvirt'

d. Verify nova-compute is actually consuming its queue. “Up” is not enough.

docker exec rabbitmq rabbitmqctl list_queues name messages consumers | grep compute.

Each compute.<hostname> queue should have consumers ≥ 1 and messages near 0. If not: docker restart nova_compute on that node.

e. Verify OVS port tags, then restart the OVS agent if any are missing:

docker exec openvswitch_vswitchd ovs-vsctl --columns=name,tag list port | paste - - - | grep -E 'tap|qvo'

VM ports (tap* / qvo*) need a numeric tag. If empty, restart nova_compute first, then neutron_openvswitch_agent, and re-check. (The OVS container name varies by deployment; in Kolla I believe it’s openvswitch_vswitchd, but check docker ps.)

f. Re-enable the host: openstack compute service set --enable <hostname> nova-compute

5. Tenant workloads. Start in batches and watch for errors:

openstack server list --all-projects --status SHUTOFF -f value -c ID | xargs -n1 openstack server start

Post-startup validation

Test on every compute node, not just one. This incident produced partial failures — inbound fine, outbound dead — that a single-node test would have missed.

Checking for a log flood

iostat -x 5 3
journalctl --disk-usage
du -sh /var/log/* | sort -h | tail

If the disk is busy, find the top writer (iotop -oPa) before restarting anything. The fixes that stuck: drop the noisy lines at the logging layer (everything else still logged), cap log file sizes, and hold the journal in memory (Storage=volatile in journald.conf; verify your current setting).

Follow-ups

• Make the fixed NIC speed persistent and test across a reboot.
• Fix the boot order so every node boots unattended.
• Alert on: RabbitMQ queues with zero consumers or growing backlog; OVS ports with no tag; sustained disk utilisation on the Kubernetes database disk.
• Add a UPS and a staggered power-on delay so nodes don’t all start together.
• Review the electrical load on the circuit that failed.
• Rehearse this runbook in a test window and replace the placeholders.