Rancher on 3-Node RKE2 Inside OpenStack: Eight Things That Broke

I stood up Rancher on a 3-node RKE2 cluster running as VMs inside my OpenStack lab. It worked in the end, but almost every obstacle looked different from the last while hiding behind one or two underlying causes. This is the list, in the order it hurt.

Environment: OpenStack 2026.1 (Kolla-Ansible 22.1.0) on 3 × Dell R440, external Ceph RBD on HDD OSDs. Rancher 2.15 on RKE2, three VMs spread across three compute hosts.

TL;DR

• Nearly every networking problem was one missing security-group rule: UDP 8472 (flannel VXLAN).
• RKE2 must run on the provider network (MTU 1500), not a VXLAN tenant network (MTU 1450). Double VXLAN doesn’t work.
• etcd belongs on a local disk, not on Ceph-on-HDD.
• Ceph and switch jumbo-frame settings need to account for header overhead.

The shape of the setup

# Flavor with an EPHEMERAL disk — that's the local disk for etcd
openstack flavor create --vcpus 4 --ram 8192 --disk 20 --ephemeral 20 rancher.ha

# Security group: RKE2 ports PLUS udp 8472
openstack security group create rancher-ha-sg
openstack security group rule create --protocol tcp --dst-port 22   --ingress rancher-ha-sg
openstack security group rule create --protocol tcp --dst-port 443  --ingress rancher-ha-sg
openstack security group rule create --protocol tcp --dst-port 6443 --ingress rancher-ha-sg
openstack security group rule create --protocol tcp --dst-port 9345 --ingress rancher-ha-sg
openstack security group rule create --protocol tcp --dst-port 2379:2380 --ingress rancher-ha-sg
openstack security group rule create --protocol tcp --dst-port 10250 --ingress rancher-ha-sg
openstack security group rule create --protocol udp --dst-port 8472 --ingress rancher-ha-sg   # CRITICAL
openstack security group rule create --protocol icmp --ingress rancher-ha-sg

F1 — UDP 8472 missing (root cause of almost everything)

Symptoms: Rancher webhook timeouts, 100% packet loss between pods on different nodes, machines failing to join, Rancher unable to start.

Cause: the security group had no rule for udp 8472, so flannel’s VXLAN traffic between nodes was dropped.

Evidence: ping 10.42.2.8 (a pod on another node) showed 100% loss; after adding the rule, 0% loss at ~1 ms.

Fix: openstack security group rule create --protocol udp --dst-port 8472 --ingress rancher-ha-sg

F2 — Double VXLAN: RKE2 on a tenant network is broken

Symptom: cross-node pod traffic dead even with the security group fixed.

Cause: the tenant network is VXLAN with MTU 1450. RKE2’s flannel VXLAN on top of that is a second layer of encapsulation, and the MTU budget runs out.

Fix: put RKE2 on the provider network (flat, MTU 1500) by setting node-ip to the provider address. flannel’s 1400 MTU then fits comfortably.

F3 — etcd on Ceph-on-HDD is too slow

Symptom: etcd logging apply request took too long (2–4 s), apiserver restarts, the cluster stuck “activating”.

Cause: etcd fsyncs every write, and an fsync on Ceph RBD backed by HDD plus VXLAN has high latency, which triggers Raft election timeouts.

Fix: keep etcd on the local ephemeral disk. Boot time also dropped from 105 s to 30 s.

F4 — containerd and etcd fighting over one disk

Symptom: etcd timeouts even on local disk.

Cause: my first attempt put all of /var/lib/rancher/rke2 on the 20 GB local disk. containerd images (~3.9 GB) plus etcd filled it and contended for I/O.

Fix: mount the local disk only at server/db (etcd). containerd and Rancher data stay on the Ceph-backed root.

F5 — The ephemeral disk arrives as vfat

Symptom: containerd: overlayfs snapshotter cannot be enabled: invalid argument.

Cause: Nova hands over the ephemeral disk pre-formatted as vfat (label ephemeral0). overlayfs can’t run on vfat.

Fix: sudo mkfs.xfs -f /dev/vdb, then mount it. Use the real UUID from blkid in /etc/fstab.

F6 — rm -rf is not a reset

Symptom: after resetting a node, bootstrap hung with Failed to test etcd connection: context deadline exceeded.

Cause: I’d cleared the directory, but the underlying disk still held 134 MB of old etcd state that etcd happily read back.

Fix: re-format the disk on every reset (mkfs.xfs), don’t just delete files.

F7 — kube-proxy not synced on a promoted node

Symptom: cert-manager pods on the third node: dial tcp 10.43.0.1:443: i/o timeout.

Cause: that node had been promoted from agent to control-plane, and its kube-proxy was missing the iptables rule for the kubernetes Service.

Fix: delete its kube-proxy pod and let it recreate and resync.

F8 — Namespace stuck Terminating after uninstall

Symptom: cattle-system hung in Terminating after helm uninstall.

Cause: the rancher-webhook, running inside that namespace, blocked deletion through its own validating and mutating webhook configurations — a deadlock.

Fix:

kubectl delete validatingwebhookconfiguration rancher.cattle.io
kubectl delete mutatingwebhookconfiguration rancher.cattle.io
kubectl replace --raw "/api/v1/namespaces/cattle-system/finalize" -f - <<EOF
{"kind":"Namespace","apiVersion":"v1","metadata":{"name":"cattle-system"},"spec":{"finalizers":[]}}
EOF

(I wrote up the related stuck-cluster case in Unblocking a Rancher Cluster Stuck in ‘Deleting’.)

Installing Rancher on HDD-backed storage

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.3/cert-manager.yaml

helm install rancher rancher-latest/rancher --namespace cattle-system --create-namespace \
  --set hostname=rancher.example.lan \
  --set bootstrapPassword=<choose-a-real-one> --set replicas=1 \
  --set auditLog.enabled=false \
  --set "extraEnv[0].name=CATTLE_FEATURES" \
  --set "extraEnv[0].value=embedded-cluster-api=false,fleet=false" \
  --timeout 10m

One more thing that’s required on HDD: Rancher takes longer than the default startup probe allows (60 s). Patch the probe, or it gets killed mid-start:

kubectl patch deploy rancher -n cattle-system --type=json -p='[{"op":"replace",
 "path":"/spec/template/spec/containers/0/startupProbe",
 "value":{"httpGet":{"path":"/healthz","port":80},"failureThreshold":60,"periodSeconds":5,"timeoutSeconds":5}}]'

Ceph tuning that helped

The jumbo-frame trap. My switch’s system mtu 9000 limits the L2 frame, not the IP MTU. A host MTU of 9000 produces a 9014-byte frame, which the switch silently drops. Set the host to 8950 (frame 8964, under 9000) and verify with ping -M do -s 8922.

Pitfalls checklist

1. UDP 8472 is mandatory for any RKE2 or K3s cluster on OpenStack.
2. If you need a floating IP’s address on a provider network, release the floating IP and create a provider port with the same fixed IP.
3. Anti-affinity: confirm the three VMs really landed on different hosts.
4. Format the etcd disk on every reset.
5. --hint group= needs a server-group UUID; use --server-group <name> for names.
6. Security groups are project-scoped. The group must be in the same project as the server.
7. Don’t run concurrent helm operations; the release lock will bite you.

Still open when I wrote this

The downstream cluster’s machines looped between Provisioning and Deleting. I’d installed Rancher with embedded-cluster-api=false to avoid a flood of CRDs, which meant the OpenStack CAPI provider was never installed, so OpenstackMachine objects had no controller. The credential and machine template were correct; the planned fix was to enable the embedded cluster API and restart. I hadn’t confirmed that when this was written.