Provisioning RKE2 from Rancher? Prepare OpenStack First

Rancher’s OpenStack node driver will happily generate a cluster that never boots, and the errors land in Rancher — not in OpenStack. By the time you are reading machine provision logs, the actual fault is usually a missing object or a wrong default several layers below.

This is the pre-flight checklist I now run before every provisioning attempt, and the four traps that taught me to run it.

The mental model

Rancher’s driver is not an OpenStack abstraction — it is a thin wrapper that creates standard OpenStack resources. Every failure maps to exactly one missing or misconfigured object:

If a machine sits in BUILD or the provision Job fails, work down this table. The answer is almost always in Neutron or Nova.

Trap 1 — The 3.5 GB root disk

The only flavor in the lab was rancher.ha (4 vCPU / 8 GB RAM / 20 GB ephemeral), with disk = 0.

disk = 0 does not mean “no disk”. It means Nova sizes the root disk from the image’s virtual size. The image ubuntu-24.04-cloud has virtual_size = 3.50 GiB.

So a node provisioned this way gets a 3.5 GiB root filesystem plus an unmounted 20 GB ephemeral device at /dev/vdb. RKE2 needs roughly 8–10 GiB for /var/lib/rancher/rke2 — containerd plus control-plane images. The install fills the root disk and the node dies mid-bootstrap.

Two fixes:

The definitive test is after the first node boots:

df -h /            # expect tens of GB, not ~3 GB
lsblk              # note the unmounted ephemeral device

Pre-flight:

openstack flavor list -f value -c Name -c VCPUs -c RAM -c Disk -c "Ephemeral"
openstack image show ubuntu-24.04-cloud -c virtual_size -c min_disk -f value

Trap 2 — Port 9345 is not optional

The security group allowed UDP 8472 from the old tenant network, not the provider network the new nodes used. And it omitted 9345 entirely.

9345 is the RKE2 supervisor port — the join endpoint. Without it, nodes register their existence and never join. The failure in Rancher reads like an agent problem, which sends you looking in the wrong place.

Minimum rule set for a working pool:

The security group must live in the same project as the instances. A group from another project is not visible to the driver, and the failure surfaces as SecurityGroupNotFound on every host — which reads like a Nova bug.

Trap 3 — A Neutron agent that is not heartbeating

openstack network agent list showed 11 of 12 agents alive: the L3 agent on compute03 was dead.

A container that looks healthy can still have an agent that stopped heartbeating. An agent that is not heartbeating cannot program the dataplane — ports on that host fail to bind, instances sit in BUILD with no network, and the driver loop starts deleting and recreating them. From Rancher’s side this looks like “machines keep failing and VM names churn”.

openstack network agent list -c "Agent Type" -c Host -c Alive -c State -f value
openstack network agent list -c Alive -f value | grep -c True     # expect the full count

# fix a downed agent
for h in 14 15 16; do
  ssh faiez@10.10.10.$h 'sudo docker restart neutron_l3_agent neutron_openvswitch_agent'
done
# re-check after ~90 s

Sixty to ninety seconds of heartbeats is the real test. A container in Up state proves nothing.

Trap 4 — Cloud-init with no CATTLE_SERVER

An empty server-url setting produces cloud-init that has no CATTLE_SERVER variable. rancher-system-agent aborts with:

[FATAL] $CATTLE_SERVER was not set

The machine provision Job fails, the plan is never applied, and — as covered in the stuck-cluster post — deleting that machine then deadlocks on a finalizer that is only released by the job’s success path.

kubectl get setting server-url -o jsonpath='{.value}'
kubectl get setting cacerts    -o jsonpath='{.value}'

Both must be non-empty before any provisioning run. cacerts matters too: agents that cannot trust Rancher’s certificate fail in exactly the same way.

Pre-flight checklist

Run this before every provisioning attempt. It takes two minutes and saves a provision-failure debug cycle.

source ~/kolla-ansible/venv/activate 2>/dev/null; source etc/admin-openrc.sh

# identity and quota
openstack token issue -c expires -f value
openstack quota show <project> -c volumes -c gigabytes -c instances -c cores -c ram

# compute
openstack compute service list -c Binary -c Host -c State -c Status -f value
openstack availability zone list --compute
openstack flavor show rancher.ha -f json | grep -E 'disk|ephemeral'

# image
openstack image show ubuntu-24.04-cloud -c virtual_size -c min_disk -c visibility -f value

# network — every one of these must exist and be healthy
openstack network show <tenant-net> -c id -c subnets -f value
openstack subnet show <subnet> -c cidr -c gateway_ip -c dhcp_enable -f value
openstack router show <router> -c external_gateway_info -f json
openstack network show ext-net -c router:external -c provider:network_type -f value
openstack security group rule list <sg> -c Protocol -c "Port Range" -c "IP Range" -f value
openstack network agent list -c "Agent Type" -c Host -c Alive -f value

What I changed in my own workflow

The provision form has around twenty fields, and the two that matter most are not the ones you would guess: bootFromVolume + volumeSize (because of the root-disk default) and configDrive (mandatory in this environment, because there is no DHCP on the provider network and the driver relies on the config drive for network metadata).

Everything else can be derived from the checklist. Which means the whole pre-flight can be scripted — and once it is scripted, a provisioning failure stops being a Rancher mystery and becomes a one-line answer from the check that failed.