Building a 3-Node OpenStack Cloud with Kolla-Ansible

This is the runbook I actually followed to stand up a live OpenStack cloud in the lab: three Dell R440s, every node doing control + compute + network, backed by external Ceph RBD. It runs OpenStack 2026.1 (Gazpacho) on kolla-ansible 22.1.0 with a single keepalived VIP fronting every API.

The goal was not a production cluster. It was a real cloud — one I could provision VMs on, run Rancher on top of, and break repeatedly — while keeping the operational model close enough to production that the lessons transfer.

Why Kolla-Ansible instead of Charmed OpenStack

This was the first decision, and it set the tone for everything after.

The deciding factor was maintainability, not elegance. Containerised services with Ansible playbooks mean a failure shows up in docker logs and a config change is a file edit plus a tagged re-run. Charmed OpenStack is a fine product, but when something breaks at 11pm I want the tool that has a decade of Stack Overflow answers behind it.

Kolla-Ansible 22.1.0 on Ubuntu 24.04 (noble) was chosen because both are explicitly in the support matrix.

Config gotcha: Kolla-Ansible loads variables from <configdir>/globals.d/*.yml. The live file is etc/globals.d/00-globals.yml — not etc/globals.yml. The latter is simply not read, and you will spend an hour wondering why your edits do nothing.

Topology

All three nodes run every role. No dedicated controller tier.

Cluster-wide: VIP 10.10.10.19 on br-ex, 56 / 51 / 57 containers per node, nine public endpoints all on the VIP.

Network layout

This is where most of the design effort went, because the interface layout decides your throughput ceiling.

The important shift: management traffic moved into the OVS bridge. The node IP now lives on br-ex, not on the physical NIC. A VIP that lands on eno1 instead of br-ex will look “up” in ip addr while every service is unreachable.

VM east-west traffic runs on the second 10G NIC via a provider VLAN. Keeping tenant traffic off the 1G management uplink is the difference between ~900 Mbit/s and ~9 Gbit/s between VMs.

Phase 1 — Prepare all three nodes

# /etc/hosts — hostnames must resolve on every node and on the deploy host
sudo tee -a /etc/hosts <<'EOF'
10.10.10.14 compute01
10.10.10.15 compute02
10.10.10.16 compute03
EOF

sudo apt update
sudo apt install -y python3-pip git docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker faiez
docker --version

The Ansible inventory uses ansible_user=faiez with ansible_become=true — not ubuntu — because these hosts were prepared with passwordless sudo for that account.

Phase 2 — Install Kolla-Ansible on the deploy host

sudo apt install -y python3-venv python3-pip git
python3 -m venv ~/kolla-ansible/venv
source ~/kolla-ansible/venv/bin/activate
pip install kolla-ansible==22.1.0
pip install 'ansible-core>=2.19,<2.20'
kolla-ansible install-deps
ansible-galaxy collection install -r venv/share/kolla-ansible/requirements.yml
ansible-galaxy collection install ansible.utils
sudo mkdir -p /etc/kolla && sudo chown $USER: /etc/kolla
cp -r ~/kolla-ansible/venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla/
cp ~/kolla-ansible/venv/share/kolla-ansible/ansible/inventory/multinode /etc/kolla/multinode

Pin ansible-core to 2.19, not the ansible 10.x meta-package. Kolla-Ansible 22.1.0 breaks on newer core modules and the failure looks like a random playbook error, not a version mismatch.

Phase 3 — Inventory

The stock multinode inventory has 190 groups. On a 3-node all-in-one, every service group contains all three nodes — only two groups are non-uniform:

[control]
compute01    ansible_host=10.10.10.14 ansible_user=faiez ansible_become=true ansible_python_interpreter=/usr/bin/python3
compute02    ansible_host=10.10.10.15 ansible_user=faiez ansible_become=true ansible_python_interpreter=/usr/bin/python3
compute03    ansible_host=10.10.10.16 ansible_user=faiez ansible_become=true ansible_python_interpreter=/usr/bin/python3

[loadbalancer]      # renamed from [network] since Xena
compute01
compute02
compute03

[compute]
compute01
compute02
compute03

[storage]
compute01
compute02
compute03

[monitoring]        # compute01 only
compute01

[deployment]
localhost ansible_connection=local ansible_python_interpreter=/home/faiez/kolla-ansible/venv/bin/python

[baremetal:children]
control
loadbalancer
compute
storage
monitoring

Two things that will bite you: [loadbalancer] (not [network] — that name died in Xena), and [kolla_toolbox] must be present, otherwise the MariaDB WSREP checks fail with a confusing “no hosts matched” error.

Phase 4 — globals

The parts that matter:

kolla_base_distro: "ubuntu"
kolla_install_type: "binary"
openstack_release: "2026.1"
workaround_ansible_issue_8743: true

# Network
network_interface: "br-ex"                    # management sits on the OVS bridge
neutron_external_interface: "eno1"            # bridge port for physnet1
neutron_bridge_name: "br-ex,br-vm60"          # physnet1, physnet2
migration_interface: "enp94s0f0np0"           # 10G — Nova live migration
kolla_internal_vip_address: "10.10.10.19"
enable_neutron_provider_networks: "yes"
nova_compute_virt_type: "kvm"

# Services
enable_horizon: "yes"
enable_cinder: "yes"
enable_cinder_backend_lvm: "yes"
cinder_volume_group: "cinder-volumes"
enable_octavia: "yes"
octavia_amp_image_tag: "2026.1"
octavia_network_type: "tenant"
enable_valkey: "yes"                          # required by the Octavia worker

# External Ceph RBD
cinder_backend_ceph: true
glance_backend_ceph: true
enable_nova_ceph: true
nova_backend: "rbd"
ceph_cinder_pool_name: "volumes"
ceph_glance_pool_name: "images"
ceph_nova_pool_name: "vms"

enable_valkey is not optional if you enable Octavia. Without it load balancers sit in PENDING_CREATE forever and the worker logs give you nothing useful.

Phase 5 — Deploy

cd ~/kolla-ansible
source venv/bin/activate
export PATH="$HOME/kolla-ansible/venv/bin:$PATH"   # subprocess ansible-playbook

kolla-ansible prechecks      -i inventory/multinode --configdir ~/kolla-ansible/etc
kolla-genpwd
kolla-ansible bootstrap-servers -i inventory/multinode --configdir ~/kolla-ansible/etc
kolla-ansible deploy         -i inventory/multinode --configdir ~/kolla-ansible/etc
kolla-ansible post-deploy    -i inventory/multinode --configdir ~/kolla-ansible/etc
source etc/admin-openrc.sh

-i and --configdir must come after the subcommand. Put them before and you get a usage error that mentions neither flag.

A full deploy takes ~40–60 minutes on a 1G uplink — almost all of it pulling container images. If it is interrupted, deploy-resume picks up where it stopped instead of starting over.

Phase 6 — Verify

openstack compute service list      # nova-scheduler / conductor / compute — all up
openstack network agent list        # OVS, DHCP, L3, Metadata — all up
openstack volume service list       # cinder rbd-1, rbd-rep1, lvm-1 — all up
openstack hypervisor list           # compute01-03 up
openstack endpoint list --interface public

Verified state on the live cluster:

The failures that actually taught me something

A clean runbook is a lie of omission. These are the ones that cost real time:

ansible-playbook: command not found mid-deploy. Kolla shells out to ansible-playbook as a subprocess. Activating the venv is not enough — the venv bin must be on PATH.

VIP up, nothing reachable. keepalived was managing the VIP on eno1 while the services listened on br-ex. ip addr shows the VIP either way. Always confirm the address is on the bridge, not the physical port.

Glance and Cinder RBD errors with a valid keyring. /etc/kolla/glance-api/ceph/ceph.conf was 0 bytes inside the container. Kolla’s templating can emit an empty file when the keyring lookup misses. The fix is to re-copy ceph.conf into /etc/kolla/<service>/ceph/ on every node and restart the containers — and then verify with wc -c inside the container, because the file on the host being correct proves nothing.

first_found cannot find keyring. Ansible’s keyring lookup runs on the deploy host, so the keyrings must be staged under etc/config/{glance,cinder,cinder-volume,cinder-backup,nova}/ — not only on the nodes.

VM east-west throughput stuck near 1 Gbit/s. Traffic was riding the 1G management NIC instead of the second 10G port on br-vm60. Worth checking with ovs-vsctl list-ports br-vm60 the moment VM-to-VM transfers feel slow.

What runs on it today

The cloud is not a demo — it hosts the downstream Kubernetes platform:

Volumes are Ceph-backed RBD through three volume types — ceph-rbd2 (replica 2), rep1 (replica 1), and local on LVM — so a VM can choose between durability and speed at boot time.

What I would do differently

The 1G management uplink is the design’s real constraint. It was fine for deployment, but every image pull and every Horizon session competes with control-plane traffic. On a rebuild I would put management on 10G too and keep the 1G strictly for out-of-band.

The all-in-one layout is also a deliberate compromise. It gave me a working cloud in a day, but there is no failure domain — losing one node takes a third of every control-plane quorum with it. That is acceptable for a lab and would be the first thing to change for production.