Stress-Testing Kubernetes on OpenStack: It Was the Storage All Along

Before using a staging cluster as the blueprint for production, I wanted to know what it could actually carry. So I ran a full stress-test suite against it: CPU, memory, disk, network, control plane, pod density, and finally a combined soak.

Seven of the eight load phases produced usable numbers. The eighth I had to abort, because it took the Rancher management cluster down for about 75 minutes. That incident turned out to be the most valuable result of the day.

The headline: this cluster is storage-bound, not CPU- or network-bound. Every hard wall traced back to one place — a single-node Ceph cluster on two spinning disks, shared by every pool.

What was tested

Two facts drive almost every result below:

1. Control plane, etcd and workload share the same 4 vCPU, so any load competes with the thing being measured.
2. The cluster nodes boot from Ceph — and so does Rancher. There is no wall between them.

Method. kubectl ran from my admin host through the Rancher proxy, so client-side latency includes that hop. Load came from stress-ng and fio in Alpine pods. Server-side API latency came from apiserver_request_duration_seconds; etcd health from etcdctl endpoint status and etcdctl check perf; real node memory from a privileged hostPID DaemonSet reading /proc/meminfo.

Scorecard

CPU: the interesting part is the knee

I ramped load rather than saturating everything at once.

At two workers the API is unaffected (157 ms is noise next to 173 ms). At four workers on the same node it jumps 2.6×. The cluster can absorb roughly half a node’s spare CPU before the control plane starts queueing.

The counter-intuitive result: saturating one node was 59% worse for the API than saturating all three. That node hosted the etcd leader, and every Kubernetes write goes through the leader. Starve it and you starve every write in the cluster while two nodes sit idle. Spread the contention evenly and nothing is disproportionately starved.

The rule I took from it: the worst neighbour for this cluster is a CPU-hungry pod next to the etcd leader. Where the load lands matters more than how much of it there is.

Memory: fine, but not for the reason you’d hope

Holding 4 GiB in one pod (out of ~6.5 GiB available) caused no OOMKill, no eviction and no MemoryPressure. API p50 stayed at 155 → 156 ms. Memory is not what breaks this cluster.

But inspecting the nodes turned up a configuration gap:

• There is no swap, and failSwapOn: false.
• kubelet’s evictionHard contains only imagefs.available and nodefs.available (both 5%) — no memory.available threshold at all. The upstream default isn’t in effect.
• vm.overcommit_memory = 1, so the kernel accepts any allocation.

So real memory exhaustion here would be handled by the kernel OOM killer, not by graceful kubelet eviction. Worth fixing before production (add a memory.available hard threshold, plus a soft one with a grace period).

Disk: the number that explains everything else

I did not complete the pod-level disk phase, and I’m not going to pretend otherwise: the fio pod is what caused the incident below. These are VM-level baselines on the same Ceph-backed storage, which already establish the ceiling.

A 4K fsync costs roughly 43 ms here, and this path manages only 23 of them per second. etcd commits every write with an fsync. That one number predicts the control-plane failure, the apiserver probe flaps and the management outage that follow.

It also can’t be tuned away today: no StorageClass exists (a PVC would sit Pending forever), the media is two HDDs, the root pool is size=2 on a single host (every write goes to the same two disks twice), and every pool shares the same OSDs with no QoS.

Network: the overlay, not the wire

Going from 1 to 8 streams changed nothing, so this isn’t a congestion or per-connection limit — it’s a hard per-packet cost. The A/B pair (same nodes, same NIC, only encapsulation differs) isolates it: an 18× penalty from VXLAN alone.

Why: the virtio NIC can’t offload VXLAN segmentation, so Calico detects unreliable checksum offload and sets ChecksumOffloadBroken=true. Felix then turns off TSO and tx-checksumming on vxlan.calico, which also has a single queue. Every tunnelled packet is segmented and checksummed in software on one core. At ~1450 bytes per packet that is about 64,500 packets/s ≈ 0.8 Gbit/s — which matches what I measured.

I ruled out a configured rate limit: overlimits 0, ecn_mark 0, and drops of 215 packets out of 44.7 million.

A validity problem in my own numbers. I checked the Nova database: all three nodes were scheduled on the same compute host. So “cross-node” traffic never touched the 10G switch; it stayed on the local OVS bridge. That’s why the no-VXLAN leg reached 13.4 Gbit/s, which a 10G line physically can’t carry. This phase measured encapsulation cost, not the 10G fabric, and 0.8 Gbit/s should be read as a best case. (It also means losing that one host loses the whole cluster.)

Control plane: etcd fails its own minimum

Reads are comfortable; writes are 4× slower, which is the fsync signature. The cluster sustains about 70% of etcd’s own minimum bar, reproducibly.

Two side effects worth recording:

• check perf ratchets your database up. Two runs wrote ~12,000 keys. I deleted them and the prefix reads back empty, but etcd doesn’t return space without compaction. The DB went from 12 MB to 39 MB and stayed there. Pair this test with compact + defrag.
• Write bursts trip the apiservers’ own health probes. During a 60-pod burst all three kube-apiserver pods logged Unhealthy on the etcd probe for about six seconds, then recovered with zero restarts. That’s a real capacity signal: the control plane will flap readiness under write bursts.

Density and scheduling

I used pause pods so the test measures scheduler, kubelet and CNI rather than application load — and pre-warmed the image on every node. A cold pull took 21 s and would have been counted as “scheduling latency”.

The ceiling is CPU requests (~81 pods at 100 m each), not memory or the 110-pod limit.

My own gate was wrong. My runbook said “if total p50 exceeds 30 s, the scheduler or apiserver is drowning”. Total p50 was 31 s. But split into legs: scheduler p50 was 1 s, and the apiserver was healthy. All the cost was in kubelet + CNI (p50 30 s), which serialises at roughly 0.36 pods/s per node. The two legs point to completely different fixes, so I now measure them separately.

The soak, and the incident

The plan: run everything at once for 30 minutes — CPU stress on all nodes, 4 GiB memory stress, fio, and 30 pause pods — and watch for stability rather than peaks.

Root cause: no storage isolation

The benchmark VMs’ disks and Rancher’s disks sat on different pools — but the same two OSDs. fio queued every other pool behind it, including the one holding the management etcd’s WAL. etcd’s fdatasync hit 1.258 s, its leader was overloaded, readiness probes failed and quorum collapsed. It couldn’t recover on its own because the load kept the disks saturated.

And the load was self-sustaining: the stress workload was a Deployment, so every time its container hit its timeout, Kubernetes restarted it. “Wait for it to finish” was never an option.

What didn’t work

What did

# 1) Stop the restart churn on every management node
for h in $MGMT_NODES; do ssh "$h" 'sudo systemctl stop rke2-server'; done

# 2) The critical step: kill the stale etcd and orphaned shims
for h in $MGMT_NODES; do
  ssh "$h" 'sudo pkill -TERM -x etcd; sleep 6; sudo pkill -KILL -x etcd;
            sudo pkill -KILL -f containerd-shim;
            sudo rm -f /run/k3s/containerd/containerd.sock'
done

# 3) Start all three at the same time — quorum needs 2 of 3
for h in $MGMT_NODES; do ssh "$h" 'sudo systemctl start rke2-server' & done; wait

Then, and only then, remove the storage contention — the management etcd can’t stabilise while the OSDs are saturated.

The diagnostic I wish I’d run first:

sudo ss -lntp | grep -E ':(2379|2380)'

If an etcd PID with days of uptime is still listening while every client request times out, it’s wedged, not slow. You can’t tell the difference from logs alone, because a wedged process has gone quiet. That one check would have saved about 40 minutes.

The access-design failure

While Rancher was down, the Kubernetes API was gone, the benchmark VMs used Rancher-generated keypairs (so my lab key didn’t work), my admin host had no route to the node VLAN, and there was no guest agent or console login. The only lifeline was an SSH path into a management node I happened to have. Without it the outage would have been unrecoverable from my host. A recovery path that depends on one unrecorded credential isn’t a recovery path.

Findings and what I’m doing about them

Immediate rules: no storage benchmarks against shared Ceph pools, ever; any long-running load must be a Job with a deadline, not a Deployment; and any abort path must work without the Kubernetes API.

What held up

Scheduling (p50 1 s under a 60-pod burst), pod density, memory headroom, the API read path (p50 25 ms, 90% of reads under 100 ms) and Calico stability all held. Both clusters came back with no data loss and no rebuilds.

Takeaway

The cluster is healthy and predictable for read-mostly workloads and bursts up to about 80 pods. It isn’t ready for control-plane-heavy work, and it has no storage isolation between tenants. Everything I worried about beforehand — CPU, memory, pod count — was fine. The thing that broke was the one thing every layer was quietly sharing.