Benchmarking a Small Cloud: Where 1G Ethernet and HDDs Actually Cap Out

After building the cloud, the obvious question was: is it fast enough, and where does it hurt?

I expected to find a software problem to tune. Instead every measurement landed exactly on a physical limit — and proving that is worth more than any tuning win, because it tells you precisely which hardware to buy next.

Environment: OpenStack Kolla-Ansible 22.1.0 (2026.1) on 3× Dell R440 (64 threads / 187 GB each), external Ceph with HDD OSDs, 1G management path.

Summary

Component	Result	Verdict
Network throughput (4× TCP streams)	908 Mbit/s	1G NIC saturated (~90% line rate)
Sequential read (1M blocks)	107 MiB/s	Network-limited, not disk
Sequential write (1M blocks)	68 MiB/s	HDD write path + WAL commit
Random 4K read/write mix (70/30)	~292 IOPS	Hardware limit — HDD seek-bound
Random 4K write	~481 IOPS	Hardware limit — HDD seek-bound
Disk utilization during test	98.2%	Confirms disk is the bottleneck

The infrastructure is balanced: every component sits at its physical limit. No amount of tuning moves these numbers, because they are not software numbers.

Test setup

VMs: bench-1 on compute01, bench-2 on compute02, with an anti-affinity policy so both cannot land on the same host
Boot volume: 30 GB on a single-copy Ceph pool via Cinder RBD
Tools: fio (libaio, direct=1) and iperf3
Network path: tenant network (VXLAN), VM-to-VM
Duration: 60 s per profile, time-based, 512 MB job file

Anti-affinity is not decoration. Two VMs on the same host measure the loopback, not the network.

Network: 908 Mbit/s is the wire

iperf3 -c <peer> -t 30 -P 4 — four parallel TCP streams:

Stream	Transfer (30 s)	Bitrate	Retransmits
1	812 MB	227 Mbit/s	462
2	811 MB	227 Mbit/s	587
3	813 MB	227 Mbit/s	470
4	812 MB	227 Mbit/s	470
SUM	3.17 GB	908 Mbit/s	1989

908 Mbit/s aggregate is ~90.8% of 1G line rate. After VXLAN encapsulation overhead (~5–8%) the wire is effectively 100% saturated.

A single stream caps at ~227 Mbit/s because of latency and TCP window sizing. Four parallel streams fill the pipe. That is normal TCP behaviour, not a defect — but it does mean any single-connection workload will feel slower than the link suggests.

1989 retransmits across 3.17 GB is 0.06% — negligible, no congestion.

Conclusion: nothing to fix. This needs 10 GbE.

I/O: two different ceilings

fio with libaio, direct=1, 512 MB, time-based, 60 s:

Profile	Read	Write	IOPS	Notes
Seq read (1M, iodepth 32)	107 MiB/s	—	~107	Network-limited (1G cap ≈ 112 MB/s)
Seq write (1M, iodepth 32)	—	68 MiB/s	~68	HDD write path + WAL commit
Rand rw 4K 70/30 (2 jobs)	812 KiB/s	357 KiB/s	~292	HDD seek-bound
Rand write 4K (2 jobs)	—	1925 KiB/s	~481	HDD seek-bound

Three separate stories here:

Sequential read at 107 MiB/s is the network, not the disk. The 1G NIC’s practical max is ~112–117 MB/s. The OSDs can deliver more than that. This is actually a healthy sign for the storage layer — the storage is not the constraint.
Sequential write at 68 MiB/s is the HDD write path. Ceph requires WAL/journal commit plus fsync batching, and the replication write doubles the head movement. Below read for a physical reason, not a configuration error.
Random 4K at ~300–480 IOPS is the physical seek rate of a 7200 RPM drive. Raw seek gives ~100–150 IOPS; NCQ lifts it to 300–500. We are at the hardware ceiling. Ceph tuning cannot help.

Disk utilization at 98.2% during the random tests confirms it: the disk, not Ceph, not the network, not the CPU, is the bottleneck.

What I tuned anyway

Even at a hardware ceiling, there are real wins in latency under contention:

# Give client IO priority over recovery and scrub
ceph config set osd osd_mclock_profile high_client_ops

# Throttle background recovery so it cannot starve clients
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 1

# Move scrubbing off-hours
ceph config set osd osd_scrub_begin_hour 2
ceph config set osd osd_scrub_end_hour 5

# Bluestore cache 1G -> 4G (the nodes have 187 GB to spare)
ceph config set osd bluestore_cache_size_hdd 4294967296

PG counts were deliberately left alone. On a small OSD count, changing them triggers heavy backfill for a marginal gain.

Upgrade path, in cost order

Upgrade	Expected gain	Effort / cost
10 GbE NIC	Sequential read/write 4–5× (400+ MB/s)	Low — cable + switch port
SSD/NVMe OSD for a hot pool	Random IOPS 10–50× (5k–15k)	Medium
More HDD OSDs	Random IOPS scale ~linearly	Low if spare disks exist
Bluestore WAL/DB on SSD	Lower write latency, higher sequential write	Medium

For a visible improvement with least effort: a small SSD for bluestore WAL/DB on the existing OSDs. WAL stops blocking the spinning head, so sequential write improves and random IOPS rise modestly.

For a production blueprint: 10G plus an SSD tier is the whole answer. The current 3× HDD + 1G design is correctly balanced — it is simply capped by physics.

And that ceiling was later confirmed by the fix: moving boot volumes to dedicated HDDs over a 10G path took random 4K write from ~481 IOPS to 4775 IOPS at p50 1.48 ms. Same software, same Ceph, different wire and different disk.

Ops notes worth keeping

fio size × job count fills small root disks. Eight jobs at 2 GB each wrote 16 GB+ of test files onto a 20 GB volume: root hit 100%, journald broke, growpart hung. Use size=512M with time_based and stonewall — test files stay small and clean themselves up.

Cinder volume extend fails when the disk is 100% full. RBD needs room for metadata. Free space, reset the volume state with openstack volume set --os-volume-api-version 3.24 --state in-use <vol>, then extend.

After extending a boot volume, reboot the VM so the kernel re-reads the partition table, then resize2fs picks up the new size automatically.

Security groups are project-scoped. A group from project A cannot be used by a server in project B — the build fails on every host with SecurityGroupNotFound, which reads like a Nova bug and is not.

--hint group=<name> requires a UUID. Use openstack server create --server-group <name> and let the CLI resolve it.