Self-Service Persistent Storage on Kubernetes with Cinder CSI

The ask was simple: an application team should be able to create a PersistentVolume by writing a normal PVC — no ticket, no manual volume creation in OpenStack, no operator in the loop.

That means the Kubernetes cluster needs to talk to Cinder itself. This is how it was built, and the four defects I hit on the way — three of which failed silently, which is the dangerous kind.

What was built

The three StorageClasses

A fourth class named default exists as a legacy alias with no default annotation, purely so old manifests that hardcode storageClassName: default keep working.

Because cinder-rbd carries the default annotation, a Deployment that never mentions a StorageClass provisions against it automatically.

How a request flows

Developer writes a PVC (3 lines)
        ↓
Kubernetes reads the default StorageClass → cinder-rbd
        ↓
Cinder CSI calls the OpenStack Cinder API
        ↓
Cinder creates a volume in Ceph pool `volumes` (replica 2)
        ↓
Volume attaches to the node and mounts into the pod
        ↓
PVC becomes Bound — the developer never touches OpenStack

What the developer actually writes

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 10Gi
      # storageClassName is not required — cinder-rbd is the default

That is the whole interface. Three meaningful lines.

The four defects

This is the part worth reading, because three of them produced a working-looking system that was not working.

1. cloud.conf used project-domain-name

Not a valid key. The INI parser discards unknown keys silently, so the file parsed cleanly and the CSI controller crash-looped with a misleading authentication error. The correct key is tenant-domain-name.

Lesson: a config file that parses is not a config file that is understood. Compare against the driver’s documented cloud.conf reference, don’t infer key names from the OpenStack CLI flags.

2. The Helm value for topology was wrong

csi.plugin.node.topology does not exist. Helm ignored it with no error — no warning, no diff, nothing. With topology enabled and nothing publishing topology keys, PVCs would have stayed Pending forever.

The correct key is csi.provisioner.topology: false.

Lesson: helm install returning STATUS: deployed means the template rendered. It does not mean your value landed in it. Verify with kubectl get deploy -o yaml and read the actual container args.

3. Snapshots of in-use volumes were rejected by Cinder

The VolumeSnapshotClass needs force-create: "true". Without it, snapshotting an attached volume fails at the API.

Be clear about what this buys: it makes the snapshot crash-consistent, not application-consistent. A database snapshot taken this way is equivalent to pulling the power cord. Quiesce the application first if that matters.

4. A dedicated volume type was mandatory

The pre-existing rep1 type had empty extra specs, so Cinder’s capacity weigher — not the operator — decided placement. A volume could land on a single-replica pool and be one disk failure from data loss.

Creating ceph-rbd2 with an explicit backend mapping makes placement deterministic. This is not tuning; it is correctness.

Verified end to end

The second row is the important one. A PVC can be Bound and still be on the wrong pool — check the backend, not just the status.

Two behaviours that look like faults and are not

A PVC stays Pending until a pod consumes it. This is WaitForFirstConsumer working as designed. Creating a standalone PVC and watching it stay Pending is expected.

The first attach can log one or two FailedAttachVolume warnings, then self-heal within 1–3 minutes. That is a retry race between the attach/detach controller and the driver, not a defect. Alerting on it produces noise.

Decisions recorded

No per-user quota or limit policy. The OpenStack project quota is the single ceiling. Usage is monitored manually rather than enforced per namespace — a deliberate trade to avoid building a policy system nobody asked for.

No ReadWriteMany. The lab has neither Manila nor an NFS server, so only ReadWriteOnce is possible. Multi-pod workloads cannot share one volume. Saying this out loud matters more than it sounds: a team that assumes RWX will design an architecture this cluster cannot run.

Known gaps

1. No offsite backup. The Ceph host is a single point of failure for every volume and snapshot in the cluster. Replicating etcd snapshots and VM disk snapshots to object storage is the only real protection, and it is not built yet.
2. The CSI credential was printed during diagnostics and should be rotated.
3. A newly built downstream cluster has no CSI. The install and default-class annotation must be re-applied per cluster — this is not baked into the provisioning template.

Gap 3 is the one that will bite in practice: cluster provisioning and storage enablement are still two separate manual steps, and nothing enforces that the second one happened.