Back to kosir.pro

Writing · 30 Jul 2026

Why removing a PV finalizer strands your disk — and what to do instead

A PersistentVolume stuck in Terminating is one of the most reliably annoying things in Kubernetes. You deleted the PVC, the PV should follow, and instead it just sits there forever, one status away from gone. The top search result is always the same fix: patch the object and null out its finalizers. It works — the object disappears in seconds. That's the problem.

What the finalizer was actually doing

A finalizer isn't decoration. It's the CSI (or in-tree) provisioner telling the API server: don't erase this record yet, I still have cleanup to do on the actual storage backend — detaching the volume, releasing the attachment, sometimes deleting the underlying disk. The PV object staying around in Terminating is the visible symptom of that cleanup being stuck, not the problem itself.

When you patch the finalizer list to empty, you don't finish that cleanup. You just tell Kubernetes to stop remembering that the cleanup was pending. The API object goes away. The disk — the vSphere VMDK, the cloud block volume, whatever it actually is — does not.

What that costs you later

Nothing points at that disk anymore, so nothing will ever finish deleting it, alert on it, or bill it to a workload you'd recognize. It just sits there consuming capacity under a volume ID that no controller is tracking. In the more unpleasant case, that ID or device path gets reused — a new PV provisioned against storage that still has stale data or a lingering attachment from the one you "deleted" — and now you're debugging corruption instead of a stuck finalizer.

Find out why it's actually stuck first

A PV usually gets stuck in Terminating for one of a small number of reasons, and all of them are diagnosable before you touch anything destructive:

  • The CSI controller pod is down, crash-looping, or can't reach the backend — so nothing is even attempting the detach.
  • The backend genuinely refuses the detach: a stale attachment record, an orphaned snapshot holding a lock, a claimRef pointing at a node that no longer exists.
  • A second, unrelated finalizer — often from a backup or snapshot tool — is also holding the object open, and it's not the storage driver you're looking at.

kubectl get pv <name> -o yaml shows you the full finalizer list, not just the one you assumed was blocking it. Cross-reference that against the CSI driver's controller logs for the actual detach/delete error, and against the backend directly (vSphere, cloud console, whatever's underneath) for the state of the volume itself. Almost always, the real fix is on the backend side — clear the stale attachment, remove the orphaned snapshot, restart the wedged controller — and the finalizer clears on its own once the condition it was waiting on is gone.

If you still have to remove it

Sometimes the backend object is legitimately already gone — deleted out-of-band, or the environment it lived in no longer exists — and the finalizer is waiting on a cleanup that can never happen. That's the one case where clearing it by hand is correct. Verify the backend side first, by hand, not by assumption. And treat it as an event worth a line in a runbook or a ticket, not a one-off shell command you forget about — the next person who finds that PV gone needs to know it was a deliberate, verified removal and not evidence of a bug.

This is the exact triage I built pvdoctor to make repeatable: it identifies the real backend cause behind a stuck PV — orphaned snapshots, stale claimRefs, dangling attachments, a backup finalizer that isn't the storage driver's — and shows you the safe fix next to the finalizer-removal shortcut, instead of just handing you the shortcut.

Back to kosir.pro