Blobstore stores attachments, execution results, and workflow filesystem data on a persistent volume. When that volume fills, writes fail and those workloads stop.
The claim is data-blobstore-0 in the 3B namespace (the data volume on the blobstore StatefulSet). The chart default is 50 GiB (blobstore.storage.size).
Pick an expandable StorageClass before you install
This section is for Kubernetes with Helm. On the single-server install, skip it and grow the VM disk instead, as described below.
Growing the volume later only works when the StorageClass has allowVolumeExpansion: true. Set that when you create or choose the class, before the first install. Confirm it:
kubectl get storageclass -o custom-columns=NAME:.metadata.name,EXPAND:.allowVolumeExpansion
If the expandable class is not your cluster default, name it with blobstore.storage.storageClass. The Helm installation guide lists this as a cluster prerequisite; sizing covers volume sizes.
A Helm upgrade that only raises blobstore.storage.size is rejected: Kubernetes does not let you change a StatefulSet’s volume claim template. The procedure below patches the PVC first.
Alarm at 60% full. Expanding still takes a PVC patch, a Helm upgrade, and time for the filesystem to grow. Sixty percent is the point to start that work, not the point the volume is already in trouble.
Metric
Scrape kubelet volume stats for the Blobstore PVC.
If you are running Prometheus on your cluster, you can try this ratio:
kubelet_volume_stats_used_bytes{persistentvolumeclaim="data-blobstore-0"}
/
kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="data-blobstore-0"}
Fire when that ratio stays at or above 0.60 for 10 minutes. Also fire if the metric is missing for 10 minutes — a silent scraper looks like a healthy disk.
Other pipelines equivalent: Datadog’s kubernetes.kubelet.volume.stats.used_bytes / capacity_bytes filtered to data-blobstore-0, or your cloud’s PVC / EBS used-percent on the volume behind that claim.
Confirming from the cluster
kubectl -n 3b get pvc data-blobstore-0 kubectl -n 3b exec statefulset/blobstore -- df -h /var/data/blobstore
What to do on Kubernetes with Helm
Patch the PVC to the new size (100Gi here; pick what you need):
kubectl -n 3b patch pvc data-blobstore-0 -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'
Wait until
kubectl -n 3b get pvc data-blobstore-0shows the new capacity.Set
blobstore.storage.sizeto that same size in your values file.Recreate the StatefulSet without deleting the pod or the volume, then apply a Helm upgrade so the chart’s claim template matches:
kubectl -n 3b delete statefulset blobstore --cascade=orphan
Then run the usual helm upgrade from upgrading on Kubernetes. --cascade=orphan removes the StatefulSet object and leaves the running pod and the PVC in place.
Wait until
dfinside the pod agrees with the new size.
The in-cluster Postgres volume is the same kind of claim. Use PVC pgdata-postgres-0, StatefulSet postgres, and postgres.storage.size.
For Blobstore CPU or memory, raise blobstore.resources instead. See how and when to scale and sizing.
What to do on the single-server install
The single-server install uses k3s’s local-path provisioner, which does not support volume expansion. Do not patch the PVC, and do not raise blobstore.storage.size in EXTRA_VALUES_FILE. Grow the VM disk (or the filesystem under /var/lib/rancher/k3s). The volumes live on that host filesystem and fill with it. Alarm on host disk as well as the PVC.