Day 10: Persistent Volumes (PV/PVC) — Managing Heavy Model Weights and Vector Stores
Pods are ephemeral. If every restart downloads a 10 to 30 GB model, you saturate bandwidth, slow every rolling update and can face 10 to 15 minute cold starts. The fix is to keep the weights on storage that outlives the
Pods are ephemeral. If every restart downloads a 10 to 30 GB model, you saturate bandwidth, slow every rolling update and can face 10 to 15 minute cold starts. The fix is to keep the weights on storage that outlives the pod.
Diagram: FIG 1 · Downloading a large model on each restart saturates bandwidth and slows rolling updates. A volume that already holds the weights turns minutes into seconds. See the animated version.
Three layers of storage
- StorageClass: how storage is provisioned: a local path, AWS EBS, Ceph RBD and so on.
- PersistentVolume (PV): a piece of storage provisioned by an administrator or dynamically.
- PersistentVolumeClaim (PVC): the developer's request. It binds to a matching PV, and pods reference the claim.
Diagram: FIG 2 · A StorageClass describes how storage is provisioned, a PersistentVolume is the storage itself and a PersistentVolumeClaim is the request that pods reference. See the animated version.
Access modes
- ReadWriteOnce (RWO): read-write on a single node. Vector databases need it, because writes from several nodes can corrupt write-ahead-log segments.
- ReadOnlyMany (ROX): read-only on many nodes at once. This is the standard way to share model weights: write them once, then mount them in dozens of stateless pods.
Diagram: FIG 3 · Databases need ReadWriteOnce so only one writer touches the index. Model weights suit ReadOnlyMany, where many stateless pods read one copy. See the animated version.
Check that your storage backend supports the mode you ask for. Network filesystems such as NFS generally support ROX, while plain block disks usually do not.
Manifests for local weights
This example keeps weights in a local folder on the nodes.
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-model-weights-pv
spec:
capacity: { storage: 50Gi }
volumeMode: Filesystem
accessModes: [ReadOnlyMany]
persistentVolumeReclaimPolicy: Retain # keep the data if the claim is deleted
storageClassName: local-storage
local:
path: /opt/ai/models
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values: [k3d-ai-cluster-agent-0, k3d-ai-cluster-agent-1]
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: shared-model-weights-pvc
namespace: ai-platform
spec:
accessModes: [ReadOnlyMany]
storageClassName: local-storage
resources:
requests: { storage: 50Gi }
A local volume is a path on one specific node, so each node needs its own copy of the files, and the nodeAffinity tells Kubernetes which nodes have them. For production, a shared network filesystem avoids copying the weights to every node.
The k3d gotcha
k3d nodes are Docker containers, so /opt/ai/models exists inside them only if you mount the host folder into every agent when you create the cluster. Without that, the PV points at an empty path.
Diagram: FIG 4 · k3d nodes are containers, so a local PV path only exists on them if you mount the host folder into every agent when you create the cluster. See the animated version.
sudo mkdir -p /opt/ai/models
k3d cluster create ai-cluster --agents 2 \
--volume /opt/ai/models:/opt/ai/models@agent:*
With the volume in place, a restarted pod mounts weights that are already on disk. Proper storage design removes repeated downloads, keeps rolling updates fast and protects your data across cluster changes.
Run it yourself
This is the exact example from the video. Run against a real K3d cluster, using a hostPath volume instead of the lesson's local path. No pod mounts the volume here, so it only shows the binding.
apiVersion: v1
kind: PersistentVolume
metadata: {name: weights-pv}
spec:
capacity: {storage: 50Gi}
accessModes: [ReadOnlyMany]
storageClassName: local-storage
hostPath: {path: /tmp/ai-models}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: weights-pvc}
spec:
accessModes: [ReadOnlyMany]
storageClassName: local-storage
resources: {requests: {storage: 50Gi}}
Run it:
kubectl apply -f storage.yaml
Output I got while recording the video:
pv weights-pv Bound -> weights-pvc
pvc weights-pvc Bound -> weights-pv
size 50Gi, mode ReadOnlyMany
Coming Up Next
Day 11: StatefulSets for vector databases: deploying Milvus or Qdrant in Kubernetes.
KubernetesStorage #PV #PVC #DataArchitecture #CloudNative #DevOps #StorageClass #MLOps
Originally published at https://sureshpallapothu.in/blog/ai-int-day-10-persistent-volumes, where this post includes animated diagrams.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.