Dev.to AI 🤖 Ai 👁 0 📖 3 min read

Day 11: StatefulSets for Vector Databases — Deploying Milvus or Qdrant in Kubernetes

A crashed stateless pod comes back with a random name and a new IP, and nobody minds. A database node that returns with a different identity can break cluster consensus and force expensive re-sharding. A StatefulSet fixe

A crashed stateless pod comes back with a random name and a new IP, and nobody minds. A database node that returns with a different identity can break cluster consensus and force expensive re-sharding. A StatefulSet fixes that with stable network identities, ordered operations and automatic per-pod storage.

Identity and storage

Diagram: FIG 1 · Each StatefulSet pod keeps a stable DNS name and its own PersistentVolumeClaim, and pods start in order. A restarted pod comes back as the same member. See the animated version.

Each pod gets a predictable DNS name, such as qdrant-0.qdrant-headless.ai-platform.svc.cluster.local, plus its own PVC. Pods start in order, qdrant-0 then qdrant-1, and the name survives a restart.

Why Deployments fail databases

Diagram: FIG 2 · Identity, storage and shutdown order all matter to a clustered database. StatefulSets handle each, and Deployments do not. See the animated version.

  1. Identity instability. Random names and IPs after a restart force the cluster to re-sync large indexes.
  2. Storage contention. A Deployment shares one claim among its pods. A StatefulSet uses volumeClaimTemplates to create a dedicated volume per replica.
  3. Uncontrolled termination. A Deployment can stop many pods at once. A StatefulSet stops them in strict reverse order, qdrant-2, then qdrant-1, then qdrant-0, which helps the cluster keep quorum.

Diagram: FIG 3 · Pods start from zero upward and stop from the highest ordinal downward, which helps clustered databases keep quorum during restarts and updates. See the animated version.

The manifest

This runs Qdrant in clustered mode. Qdrant needs each new peer to know where to bootstrap from, so the first pod starts the cluster and the others join it.

apiVersion: v1
kind: Service
metadata:
  name: qdrant-headless
  namespace: ai-platform
spec:
  clusterIP: None
  selector: { app: qdrant }
  ports:
    - { name: http, port: 6333 }
    - { name: internal-p2p, port: 6335 }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: qdrant
  namespace: ai-platform
spec:
  serviceName: qdrant-headless
  replicas: 3
  podManagementPolicy: OrderedReady
  selector:
    matchLabels: { app: qdrant }
  template:
    metadata:
      labels: { app: qdrant }
    spec:
      containers:
        - name: qdrant
          image: qdrant/qdrant:v1.9.0
          env:
            - { name: QDRANT__CLUSTER__ENABLED, value: "true" }
            - { name: QDRANT__CLUSTER__P2P__PORT, value: "6335" }
          command: ["/bin/sh", "-c"]
          args:
            - |
              HOST="$(hostname).qdrant-headless"
              if [ "$(hostname)" = "qdrant-0" ]; then
                ./qdrant --uri "http://${HOST}:6335"
              else
                ./qdrant --bootstrap "http://qdrant-0.qdrant-headless:6335" --uri "http://${HOST}:6335"
              fi
          ports:
            - { name: http, containerPort: 6333 }
            - { name: p2p, containerPort: 6335 }
          resources:
            requests: { cpu: "1000m", memory: 2Gi }
            limits:   { cpu: "2000m", memory: 4Gi }
          volumeMounts:
            - { name: qdrant-data, mountPath: /qdrant/storage }
  volumeClaimTemplates:
    - metadata: { name: qdrant-data }
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests: { storage: 20Gi }

Check the bootstrap flags against the Qdrant version you run, because clustering options have changed between releases.

How many replicas?

Raft-based clusters need a majority of nodes to agree. One and two nodes tolerate no failures, three tolerate one, and five tolerate two. Two replicas are fine for a lab, but for real fault tolerance run three.

Diagram: FIG 4 · A Raft cluster needs a majority of nodes to agree. Three nodes tolerate one failure, five tolerate two, and two nodes tolerate none, so use at least three replicas when availability matters. See the animated version.

Proper StatefulSet design protects your vector indexes through restarts and upgrades, and it gives the rest of the stack a database that behaves predictably.

Run it yourself

This is the exact example from the video. Run against a real K3d cluster. A small nginx image stands in for Qdrant so three pods start quickly.

{apiVersion: v1, kind: Service, metadata: {name: qdrant-headless},
 spec: {clusterIP: None, selector: {app: qdrant}}}
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: qdrant}
spec:
  serviceName: qdrant-headless
  replicas: 3
  selector: {matchLabels: {app: qdrant}}
  template:
    metadata: {labels: {app: qdrant}}
    spec: {containers: [{name: qdrant, image: "nginx:1.27-alpine"}]}
  volumeClaimTemplates:
    - metadata: {name: data}
      spec: {accessModes: [ReadWriteOnce],
        resources: {requests: {storage: 1Gi}}}

Run it:

kubectl apply -f statefulset.yaml

Output I got while recording the video:

qdrant-0 Running
qdrant-1 Running
qdrant-2 Running
data-qdrant-0 Bound
data-qdrant-1 Bound
data-qdrant-2 Bound
headless clusterIP: None

Coming Up Next

Day 12: Auto-scaling AI inference with Horizontal Pod Autoscalers for spiky API traffic.

StatefulSet #VectorDatabase #Qdrant #Milvus #CloudNative #DataEngineering #SystemDesign

Originally published at https://sureshpallapothu.in/blog/ai-int-day-11-statefulsets-vector-db, where this post includes animated diagrams.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.