Docker Compose coordinates services on one machine. Enterprise AI needs more: traffic surges, nodes fail, and upgrades must happen with zero downtime. Kubernetes (often written K8s) is the industry-standard control plane for that. Learning it starts with three building blocks.

In plain terms: Compose is a manager running one kitchen. Kubernetes is a head office running a chain of kitchens: it opens a new branch when the queue grows, replaces a branch that burns down, and rolls out a new menu one branch at a time.

Diagram: Docker Compose runs services on one host. Kubernetes is a control plane over many nodes: it reschedules workloads after failures, scales on demand and performs rolling updates without downtime. See the animated version.

1. Pods: the atomic unit

A Pod is the smallest thing Kubernetes schedules. It wraps one or more tightly coupled containers that share a network address and can share storage volumes. An inference Pod usually pairs the model server with a sidecar container that handles logging, telemetry or request sanitising.

Diagram: A Pod is the smallest unit Kubernetes schedules. An inference Pod usually pairs the model server with a sidecar container for logging or telemetry. The containers share one network address and can share volumes. See the animated version.

2. Deployments: updates and self-healing

You rarely create Pods by hand. A Deployment declares what you want, for example "three replicas of this model server image", and a controller keeps reality matching that.

• Rolling updates. When you change the model version or a prompt template, the Deployment starts new Pods, waits for their readiness checks to pass, then retires the old ones, so in-flight requests are not dropped.
• Self-healing. If a heavy context pushes a container past its memory limit and it is killed (OOMKilled), Kubernetes restarts or replaces it to bring the count back to what you declared.

Diagram: A Deployment keeps the declared number of Pod replicas running. For an update it starts new Pods, waits for their readiness checks, then stops old ones so in-flight requests are not dropped. If a Pod is killed for running out of memory, Kubernetes restarts or replaces it to restore the desired count. See the animated version.

3. Services: one stable address

Pods come and go, and each new Pod gets a new IP. A Service sits in front of them with a stable virtual IP and DNS name.

• ClusterIP exposes the model only inside the cluster, so internal applications can reach it while it stays hidden from the public internet.
• Readiness-aware routing. The Service only sends traffic to Pods that pass their readiness probe. If a Pod is busy and its readiness check starts failing, new requests go to its idle siblings. Tune the probe to your model server, since a probe that is too strict can take every Pod out of rotation at once.

Diagram: A Service gives a changing set of Pods one stable virtual IP and DNS name. With a ClusterIP the model is reachable only inside the cluster. The Service sends traffic only to Pods that pass their readiness probe, so a Pod that reports itself busy is skipped. See the animated version.

Here is a compact example. It assumes an image called model-api that serves a /health endpoint on port 8000. Adjust the names, resources and probe to your own server.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-api
spec:
  replicas: 3
  selector:
    matchLabels: { app: model-api }
  strategy:
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }   # never drop capacity during an update
  template:
    metadata:
      labels: { app: model-api }
    spec:
      containers:
        - name: model-api
          image: registry.example.com/model-api:1.4.0
          ports: [{ containerPort: 8000 }]
          resources:
            requests: { cpu: "2", memory: 8Gi }
            limits:   { memory: 8Gi }                   # exceeding this gets the container OOMKilled
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: model-svc
spec:
  type: ClusterIP            # reachable only inside the cluster
  selector: { app: model-api }
  ports: [{ port: 80, targetPort: 8000 }]

Apply it with kubectl apply -f model.yaml, and inside the cluster your application calls http://model-svc. These three abstractions are the baseline for scaling AI microservices with automated resilience.

Coming Up Next

Day 20: Persistent storage in Kubernetes: managing state, weights and vector indices.

Kubernetes #K8s #MLOps #DevOps #CloudNative

Originally published at https://sureshpallapothu.in/blog/day-19-kubernetes-for-ai, where this post includes animated diagrams.