Building Disaggregated Inference with NVIDIA Dynamo

NVIDIA Dynamo is an open-source inference framework for deploying large-scale generative AI models across multi-node, multi-GPU environments. Disaggregated serving architecture separates prompt processing (prefill) and token generation (decode) phases across different GPU workers, enabling independent scaling and optimization of each phase based on their distinct computational characteristics. This guide covers building disaggregated inference deployments with NVIDIA Dynamo using AIConfigurator for optimal configuration discovery, deploying on Kubernetes with RDMA, and validating performance. Disaggregated architecture can achieve up to 1.7x throughput improvement over aggregated serving for large models (32B+) with long input contexts; for smaller models, AIConfigurator automatically determines when aggregated serving is more efficient. By the end, you'll have used AIConfigurator to find an optimal serving configuration, deployed it on a GPU Kubernetes cluster with RDMA-accelerated KV cache transfer, and validated throughput with AIPerf.

Prerequisites

Before you begin, ensure you have one of the following environments:

For Kubernetes Deployment:

• A Kubernetes cluster with NVIDIA GPU nodes and the RDMA device plugin installed.
• kubectl configured to access your cluster.
• An RDMA-capable network (InfiniBand or RoCE).
• The NVIDIA Dynamo platform installed on your cluster, including the Dynamo Operator and GPU resource management configured.

For Standalone Instance Deployment:

• A GPU-enabled server with NVIDIA GPUs and the NVIDIA Container Toolkit configured.
• 2+ GPUs for disaggregated serving.
• Docker Engine and Docker Compose.
• A non-root user with sudo privileges.

Common Requirements:

• Python 3.10+ for running AIConfigurator.
• A Hugging Face account with an access token for gated models.

Understanding Disaggregated Serving Architecture

Disaggregated serving splits LLM inference into two specialized phases, each optimized for different computational characteristics.

Prefill Phase vs Decode Phase

Architecture Components

Prefill Workers:

• Process incoming prompts and generate KV cache.
• Run with low batch sizes for optimal TTFT.
• Can use different tensor parallelism (TP) than decode workers.
• Transfer KV cache to decode workers via NIXL.

Decode Workers:

• Receive KV cache from prefill workers.
• Generate output tokens sequentially.
• Handle high concurrency with large batch sizes.
• Optimized for sustained token throughput.

Request Routing:

• Frontend routes requests to available prefill workers.
• PrefillRouter selects prefill workers based on KV cache affinity.
• After prefill, decode workers receive KV cache and continue generation.
• NIXL enables GPU-to-GPU KV cache transfer with RDMA.

When to Use Disaggregated Serving

Use Disaggregated Serving When:

• Model size is large (32B+ parameters) — smaller models have lower KV cache transfer overhead that doesn't justify phase separation.
• Input sequence length (ISL) significantly exceeds output sequence length (OSL) — typically ISL > 4000 tokens.
• Workload requires independent scaling of prefill and decode capacity.
• RDMA-capable network infrastructure is available (critical requirement).
• Maximizing per-GPU throughput is the primary objective.

Use Aggregated Serving When:

• Model size is small (<10B parameters) — KV cache transfer overhead outweighs benefits.
• Balanced ISL/OSL ratios (between 2:1 and 10:1).
• Simpler deployment with lower operational complexity is preferred.
• RDMA infrastructure is not available.
• Strict latency SLAs require minimal inter-worker communication overhead.

Performance Benefits (for Large Models):

• Up to 1.7x higher throughput compared to aggregated serving for large models with long-context workloads.
• Better GPU utilization through phase-specific optimization.
• Independent scaling of prefill and decode resources based on workload patterns.

Note: RDMA is required. Disaggregated serving requires RDMA for KV cache transfer. Without RDMA, performance degrades by 40x (TTFT increases from ~400ms to 10+ seconds) as the system falls back to TCP transfer, creating a severe bottleneck.

1. Install and Configure AIConfigurator

AIConfigurator is a performance optimization tool that automatically determines the optimal configuration for deploying LLMs with Dynamo by evaluating both aggregated and disaggregated architectures.

1. Create a Python 3 virtual environment:

$ python3 -m venv aiconfigurator_env

2. Activate the virtual environment:

$ source aiconfigurator_env/bin/activate

3. Install AIConfigurator using pip:

$ pip3 install aiconfigurator

4. Verify the installation:

$ aiconfigurator version

The output displays the installed version.

5. View available parameters and help documentation:

$ aiconfigurator cli default --help

Understanding AIConfigurator Parameters

AIConfigurator requires the following key parameters:

How AIConfigurator Works:

1. Evaluates all possible configurations (tensor/pipeline parallelism combinations).
2. Simulates performance for both aggregated and disaggregated architectures.
3. Filters configurations that meet SLA targets (TTFT, TPOT).
4. Ranks results by per-GPU throughput (tokens/s/gpu).
5. Automatically selects the best architecture (agg or disagg) based on actual performance.
6. Generates deployment manifests for top-ranked configurations.

AIConfigurator considers factors like model size, KV cache transfer overhead, and GPU utilization to determine whether disaggregated serving will outperform aggregated for your specific workload.

2. Find the Optimal Disaggregated Configuration

Use AIConfigurator to discover the best configuration for your model and hardware.

1. Run AIConfigurator with your model and system specifications:

$ aiconfigurator cli default \
  --hf_id nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 \
  --system h200_sxm \
  --total_gpus 4 \
  --isl 4000 \
  --osl 500 \
  --ttft 600 \
  --tpot 16.67 \
  --backend vllm \
  --backend_version 0.12.0 \
  --save_dir ./results_vllm

The command evaluates all possible configurations for 4x H200 GPUs with the specified model and SLA targets, generating deployment manifests in the ./results_vllm directory.

2. Review the AIConfigurator output summary:

********************************************************************************
*                     Dynamo aiconfigurator Final Results                      *
********************************************************************************
  ----------------------------------------------------------------------------
  Input Configuration & SLA Target:
    Model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 (is_moe: False)
    Total GPUs: 4
    Best Experiment Chosen: agg at 3785.20 tokens/s/gpu (disagg 0.81x better)
  ----------------------------------------------------------------------------
  Overall Best Configuration:
    - Best Throughput: 15,140.82 tokens/s
    - Per-GPU Throughput: 3785.20 tokens/s/gpu
    - Per-User Throughput: 61.41 tokens/s/user
    - TTFT: 395.88ms
    - TPOT: 16.28ms
    - Request Latency: 8521.26ms
  ----------------------------------------------------------------------------

The summary shows that for this 4B model, aggregated serving performs better than disaggregated (3785.20 vs 3067.51 tokens/s/gpu). This is expected for smaller models where the overhead of KV cache transfer outweighs the benefits of phase separation. Disaggregated serving becomes advantageous with larger models (32B+) and longer context lengths.

3. Review the detailed configuration comparison:

agg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | request_latency | concurrency  | total_gpus (used) | replicas | parallel | bs |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+
|  1   |   3785.20    |     61.41     | 395.88 |     8521.26     | 256 (=64x4)  |     4 (4=4x1)     |    4     |  tp1pp1  | 64 |
|  2   |   3036.90    |     65.06     | 331.10 |     8000.51     | 192 (=96x2)  |     4 (4=2x2)     |    2     |  tp2pp1  | 96 |
|  3   |   2305.76    |     65.36     | 249.55 |     7883.93     | 144 (=144x1) |     4 (4=1x4)     |    1     |  tp4pp1  | 144|
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+

disagg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | request_latency | concurrency  | total_gpus (used) | replicas | (p)parallel | (d)parallel |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+
|  1   |   3067.51    |     63.99     | 111.33 |     7909.20     | 208 (=104x2) |     4 (4=2x2)     |    2     |   tp1pp1    |  tp1pp1     |
|  2   |   3011.39    |     62.82     | 111.33 |     8054.41     | 208 (=208x1) |     4 (4=1x4)     |    1     |   tp1pp1    |  tp2pp1     |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+

Understanding the Output:

• tokens/s/gpu: Overall throughput efficiency — higher is better (agg: 3785.20 vs disagg: 3067.51 = agg is 23% faster).
• tokens/s/user: Per-request generation speed (inverse of TPOT).
• TTFT: Predicted time to first token (395.88ms for agg, 111.33ms for disagg prefill).
• concurrency: Total concurrent requests (e.g., 256 (=64x4) = batch size 64 x 4 replicas).
• parallel: Tensor/pipeline parallelism (tp=tensor parallel, pp=pipeline parallel).
• Result: For this 4B model, aggregated serving is more efficient due to lower overhead.

Note: model size matters. Disaggregated serving becomes advantageous with larger models (32B+) where the benefits of phase-specific optimization outweigh KV cache transfer overhead. For smaller models like this 4B example, aggregated serving is more efficient.

4. Examine the generated directory structure:

$ tree results_vllm/

results_vllm/
└── nvidia/
    └── Llama-3.1-Nemotron-Nano-4B-v1.1_isl4000_osl500_ttft600_tpot16_472853/
        ├── agg/
        │   ├── best_config_topn.csv
        │   ├── config.yaml
        │   ├── pareto.csv
        │   ├── top1/
        │   │   ├── generator_config.yaml
        │   │   ├── k8s_deploy.yaml
        │   │   └── run_0.sh
        │   ├── top2/
        │   │   └── ...
        │   └── top3/
        │       └── ...
        ├── disagg/
        │   ├── best_config_topn.csv
        │   ├── config.yaml
        │   ├── pareto.csv
        │   ├── top1/
        │   │   ├── generator_config.yaml
        │   │   ├── k8s_deploy.yaml
        │   │   └── run_0.sh
        │   └── top2/
        │       └── ...
        └── pareto_frontier.png

The results are organized by model name and experiment parameters. While aggregated serving is optimal for this 4B model, the following sections demonstrate disaggregated deployment for learning purposes and larger model scenarios.

3. Deploy on Kubernetes

Note: while AIConfigurator determined that aggregated serving is optimal for this 4B model, the following sections demonstrate disaggregated deployment for learning purposes. Use disaggregated architecture when deploying larger models (32B+) or when AIConfigurator recommends it for your specific workload.

Kubernetes deployment provides the infrastructure for disaggregated serving with RDMA-accelerated KV cache transfer.

Install the NVIDIA Dynamo Platform

Before deploying disaggregated workloads, install the NVIDIA Dynamo platform (the Dynamo Operator) on your Kubernetes cluster, configure GPU resource management, set up the DynamoGraphDeployment CRD, and verify the cluster is ready for Dynamo workloads.

Prepare the Kubernetes Environment

1. Set your Kubernetes namespace as an environment variable. Replace YOUR_NAMESPACE with the namespace where you want to deploy Dynamo workloads:

$ export LLM_NAMESPACE=YOUR_NAMESPACE

2. Create the namespace:

$ kubectl create namespace $LLM_NAMESPACE

3. Export your Hugging Face token. Replace YOUR_HF_TOKEN with your Hugging Face access token:

$ export HF_TOKEN=YOUR_HF_TOKEN

4. Create a Hugging Face token secret for accessing gated models:

$ kubectl create secret generic hf-token-secret \
  --namespace $LLM_NAMESPACE \
  --from-literal=HF_TOKEN="$HF_TOKEN"

5. Create a persistent volume claim for caching model weights:

$ cat << EOF > model-cache-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
  namespace: $LLM_NAMESPACE
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
EOF

Note: this PVC uses ReadWriteMany (RWX) access mode, which allows multiple pods to mount the volume simultaneously for shared model caching. Ensure your cluster has a StorageClass that supports RWX access mode (such as NFS, CephFS, or cloud-provider RWX storage). Verify available StorageClasses:

$ kubectl get storageclass

6. Apply the PVC configuration:

$ kubectl apply -f model-cache-pvc.yaml

7. Verify the RDMA device plugin is running:

$ kubectl get pods -n kube-system | grep rdma

The output should show RDMA device plugin pods running on GPU nodes. If not installed, follow the Mellanox RDMA device plugin installation guide.

8. Verify RDMA resources are available on your nodes. Replace NODE-NAME with your GPU node name:

$ kubectl get node NODE-NAME -o json | jq '.status.allocatable'

The output should show rdma/hca_shared_devices resources:

{
......
  "rdma/hca_shared_devices": "1k",
......
}

The rdma/hca_shared_devices: "1k" entry confirms RDMA resources are available. If this resource is missing, verify the RDMA device plugin DaemonSet configuration and node labels.

Deploy Using Generated Manifests

1. Review the generated disaggregated deployment manifest:

$ cat results_vllm/nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1_isl4000_osl500_ttft600_tpot16_*/disagg/top1/k8s_deploy.yaml

The file contains a DynamoGraphDeployment custom resource with frontend, prefill workers, and decode workers configured according to AIConfigurator's recommendations.

2. Create the complete deployment manifest with RDMA configuration:

$ cat << EOF > dynamo-disagg-deployment.yaml
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: dynamo-disagg
  namespace: $LLM_NAMESPACE
spec:
  backendFramework: vllm
  pvcs:
    - name: model-cache
      create: false
  services:
    Frontend:
      componentType: frontend
      replicas: 1
      volumeMounts:
        - name: model-cache
          mountPoint: /opt/models
      envs:
        - name: HF_HOME
          value: /opt/models
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
          imagePullPolicy: IfNotPresent

    VLLMPrefillWorker:
      envFromSecret: hf-token-secret
      componentType: worker
      subComponentType: prefill
      replicas: 2
      resources:
        limits:
          gpu: "1"
      sharedMemory:
        size: 16Gi
      volumeMounts:
        - name: model-cache
          mountPoint: /opt/models
      envs:
        - name: HF_HOME
          value: /opt/models
        - name: POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: UCX_TLS
          value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc"
        - name: UCX_RNDV_SCHEME
          value: "get_zcopy"
        - name: UCX_RNDV_THRESH
          value: "0"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
          workingDir: /workspace
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          resources:
            limits:
              rdma/hca_shared_devices: "1"
            requests:
              rdma/hca_shared_devices: "1"
          command: ["python3", "-m", "dynamo.vllm"]
          args:
            - --model
            - "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1"
            - "--tensor-parallel-size"
            - "1"
            - "--kv-cache-dtype"
            - "fp8"
            - "--max-num-seqs"
            - "1"
            - --trust-remote-code
            - --is-prefill-worker

    VLLMDecodeWorker:
      envFromSecret: hf-token-secret
      componentType: worker
      subComponentType: decode
      replicas: 2
      resources:
        limits:
          gpu: "1"
      sharedMemory:
        size: 16Gi
      volumeMounts:
        - name: model-cache
          mountPoint: /opt/models
      envs:
        - name: HF_HOME
          value: /opt/models
        - name: POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: UCX_TLS
          value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc"
        - name: UCX_RNDV_SCHEME
          value: "get_zcopy"
        - name: UCX_RNDV_THRESH
          value: "0"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
          workingDir: /workspace
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          resources:
            limits:
              rdma/hca_shared_devices: "1"
            requests:
              rdma/hca_shared_devices: "1"
          command: ["python3", "-m", "dynamo.vllm"]
          args:
            - --model
            - "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1"
            - "--tensor-parallel-size"
            - "1"
            - "--kv-cache-dtype"
            - "fp8"
            - "--max-num-seqs"
            - "1024"
            - --trust-remote-code
            - --is-decode-worker
EOF

Critical RDMA Configuration:

• rdma/hca_shared_devices resources: Request RDMA resources matching tensor parallelism size (1 for both prefill and decode with TP1).
• IPC_LOCK capability: Required for RDMA memory registration.
• POD_UID environment variable: Retrieves pod UID from metadata for worker identification.
• UCX_TLS environment variable: Enables RDMA transports (rc_x, dc_x for InfiniBand).
• UCX_RNDV_SCHEME=get_zcopy: Enables zero-copy RDMA transfers for maximum performance.
• UCX_RNDV_THRESH=0: Forces RDMA usage for all message sizes.
• --trust-remote-code flag: Allows execution of custom model code from Hugging Face.
• CUDA 13 runtime: Uses nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 for GPU compatibility.

3. Deploy the configuration to Kubernetes:

$ kubectl apply -f dynamo-disagg-deployment.yaml

4. Monitor the deployment progress:

$ kubectl get pods -n $LLM_NAMESPACE -w

Wait for all pods to reach Running status. This may take several minutes as models are downloaded and loaded.

Verify the Kubernetes Deployment

1. Check pod status and resource allocation:

$ kubectl get pods -n $LLM_NAMESPACE

The output shows all pods running successfully:

NAME                                      READY   STATUS    RESTARTS   AGE
dynamo-disagg-0-frontend-77bn5            1/1     Running   0          2m42s
dynamo-disagg-0-vllmdecodeworker-gzgbw    1/1     Running   0          2m42s
dynamo-disagg-0-vllmdecodeworker-kn8xz    1/1     Running   0          2m42s
dynamo-disagg-0-vllmprefillworker-2b7hm   1/1     Running   0          2m42s
dynamo-disagg-0-vllmprefillworker-7p4mq   1/1     Running   0          2m42s

2. Verify RDMA is active by checking worker logs. Replace PREFILL-WORKER-POD with your prefill worker pod name from the previous step:

$ kubectl logs -n $LLM_NAMESPACE PREFILL-WORKER-POD | grep -i "UCX\|NIXL"

The output shows NIXL initialization logs:

2026-03-03T22:52:23.152441Z  INFO nixl_connector: NIXL is available
2026-03-03T22:52:23.154049Z  INFO factory.create_connector: Creating v1 connector with name: NixlConnector and engine_id: e0dc9d5a-80b5-4b06-aeb9-05958f83d6ea
(Worker pid=1241) 2026-03-03 22:52:25 NIXL INFO    _api.py:363 Backend UCX was instantiated
(Worker pid=1241) 2026-03-03 22:52:25 NIXL INFO    _api.py:253 Initialized NIXL agent: 93612abb-847f-435f-bd56-73bfc8d5f3d6
2026-03-03T22:52:25.487423Z  INFO nixl_connector.register_kv_caches: Registering KV_Caches. use_mla: False, kv_buffer_device: cuda, use_host_buffer: False

The key indicators confirm RDMA is working:

• "NIXL is available": NIXL connector initialized successfully.
• "Backend UCX was instantiated": UCX backend (RDMA transport) is active.
• "Initialized NIXL agent": NIXL agent ready for KV cache transfer.
• "Registering KV_Caches": KV cache buffers registered for GPU-to-GPU transfer.

3. Test inference through the frontend service:

$ kubectl port-forward -n $LLM_NAMESPACE service/dynamo-disagg-frontend 8000:8000

In a separate terminal, send a test request:

$ curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
    "messages": [{"role": "user", "content": "Explain the importance of tensor parallelism in LLM inference."}],
    "max_tokens": 100
  }'

The output displays the model's response in JSON format.

4. Monitor worker metrics to verify disaggregated flow:

$ curl -s localhost:8000/metrics | grep dynamo_frontend_requests_total

The output displays the metrics showing successful request processing:

# HELP dynamo_frontend_requests_total Total number of LLM requests processed
# TYPE dynamo_frontend_requests_total counter
dynamo_frontend_requests_total{endpoint="chat_completions",model="nvidia/llama-3.1-nemotron-nano-4b-v1.1",request_type="unary",status="success"} 1

4. Deploy on Standalone Instances

For standalone instance deployments (non-Kubernetes), refer to the backend-specific deployment guides that cover disaggregated serving:

• Deploy vLLM Disaggregated Serving: 2 prefill workers + 2 decode workers with NIXL-based KV cache transfer.
• Deploy TensorRT-LLM Disaggregated Serving: TRT-LLM disaggregated configuration with KV cache optimization.
• Deploy SGLang Disaggregated Serving: SGLang disaggregated setup with bootstrap_info transfer.

These guides provide complete deployment scripts with environment variable configuration, container runtime setup with GPU allocation, NIXL configuration for inter-worker KV cache transfer, and verification/troubleshooting steps.

5. Validate Performance with AIPerf

After deployment, validate AIConfigurator's predictions against actual performance using AIPerf.

1. Install the AIPerf benchmarking tool:

$ pip3 install aiperf

2. Derive AIPerf parameters from the AIConfigurator output. Values used for testing:

• --concurrency 30 (AIConfigurator recommends 208 for max throughput)
• --num-requests 800 (Recommended: 8320 = concurrency x 40 for statistical stability)
• --isl 4000 --osl 500 (Match your AIConfigurator inputs)
• --extra-inputs "ignore_eos:true" (Ensures exact OSL tokens generated)

3. Run the AIPerf benchmark against the deployment:

$ aiperf profile \
  -m nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 \
  --endpoint-type chat \
  -u http://localhost:8000 \
  --isl 4000 --isl-stddev 0 \
  --osl 500 --osl-stddev 0 \
  --num-requests 800 \
  --concurrency 30 \
  --streaming \
  --extra-inputs "ignore_eos:true" \
  --num-warmup-requests 40 \
  --ui-type simple

Note: the command above uses --concurrency 30 and --num-requests 800 for early success and faster testing. For best throughput validation, use the recommended values: --concurrency 208 and --num-requests 8320 (concurrency x 40 for statistical stability).

The benchmark runs 800 requests with 30 concurrent connections. The output displays performance metrics:

                                         NVIDIA AIPerf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃                               Metric ┃      avg ┃      min ┃      max ┃      p99 ┃      p90 ┃      p50 ┃      std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│             Time to First Token (ms) │   496.77 │    55.53 │ 8,411.38 │ 6,972.51 │    64.59 │    59.78 │ 1,606.66 │
│            Time to Second Token (ms) │     0.24 │     0.00 │     4.72 │     3.55 │     0.00 │     0.00 │     0.81 │
│      Time to First Output Token (ms) │   496.77 │    55.53 │ 8,411.38 │ 6,972.51 │    64.59 │    59.78 │ 1,606.66 │
│                 Request Latency (ms) │ 2,367.45 │ 1,505.01 │ 9,807.65 │ 8,869.04 │ 1,996.11 │ 1,972.56 │ 1,528.95 │
│             Inter Token Latency (ms) │     3.75 │     2.74 │     3.90 │     3.89 │     3.85 │     3.83 │     0.25 │
│     Output Token Throughput Per User │   268.03 │   256.55 │   365.49 │   357.48 │   279.88 │   260.84 │    22.84 │
│                    (tokens/sec/user) │          │          │          │          │          │          │          │
│      Output Sequence Length (tokens) │   499.53 │   487.00 │   510.00 │   502.00 │   500.10 │   500.00 │     1.30 │
│       Input Sequence Length (tokens) │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │     0.00 │
│ Output Token Throughput (tokens/sec) │ 6,232.31 │      N/A │      N/A │      N/A │      N/A │      N/A │      N/A │
│    Request Throughput (requests/sec) │    12.48 │      N/A │      N/A │      N/A │      N/A │      N/A │      N/A │
│             Request Count (requests) │   800.00 │      N/A │      N/A │      N/A │      N/A │      N/A │      N/A │
└──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

Benchmark Duration: 64.12 sec

4. Compare actual vs predicted performance:

The per-GPU throughput (6,232.31 / 2 GPUs = 3,116 tokens/s/gpu) closely matches AIConfigurator's prediction of 3,067.51 tokens/s/gpu. The total throughput (6,232.31 tokens/s) actually exceeds the prediction for 2 GPUs (6,135.02 tokens/s). This benchmark was run with 2 GPUs (1 prefill + 1 decode worker); scaling to 4 GPUs (2 prefill + 2 decode workers as shown in the deployment manifest) would double the throughput to approximately 12,464 tokens/s. The lower throughput compared to maximum capacity is due to using concurrency 30 instead of the recommended 208. The Inter Token Latency (3.75ms) significantly outperforms predictions, demonstrating efficient token generation.

Note: run benchmarks inside the Kubernetes cluster (using a Job pod) to eliminate network latency. External benchmarking adds latency that affects TTFT measurements but provides realistic end-to-end performance metrics.

Configuration Best Practices

GPU Allocation Strategies

Prefill Workers:

• Allocate fewer GPUs with higher memory bandwidth (memory-bound workload).
• Use lower tensor parallelism (TP1 or TP2) to minimize communication overhead.
• Deploy multiple replicas for higher prefill throughput.

Decode Workers:

• Allocate more GPUs for parallel token generation (compute-bound workload).
• Use higher tensor parallelism (TP4 or TP8) to distribute computation.
• Single replica with high batch size often performs best.

Example Configuration (4 GPUs):

• 2x Prefill workers with TP1 (2 GPUs total, 1 GPU each).
• 2x Decode workers with TP1 (2 GPUs total, 1 GPU each).
• Achieves balanced prefill and decode capacity with minimal tensor parallelism overhead.

Tensor Parallelism Considerations

Batch Size Tuning

Prefill Workers:

• Use low batch size (1-4) for optimal TTFT.
• Higher batch sizes increase memory pressure and TTFT latency.
• Prefill throughput comes from multiple workers, not high batch size.

Decode Workers:

• Use high batch size (256-1024) for optimal throughput.
• Decode operations are compute-bound and benefit from high concurrency.
• Adjust based on available GPU memory and model KV cache requirements.

RDMA Requirements

Network Requirements:

• InfiniBand (preferred) or RoCE network with RDMA support.
• Sufficient bandwidth: 200 Gbps+ recommended for optimal performance.
• Low latency: <10μs for RDMA operations.

Verification:

• Check worker logs for "NIXL is available" and "Backend UCX was instantiated" messages.
• Monitor RDMA device utilization: rdma resource show.
• Measure RDMA bandwidth: ib_write_bw or rdma_bw tools.

Next Steps

• Re-run AIConfigurator against a 32B+ model to see disaggregated serving's throughput advantage materialize.
• Scale prefill and decode replicas independently based on observed ISL/OSL ratios in production traffic.
• Run AIPerf with the full recommended concurrency (208) and request count (8320) for a production-grade validation.
• Review the backend-specific deployment guides (vLLM, TensorRT-LLM, SGLang) if you need a standalone, non-Kubernetes setup.

For the full guide with additional tips, visit the original article on Vultr Docs.