Introducing GPUStack v2.3: Optimizing LLM Inference with PD Disaggregation, Shared KV Cache, and Elastic Scheduling

The open-source edition of GPUStack v2.3.0 is now officially available!

Automated PD disaggregation orchestration, shared KV cache pools, vGPU partitioning, intelligent benchmarking, and scheduled autoscaling give model services more flexible resource configuration options. They also enable teams to deploy, validate performance, and continuously optimize services through a unified workflow.

These capabilities address practical challenges that arise when models move into production: once Prefill and Decode are separated, how should instances work together? After a service scales out, how can it reuse existing caches? And how many resources should be provisioned for traffic peaks?

GPUStack v2.3 connects the management of resources, instances, caches, and traffic, allowing teams to focus more on model and business optimization while reducing repetitive work in multi-node deployment and day-to-day operations.

A Layer of Systems Engineering Above the Inference Engine

Inference engines such as vLLM and SGLang continue to improve model execution efficiency while adding capabilities such as distributed inference, PD disaggregation, and routing. These advances form an important foundation for the model services delivered by GPUStack.

When enterprises adopt these capabilities, they often need to support multiple models, different inference backends, and heterogeneous GPUs at the same time. Beyond optimizing individual components, they must coordinate resource allocation, instance deployment, request routing, and configuration changes.

PD disaggregation, for example, requires teams to organize Prefill and Decode instances and determine the right ratio between them, route requests through the corresponding processing stages, and maintain coordination among components as replica counts change. Each of these factors affects whether the service can continue to operate reliably.

GPUStack incorporates this engineering work into the model service management workflow: users configure a service, and the platform organizes the underlying runtime components. Version 2.3 further connects deployment, benchmarking, and runtime management, making advanced inference architectures easier to put into production and continuously tune.

Automated PD Disaggregation Orchestration: Configuring Resources Separately for Each Inference Stage

Processing an LLM request primarily involves two stages: Prefill, which processes the input context, and Decode, which generates output tokens one at a time.

These stages have different workload characteristics, and their resource requirements vary with input length, output length, and request volume. Deploying them separately makes it possible to tune resources and replica ratios independently for different production workloads.

Version 2.3 introduces automated orchestration for PD-disaggregated deployments, supporting the independent deployment, scheduling, and scaling of Prefill and Decode. Users configure the resources and replica counts for both roles, while GPUStack coordinates instance deployment, request routing, and collaboration as replica counts change.

This allows teams to optimize input processing and output generation separately while reducing the effort required to maintain multi-instance configurations manually. After deployment, teams can also benchmark the service against actual hardware and request patterns to validate and adjust the P/D ratio.

Resources and replicas are configured separately for Prefill and Decode, while the platform organizes them into a single model service.

Shared KV Cache: Reducing Repeated Context Computation

System prompts, knowledge base materials, and multi-turn conversation histories are often repeated across requests. Reprocessing the same content on every instance results in redundant computation.

Version 2.3 introduces a shared KV cache pool that, with compatible models and configurations, enables cache reuse across replicas and nodes. Multiple instances can reuse previously computed context and reduce repetitive Prefill work.

GPUStack integrates shared caching solutions such as LMCache, Mooncake, and XSKY MeshFusion into model service management. Teams can configure caching at the service level and evaluate its effectiveness using cache hit rates and time to first token.

Cache reuse must also be considered alongside request distribution and instance load. When a service scales from one instance to multiple instances, the results of prior context computation become a reusable resource. Teams can assess the value of caching as they scale services and adjust routing.

Configuring a shared cache around a model service makes repeated context reusable across instances.

vGPU Partitioning: Allocating Resources to Match Workloads

Small models, development environments, and validation jobs do not necessarily need exclusive access to an entire physical GPU. Coarse-grained resource allocation can leave a portion of compute capacity idle for extended periods.

Version 2.3 introduces vGPU partitioning, supporting model and GPU instance deployment using both software-based and hardware-based resource partitioning. This allows different workloads to consume resources on demand. The available partitioning methods and isolation capabilities depend on the hardware and the corresponding solution.

Partitioning support for eight types of heterogeneous GPUs.

In Kubernetes environments, K8s workloads can directly request full, shared, or partitioned GPU resources provided by GPUStack Operator KDP.

This extends the resource services introduced in v2.2: the same compute management system supports both model inference and development or experimentation. From full GPUs to partitioned resources, teams can allocate compute according to workload requirements and give smaller workloads an appropriately sized resource profile. Instance monitoring then helps teams observe actual utilization and continue refining their configurations.

Both model deployments and GPU instances can use partitioned resources, allowing teams to choose an allocation method suited to each workload.

Intelligent Benchmarking and Scheduling: Aligning Service Capacity with Business Workloads

Once a model is running, teams still need to understand how many requests the current configuration can handle and what kind of response experience it can deliver.

A single throughput figure cannot adequately answer this question. As concurrency increases, both throughput and latency change. Differences in input and output lengths can also invalidate earlier test results.

Version 2.3 introduces intelligent performance benchmarking. A single test covers multiple concurrency levels and produces load curves and a recommended operating point. This helps teams explore the model's performance limits and determine service capacity based on the latency their applications can tolerate.

Use throughput and latency across multiple load levels to select an operating point that meets application response-time requirements.

With capacity data in hand, teams can then plan traffic distribution and resource provisioning.

Version 2.3 enhances gateway routing with load-balancing strategies such as session affinity and least requests, as well as model selection through a decision service. The former manages request distribution across instances, while the latter selects an appropriate model for each request, enabling teams to configure traffic strategies based on service characteristics.

For predictable peaks and troughs, the platform also provides time-based autoscaling with tidal scheduling. It adjusts model replica counts according to configured time windows, adding instances before recurring traffic peaks and reducing resource consumption during off-peak periods.

Adjust replica counts by time window to provision resources ahead of predictable traffic peaks.

For example, a long-context knowledge base service can use shared caching to reuse repeated materials, run benchmarks to identify an operating point that meets response-time requirements, and then configure routing and a pre-peak scale-out plan. Every step revolves around the same service, and the configuration and observations from one step inform adjustments in the next.

From “How many requests can the service handle?” to “How should requests be distributed, and when should resources be provisioned?”, performance data becomes the basis for day-to-day scheduling.

OTA and Deployment Management: Keeping Pace with Technology While Preserving a Recovery Path

Models and inference backends evolve rapidly, and production services need a controlled way to adopt these changes.

Version 2.3 introduces OTA updates for the model catalog and inference backends. Without upgrading or restarting GPUStack, teams can update the model catalog, available inference backend versions, and the community backend marketplace online, gaining timely access to new deployment options.

For individual services, teams can use YAML import and export to save and reuse deployment configurations, apply validated configurations to subsequent deployments and environment migrations, and use deployment version history and rollback to retain a record of configuration changes and restore a previous version when necessary.

Reuse configurations through YAML, trace changes through version history, and restore a specific configuration version when necessary.

OTA updates bring new capabilities to the platform promptly, while configuration reuse and version management allow teams to validate and adopt them at their own pace. From introducing new deployment options to reusing validated configurations and restoring previous versions, service updates now have a clearer operational path.

Turning Advanced Inference Capabilities into Model Services Built for Sustainable Operations

The value of an inference optimization is ultimately realized in a production service: are resources allocated appropriately, is the deployment architecture maintainable, has capacity been validated, and are application requests handled as expected?

GPUStack v2.3 connects PD disaggregation, shared caching, resource partitioning, and elastic scheduling through a unified service management workflow. Teams can choose an inference architecture based on application needs, validate its actual capacity, and continuously adjust resource and traffic configurations.

This is GPUStack's approach to systematic inference engineering: turning advanced inference capabilities into model services that can be delivered at scale and operated sustainably.

Try GPUStack v2.3 with a real-world workload and explore the deployment and operating model that best fits your needs. We also invite you to share your practical experience with the community.