Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 8 min read

SGLang vs vLLM: what is really different inside, and how to compare them fairly

Both engines share almost every big idea in modern LLM serving. The real differences are in how they reuse the KV cache, and most published benchmarks change more than one thing at a time. Here is how I would explain it

SGLang vs vLLM: what is really different inside, and how to compare them fairly

Both engines share almost every big idea in modern LLM serving. The real differences are in how they reuse the KV cache, and most published benchmarks change more than one thing at a time. Here is how I would explain it to a new engineer on the team.

Once a team decides to serve an open model on its own GPUs, the next question is almost always the same: should we run SGLang or vLLM? Most of the answers you find are a single benchmark chart from one team's blog, with no word on how it was measured. That is not much to base a production choice on.

In this post I go through what the two engines share, where they really differ inside, how to compare them without fooling yourself, and how to choose. One thing up front: every measured number below is a vLLM number, from vLLM 0.33 on a rented RTX 4090 in our course labs. We have not yet run SGLang in the course, so I make no speed claim about it. Where I describe SGLang, it is what its own project describes.

What the two engines are

Both are model servers. A model server is a program that loads a language model onto one or more GPUs, keeps it in memory, and answers many people's requests at the same time. Both are free and open source, both are written mainly in Python with fast GPU code underneath, and both answer in the same format as OpenAI's API.

vLLM came out of the University of California, Berkeley, together with the 2023 paper that introduced PagedAttention. SGLang came out of the LMSYS research group and introduced RadixAttention, a way of reusing the work for prompts that start the same way. The versions our course teaches, checked on 12 November 2026, are vLLM 0.33 and SGLang 0.5.23.

They have more in common than the arguments suggest

Almost every big idea in modern serving is in both engines:

  • Continuous batching, where requests join and leave the batch at every step instead of waiting for the whole batch to finish.
  • Paged storage of the KV cache, the notes the model keeps for every token of every conversation.
  • Reuse of shared prompt beginnings, so a prompt the server has already read is not read again.
  • Chunked prefill, which reads a long prompt in pieces so it does not hold up everyone else.
  • Speculative decoding, where a fast guesser proposes tokens that the large model checks in one pass.
  • Tensor parallelism across several GPUs, quantized models, and answers forced to follow a JSON schema.

So the choice is rarely about a missing feature. It is about how each engine performs on your workload, and how comfortable your team is running it.

Can you switch without changing your code?

Mostly, yes. Both serve OpenAI-compatible addresses such as /v1/chat/completions, so the OpenAI Python library works with either once you change the address and the model's name. In our vLLM run, the library sent a question to Qwen3-0.6B on an RTX 4090 and got the first word back after 0.167 seconds, with the token counts in the usual usage field.

What does change is the server side: the command that starts it, the names of its settings, and some extra request fields each engine accepts beyond OpenAI's own. Test those before you switch a live service.

The real difference: how each one reuses a prompt it has read before

Both engines keep the notes of a prompt beginning they have already read, so the next request that starts the same way skips that work. They keep the books differently.

A hash chain of page labels: each full page of 16 tokens is labelled with a hash of the label before it plus its own tokens. In the first question all three pages are worked out and kept. In the next question a few tokens near the start differ, so every page after them gets a new label and none is found, even a page whose own tokens are the same.

How vLLM finds a prompt it has read before: a chain of page labels.

vLLM gives each full page of 16 tokens a label made from the label of the page before it and its own tokens. When a new prompt arrives, it reuses pages from the start until the first page that differs, and stops looking there. Because each label depends on the one before, a change on page 1 changes the label of every page after it.

SGLang keeps the cached beginnings in a radix tree, a tree in which requests that share a beginning share a branch, and looks up the longest matching branch.

The goal is the same, and the effect can be large. vLLM's scheme made our first word about 7 times faster on a hit. Eight questions behind the same 2,958-token handbook found 87% of their prompt in the cache, and the first word came after about 46 ms instead of 326 ms.

A line chart of the wait for the first word, question by question, from four runs on the rented RTX 4090. With nothing found the wait stays near 326 to 330 ms. With the handbook found it drops from the second question onwards to about 46 ms, and 37 ms with the time at the end of the system message.

Reuse made the first word about 7 times faster in our vLLM runs.

What breaks reuse in either engine

Anything that changes near the start of the prompt: a timestamp, a request number, a user's name, or a shuffled list of tools. Both engines can only reuse a beginning that matches token for token.

In our vLLM run, a clock time at the start of the system message cut the hit rate from 87% to 0%. The same line at the end of the system message gave 86%. Put the parts that never change first, whichever engine you choose.

Both store the KV cache in pages

Neither engine reserves room for the longest possible conversation, which would waste most of the memory. vLLM's PagedAttention uses pages of 16 tokens by default, listed for each conversation in a block table. SGLang keeps cache entries in a shared memory pool indexed by its radix tree.

On our RTX 4090, vLLM made 2,076 pages for Qwen3-8B. Sixty-four conversations of about 340 tokens used only 66.8% of them at once, where reserving 32,768 tokens for each would have fitted just one.

How the main settings map

The ideas match even where the names do not:

What it controls vLLM SGLang
Share of GPU memory the engine may use --gpu-memory-utilization (0.92 by default in vLLM 0.33) --mem-fraction-static
Most conversations running at once --max-num-seqs --max-running-requests
Token budget of one step --max-num-batched-tokens --chunked-prefill-size

Both also have a scheduling policy setting. Defaults differ between the engines and between versions, so read each engine's start-up log to see how much room it gave the KV cache before you compare anything.

Structured output and speculative decoding

Both can force the model's answer to follow a JSON schema or a grammar at serving time, by allowing only the tokens that keep the answer valid. Both can also parse tool calls from the model's output into OpenAI's tool-call format. SGLang also comes with its own frontend language, a way of writing programs that make several model calls with branches and forced formats, which is where its prefix tree pays off. vLLM keeps to the server and the OpenAI-style API, with many settings for parsers and backends.

Both support speculative decoding. vLLM 0.33 accepts methods from simple n-gram matching to a draft model, EAGLE-3 heads and multi-token prediction, and SGLang supports EAGLE-style guessing too. It helps most when few people are using the GPU. On a busy server the extra checking work can make things slower, so measure it at your real load.

How to compare them fairly

Most comparisons online are not wrong on purpose. They just change more than one thing at a time. These are the rules I would hold any benchmark to, including my own.

Keep everything else the same. The same model files, the same number format, the same GPU and the same driver. Give both the same share of GPU memory and check in each log how much room the KV cache really got. Send the same workload: the same prompt and answer lengths, the same number of people at once or the same arrival rate, and the same sampling settings.

Use fresh prompts in every run. Both engines remember prompts they have read. A test that sends the same questions again measures the prefix cache, not the GPU; our first-word wait fell from 326 ms to 46 ms on a hit. Give every run its own made-up questions, unless reuse is the thing you are measuring, and in vLLM check that vllm:prefix_cache_hits_total stays at 0 during the test.

Compare second starts, not first starts. vLLM compiles the model and measures its working space during the first start. On our card, a first start of Qwen3-8B got 33,216 tokens of KV cache, and the next start with the same settings got 38,000. If one engine is measured cold and the other warm, you are partly comparing start-up work.

Send enough requests for the slow end to mean something. The p99 is the value 99 in 100 requests beat, so with a few dozen requests it is simply the slowest one. In our vLLM test with 48 answers, that is exactly what the p99 was. Use hundreds of requests per level, say how many each percentile came from, and check that the failed request count is 0. Know what your tool sends, too: vLLM 0.33's vllm bench serve sends no temperature unless you add one.

Report more than one number. Warm both servers up, run each test several times, and report the spread as well as the best run. Compare the median and p99 wait for the first word, the median and p99 gap between words, and total tokens a second, all at the same load. One number on its own hides the trade.

So which one should you pick?

vLLM is the safer default when you want the widest set of supported models, settings and integrations, and the most material to learn from. Many other systems build on it: llm-d, for example, runs vLLM underneath its routing layer on Kubernetes.

SGLang is worth a test when your traffic reuses long beginnings heavily: agents that resend the same long instructions and history on every turn, many branches from one prompt, or programs that make several structured calls in a row. That is the kind of work RadixAttention was designed for. If it keeps your promises at a lower cost, the switch is mostly a server change, because your client code speaks the same OpenAI-style API.

Learn the ideas before the engine. Prefill and decode, the KV cache, batching, paging and prefix reuse carry over from one engine to the other. Learn one engine deeply, and once you know why each of its settings exists, the other engine's settings read like the same ideas with different names.

The full guide, with every question and the lesson each number comes from, is on our site: SGLang vs vLLM: what is different, and how to choose.

Go deeper: Hands-On Inference Engineering

These numbers come from a course I teach on System Design Academy, Hands-On Inference Engineering. You rent the GPU yourself, serve a model with vLLM, learn every setting that changes a number, and then run a fair head-to-head benchmark against SGLang on the same GPU. The SGLang part is still being written, which is why this post makes no speed claim about it. It is written for backend and AI Engineering teams who want to serve their own models: see the Hands-On Inference Engineering course.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.