vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

TL;DR

• What shipped: vLLM 0.33.0 (PyPI, 2026-11-02) includes FLy, an opt-in speculative decoding verifier that deliberately accepts some draft tokens the target model would have rejected. The PR states it "does not preserve the target distribution and may reduce output quality." Standard verification stays the default.
• What the PR reports: decode throughput 1.041x to 1.310x over standard speculative decoding at all 48 model, platform and batch points, with minimum accuracy retention of 96.3% against target-only decoding.
• What I computed from the PR's own tables (not a new benchmark): across the 32 accuracy cells, standard speculative decoding, which is lossless by design, lands a mean 1.68 points away from target-only. FLy lands 1.80 points away. The tables measure run-to-run noise at least as much as they measure FLy.
• What a real quality gate needs: a paired per-question comparison. With 6% of questions flipping either way, detecting a 2-point net loss at 80% power takes about 1,175 questions. The PR's evaluation had 428 per model and platform.
• The bigger win in the same release is an accounting fix: a scheduler change for DSpark and DFlash drafters took one reported run from 1,373 to 3,390 output tokens/s, and no kernel changed.

What does FLy change in speculative verification?

In standard speculative decoding, a small drafter proposes K tokens and the target model verifies them in one forward pass. Verification stops at the first rejected token, and the accept/reject rule is built so the output distribution matches what the target would have produced alone.

FLy, from the paper "Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match" (arXiv 2511.22972), relaxes the stop. Per PR #53987 (merged 2026-10-19), a rejected position is forgiven when three conditions hold:

1. The rejected draft token is not masked out by target constraints.
2. The target's entropy at that position meets a threshold, meaning the target itself was unsure.
3. The next W draft tokens all pass the normal verification rule.

If any of the next W tokens is rejected, the original rejection stands. A short trailing window is never deferred. The PR notes the entropy is computed over only the top-k target probabilities (default 3) without renormalizing, which differs from the paper's full-vocabulary version. It needs no extra checkpoint and no extra target forward pass.

Enabling it is one block in the speculative config:

from vllm import LLM

llm = LLM(
    model="Qwen/Qwen3-8B",
    speculative_config={
        "method": "draft_model",
        "model": "Qwen/Qwen3-0.6B",
        "num_speculative_tokens": 8,
        "rejection_sample_method": "fly",   # default is standard verification
        "fly_window_size": 6,
        "fly_entropy_threshold": 0.3,
        "fly_entropy_top_k": 3,
    },
)

Two details matter for reproducing the published numbers. fly_window_size defaults to min(6, num_speculative_tokens - 1), and FLy needs at least two speculative tokens. The PR's ModelRunner V2 measurements, though, were run with window=2. A DeepSeek setup with K=7 would default to a window of 6, so the default is not the configuration that was benchmarked. The engine also logs a one-time warning that the method is lossy.

What do the published speed numbers show?

The PR's multi-batch table covers four models: DeepSeek-V4-Flash with DSpark (K=7), Qwen3.8-27B with MTP (K=3), GLM-5.2-FP8 with MTP (K=5) and Qwen3.5-122B-A10B-FP8 with MTP (K=8). They ran on AMD MI355X and NVIDIA B300 at batch sizes 1, 4, 8, 16, 32 and 64, with 1,024 generated tokens per request and a fixed pool of 64 prompts.

I parsed all 48 rows. The speedup range matches the PR text (1.041x to 1.310x), and the median is 1.147x. Averaged over the six batch sizes:

The speedup tracks the acceptance gain closely: the correlation across the 48 points is r = 0.82. That is what you would expect if FLy converts "extra accepted tokens" into "fewer target passes" with little other overhead. It also tells you where the method pays. A model whose drafter already agrees with it (Qwen3.8 at K=3, only +7.6 to +10 points) gains 6 to 8%. A model with a weaker drafter (Qwen3.5-122B, +16 to +17 points) gains 24 to 28%. A flat 1.2x is not a safe planning number for your own pair.

Can the accuracy tables show that FLy is safe?

Not by themselves, and the PR does not claim they can. It says the retention figures "are single-pass measurements, not a quality guarantee." The question is how much they can say, so I rebuilt the accuracy table.

Each model and platform was scored on GSM8K (first 100 questions), MATH-500 (first 100), AIME25 (30) and GPQA-Diamond (198), which is 428 questions, in three decoding modes: target-only, Standard SD and FLy. That is 2 platforms x 4 models x 4 datasets = 32 cells. Standard SD is distribution-preserving by construction, so any gap between it and target-only is noise from the run itself (batched floating-point differences, sampling, a single pass). Here is the mean absolute gap to target-only in each cell, grouped by dataset:

The largest lossless gap is DeepSeek-V4-Flash on B300 for AIME25: target 26/30, Standard SD 28/30. The largest FLy gap is Qwen3.5-122B on MI355X for GSM8K: target 82, FLy 92 out of 100, where the lossy method scored 10 points better. Neither number is evidence about the method. They show that with 30 to 198 questions per cell, one cell can move by several points for reasons unrelated to the verifier.

Pooling helps a little. Summed over all 8 model and platform pairs (3,424 scored answers per decoding mode):

FLy is 19 answers above target-only, and the lossless method is 27 above it. A lossless method cannot genuinely beat the target on average, so the 0.8-point spread is the noise scale of this setup. A rough 95% interval for the difference between two unpaired accuracies near 90% is about plus or minus 4.0 points at 428 questions, 5.9 at 198, 8.3 at 100 and 15 at 30 (normal approximation, poor at n=30 near the ceiling). A real 1 to 2 point quality loss would sit inside all of those bands.

None of this says FLy hurts quality. The evidence is consistent with a small loss and with none, and the PR's minimum retention (96.43% on AMD, 96.30% on NVIDIA) is a one-problem difference on AIME25, where each question is 3.3 points.

How many questions does a real quality gate need?

The fix is a paired design. Run both configurations on the same questions with the same seeds, record per-question correctness, and count only the discordant pairs: questions FLy lost and questions it gained. McNemar's exact test then asks whether the net movement is bigger than chance. Pairing removes the between-question variance that makes the unpaired interval so wide.

import math

def paired_gate(base, cand):
    """base, cand: per-question 0/1 correctness for the SAME questions and seeds."""
    lost = sum(1 for x, y in zip(base, cand) if x and not y)
    gained = sum(1 for x, y in zip(base, cand) if y and not x)
    n, k = lost + gained, min(lost, gained)
    p = min(1.0, 2 * sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n) if n else 1.0
    return {"lost": lost, "gained": gained, "net": gained - lost, "exact_p": round(p, 4)}

def questions_needed(net_loss, discordant, z_a=1.96, z_b=0.8416):
    # two-sided alpha 0.05, power 0.80; discordant = fraction of questions that flip either way
    return math.ceil((z_a * math.sqrt(discordant)
                      + z_b * math.sqrt(discordant - net_loss ** 2)) ** 2 / net_loss ** 2)

I ran this on synthetic data (illustrative: 428 questions at 89% base accuracy, with 5% of answers randomly flipped). It reported 25 lost, 2 gained, exact p below 0.0001. A deliberately damaged candidate is caught easily. The harder question is the subtle loss, and the sample size you need depends on how often the two runs disagree at all:

(Normal-approximation sample sizes from the formula above; the discordant fraction is an assumption you measure by running Standard SD twice on a pilot set.) At a 6% flip rate, a 2-point loss needs about 1,175 questions and a 1-point loss about 4,700. A 428-question suite per cell cannot resolve either, which is why a short eval is a smoke test and not a gate. Build the gate from your own task mix, because the entropy rule fires more often on open-ended generation than on short arithmetic answers, and your traffic is neither GSM8K nor GPQA.

What is the bigger win in the same release?

PR #59468 (merged 2026-10-15) fixes how the scheduler charges token budget for block-diffusion drafters. DFlash and DSpark were reserving draft slots in each request's share of max_num_batched_tokens even though they run their draft pass separately, not inside the target batch. For DSpark with K=5, a decode request was charged 10 tokens instead of 6. The PR sets the reserved slots to 0 and instead caps running requests at max_num_batched_tokens // num_lookahead_tokens, so the separate draft buffers still fit.

The reported end-to-end result (DeepSeek-V4.1-Flash with DSpark, TP4 on one GB300, --max-num-batched-tokens 512, one run per side):

Admission rose 1.23x and throughput 2.47x. The PR's explanation for the gap is scheduling, not drafting: on main, the 2 to 32 leftover tokens split each new prefill into small chunks, so most steps mixed prefill into the decode batch and missed the full decode CUDA graph (ITL p50 169 ms on main against 30 ms with the fix). Acceptance length is unchanged at 4.72 versus 4.73, and greedy outputs for 20 fixed prompts were byte-identical (3,197 completion tokens). The author reports that an earlier revision, measured with two interleaved runs per side, was within 2% of these numbers. DFlash was not run end to end.

The admission arithmetic is easy to check:

def max_running(budget, max_seqs, tokens_per_decode_request):
    return min(max_seqs, budget // tokens_per_decode_request)

for label, charge in (("before, K=5", 10), ("after, K=5", 6)):
    print(label, max_running(512, 64, charge), max_running(512, 128, charge))
# before, K=5 51 51
# after, K=5 64 85

Simple division gives 51 for the old charge where the PR observed 52, a remainder effect I did not trace. For the new charge with --max-num-seqs 128 it gives 85, which matches the PR exactly, and the new request cap of 102 (512 // 5) does not bind. You can see the cap in effect at startup: the log line reads like "The dspark draft pass needs 5 query tokens per request within max_num_batched_tokens=512. Overriding max_num_active_seqs from 128 to 102."

The operational lesson generalizes beyond drafters. If you run speculative decoding and your running-request count plateaus well below --max-num-seqs under saturating load, token-budget accounting is binding and the number you set is not the number you get:

curl -s localhost:8000/metrics | grep '^vllm:num_requests_running'

What else in 0.33 touches a speculative serving stack?

Medusa and the MLP speculator were removed (#61701), lookahead block hashes for EAGLE and MTP prefix-cache hits are now on by default (#61589), CUDA 12.9 images are no longer published, and the NIXL connector needs NIXL 1.5 or newer. Read those before upgrading a fleet that uses drafters.

A rollout checklist for FLy

1. Leave standard verification on wherever outputs feed an unreviewed downstream decision. That is my recommendation, not the PR's, and the reason is the lack of a distribution guarantee.
2. Reproduce the configuration that was measured, including fly_window_size, or measure your own window.
3. Build a paired eval from your traffic, with at least 1,000 questions to resolve a 2-point loss, fixed seeds, and per-question results stored so you can run the exact test.
4. Track acceptance, not only tokens/s. The speedup follows the acceptance gain, so a workload where the draft already agrees will show little benefit.
5. Alert on the one-time lossy warning in logs so a config drift does not turn the method on silently.

I did not run FLy myself. Every number above comes from the PR authors' tables or from arithmetic on them, and the throughput results are their measurements on their hardware.

FAQ

Is FLy on by default in vLLM 0.33?
No. Standard verification is the default, and FLy requires rejection_sample_method set to "fly" in the speculative config, with at least two speculative tokens.

Does FLy preserve the target model's output distribution?
No. The PR says it does not and that it may reduce output quality. Standard speculative decoding does preserve the distribution.

How much faster is it?
The PR reports 1.041x to 1.310x over standard speculative decoding across 48 model, platform and batch combinations, with a median of 1.147x in my parse of its table. The gain follows the draft acceptance gain, so it varies by model pair.

Do the published accuracy results show no quality loss?
They show no loss that a single run on 30 to 198 questions per dataset could detect. In the same tables, the lossless method differs from target-only by a mean 1.68 points, close to FLy's 1.80, so the noise is of similar size to any plausible effect.

How many questions do I need to detect a 2-point accuracy loss?
About 1,175 paired questions at 80% power and 6% of questions flipping either way, rising to about 1,570 at 8%. Use a paired exact test on per-question results.

Does the scheduler fix need a config change?
No. For DFlash and DSpark the scheduler lowers the active-request cap at startup when needed and logs it, and it does not alter --max-num-seqs itself.

Further reading

• Speculative Decoding at High Batch Size: Where the 3x Speedup Turns Into a Tax
• Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data

Sources

• vLLM v0.33.0 release notes: https://github.com/vllm-project/vllm/releases
• FLy pull request #53987: https://github.com/vllm-project/vllm/pull/53987
• Scheduler budget pull request #59468: https://github.com/vllm-project/vllm/pull/59468
• Paper, "Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match": https://arxiv.org/abs/2511.22972