How to run DeepSeek's DeepInfer: build, deterministic outputs and JSON schema

DeepSeek open-sourced DeepInfer, the inference framework behind its large-scale production services. This release does offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.

This guide covers the three things most engineers will want first: getting a build running, making outputs repeatable for regression tests, and forcing output into a JSON schema. Everything below comes from the repo's README.

Before you start: what this release is and is not

It is offline inference with optimized kernels, attention DP and MoE expert parallelism, draft-token decoding, asynchronous scheduling, batch-invariant inference, deterministic sampling and structured outputs.

It is not a complete serving stack. The README lists what you would still need: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. Plan accordingly.

Prerequisites: Linux, GCC 13.0 or newer, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend).

Step 1: Build from source

Create and activate an empty Python 3.12 environment first. Then:

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Step 2: Run offline inference

python examples/offline_inference.py

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, downloads into the Hugging Face cache on the first run. To use a local copy instead:

export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017

The repo's examples/README.md has the details of each example.

Step 3: Make outputs repeatable for regression tests

Before and after comparisons are useless if outputs drift with batch composition or sampling randomness. DeepInfer has two switches for that, both NVIDIA only:

export DEEPINFER_BATCH_INVARIANT=1
export DEEPINFER_ENABLE_MTP_VARLEN=0

Then pass a fixed seq_seed to SamplingParams in your script. Same input tokens plus same sampling parameters now give the same output. A simple check: run the same prompt file twice and diff the two outputs. An empty diff is a pass.

Note the tradeoff: DEEPINFER_ENABLE_MTP_VARLEN=1 is what turns on the draft-token decoding speedup, so deterministic runs give that up.

Step 4: Force output to match a JSON schema

Structured outputs constrain generation to a regex, a JSON schema or an EBNF grammar. Pass them through SamplingParams:

so_args={"schema_type": ..., "schema": ...}

Start from examples/_structured_output.py in the repo, which shows how the constraint is built, then parse the returned text with json.loads and check the required keys.

Step 5 (optional): Reproduce the published benchmarks

The README's prefill and decode charts were measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. The sweep script is at examples/sweep/run_sweep.sh. On a smaller setup your numbers will not match the charts directly, so compare shapes, not absolutes.

Use cases with copy-paste agent prompts

• Run offline inference on DeepSeek-V4.1-Flash locally
• Get repeatable model outputs for regression tests
• Force model output to match a JSON schema
• Reproduce DeepInfer prefill and decode benchmarks

DeepInfer is released under the Apache License 2.0.

Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com