DeepSeek launches DeepInfer, the inference engine used to run its large-scale models

DeepSeek releases DeepInfer, its high-performance inference engine, and deepseek-recipe, a library for reproducing its API formats.

Publishing the weights of an artificial intelligence model is not enough to replicate the performance of a service like DeepSeek. Between the model and the user lies an infrastructure responsible for distributing computations across accelerators, managing concurrent requests, and producing responses with controlled latency. It is part of this infrastructure that DeepSeek is now making public with DeepInfer, the inference engine it uses for its large-scale services. Distributed under the Apache 2.0 license, the framework currently supports DeepSeek-V4.1-Flash and combines several optimization techniques designed to reduce computational costs. Concurrently, the company is releasing deepseek-recipe, a library dedicated to encoding requests and rendering responses in the formats of its API. From Open Model to Its Execution

Language models are generally evaluated on their capabilities in reasoning, programming, or multimodal understanding. But once deployed, their performance also depends on the software that executes them. The inference engine must, in particular, organize operations on GPUs, use memory efficiently, and process multiple requests without unnecessarily multiplying wait times. This dimension becomes particularly important for large models using a mixture-of-experts architecture, whose computations are distributed across multiple accelerators.

DeepInfer brings together the software components that DeepSeek developed to meet these constraints. The project uses several libraries previously released by the company, including FlashMLA for attention operations, DeepGEMM for matrix computations, DeepEP for inter-expert communications, and DeepSelect for certain selection operations. It also integrates TileKernels and specialized kernels written with TileLang. These various elements are assembled into an environment that includes the model's execution engine, sampling management, and request scheduling.

The release extends the work carried out around DeepSeek-V4.1-Flash-1017, a recent version of the Flash model. For now, it is the only family explicitly supported by DeepInfer. DeepSeek announces that the necessary adaptations for its future models and new optimizations can be integrated into the same repository. Multiple GPUs, but a Single Response to Produce

Operating a distributed large model requires distributing computations across multiple accelerators without communications becoming more costly than the operations themselves. DeepInfer combines different forms of parallelism, including the distribution of attention, context, and experts in a Mixture-of-Experts architecture. A load balancing system, or EPLB, also seeks to prevent certain accelerators from remaining overloaded while others wait.

The architecture comes into play at two distinct moments in inference. The prefill phase corresponds to the initial processing of information transmitted in the request, while the decode phase refers to the progressive generation of the response. These phases do not necessarily use resources in the same way: the former must absorb a potentially large volume of tokens, whereas the latter must maintain a high throughput without slowing down users.

DeepSeek publishes separate performance curves for these two operations. The measurements presented in the repository were conducted on four NVIDIA SM100 GPUs for prefill and 64 for decoding, with each configuration contained within a single hardware domain featuring fast interconnects. The code required to reproduce these evaluations is also available, but the main documentation does not provide a universal overall gain compared to vLLM or other inference engines. Producing Faster with Anticipated Tokens

Another optimization concerns speculative decoding. Rather than generating each token separately by systematically deploying the main model, this method consists of preparing several candidate tokens and then verifying their validity. When a portion of the anticipated sequence is accepted, the system can reduce the number of operations needed to produce the final text.

To this end, DeepInfer integrates DSpark, accompanied by a verification mechanism scheduled according to confidence levels. The length of the proposals can be adjusted according to execution parameters, with a default maximum value of five anticipated tokens. The objective is not to change the model's capabilities, but to make its generation more efficient by taking advantage of sequences that can be validated together.

The framework also features an optimization called SWA Bounded Replay, related to sliding window attention. It limits certain recalculations by retaining only the relevant portion of the context during resume operations. For the V4.1 configuration described in the repository, DeepSeek notably recommends keeping a suffix of 140 tokens. These settings illustrate the targeted level of optimization: reducing repetitive operations that, at the scale of millions of requests, represent a significant amount of computation. Processing Multiple Requests Without Modifying Results

DeepInfer uses asynchronous scheduling that overlaps certain batch preparation operations on the central processor with the execution of computations on the accelerator. Enabled by default, this mechanism aims to limit periods during which the hardware waits for the software to organize the next task.

The framework also includes two less visible functions, which are nevertheless important for reproducibility. The first enables batch-invariant inference: when this option is activated, the output of a request remains identical even if the other requests executed simultaneously change. The second concerns deterministic sampling, which allows a generation to be reproduced exactly using the same input tokens, the same parameters, and a fixed seed.

These features are currently reserved for NVIDIA GPUs and require explicit parameters. Full reproducibility notably requires disabling certain speculative mechanisms. They