EAGLE-4 accelerates AI model generation by up to four times without altering their responses.

EAGLE-4 accelerates Qwen and Nemotron model inference on GPUs, CPUs, and smartphones through hardware-adapted speculative decoding.

Generating more tokens per second without retraining a large language model is the central objective of EAGLE-4, a new inference acceleration method introduced by Eagle AI. On Qwen3.8-27B, it delivers 3.59 times the decoding throughput of a conventional single-request setup in the researchers' matched comparison, while retaining an advantage when servers handle hundreds of concurrent requests. The technique also works on AMD GPUs and conventional CPUs, including a Motorola smartphone where it reduces energy consumed per token by more than half in one experiment. The innovation does not concern the model's knowledge or reasoning capabilities, but the computational process used to generate its answers. Predicting Several Tokens to Reduce Repeated Computation

Autoregressive language models produce answers incrementally, one token at a time. Although this allows them to generate complex sequences, the underlying model must be executed repeatedly. For models containing tens or hundreds of billions of parameters, that process can become particularly expensive, especially when many requests are handled simultaneously.

Speculative decoding addresses this constraint by introducing a smaller auxiliary model, known as a drafter, that anticipates several possible future tokens. The larger target model verifies those proposals in a grouped operation. If a proposed prefix is accepted, the system advances through multiple tokens without needing the same number of separate target-model executions.

The process does not simply allow a weaker model to replace the main one. When correctly implemented, speculative verification preserves the target model's output distribution. At temperature zero, accepted draft tokens match the deterministic predictions of the target model. The speedup therefore comes from restructuring the computation rather than relaxing the verification requirements.

The EAGLE family has explored this approach since January 2024, gradually introducing improved prediction networks and adaptive draft trees. EAGLE-4 adds explicit awareness of the computational cost of each speculative operation, aiming to avoid situations where the work required for acceleration becomes more expensive than conventional generation. One Drafting Strategy That Adjusts to Server Load

A central difficulty with speculative decoding is that an effective strategy for a single request may perform poorly when a server handles hundreds of conversations. When an accelerator has unused capacity, preparing a longer sequence of draft tokens can be advantageous. Under heavy load, the same extra operations may reduce overall throughput.

EAGLE-4 introduces EAGLE-Harness, a runtime system that adapts draft depth, tree width, and verification budgets based on hardware performance, server load, and token confidence. During warm-up, it profiles the computational cost of drafting and verification, storing those measurements for use during generation.

The system can expand a tree of candidate continuations when the expected benefit justifies the additional computation, or stop drafting when another step is unlikely to pay for itself. At low concurrency, it may prepare more candidate tokens. Under heavy load, it can limit speculative work to reduce wasted computation. A single trained drafter for each target model can therefore adapt to different serving configurations without requiring separately trained versions for each batch size.

This is the main distinction EAGLE's researchers emphasize when comparing their approach with DFlash 2 and DSpark. Those systems also use speculative decoding, but their methods of generating and verifying candidates differ. DSpark includes adaptive verification mechanisms of its own; EAGLE-4's particular emphasis is on deciding how much drafting computation to perform before that computation takes place. Qwen3.8-27B Reaches More Than 860 Tokens per Second

The team evaluated EAGLE-4 using Qwen3.8-27B on four NVIDIA B300 GPUs. In the paper's main matched comparison on SPEED-Bench, with temperature zero and the same inference engine across methods, EAGLE-4 achieves 863.8 tokens per second for one request. Conventional decoding delivers 240.5 tokens/s, compared with 672.6 for DFlash 2 and 618.6 for DSpark.

That represents 3.59 times conventional throughput, 28% more than DFlash 2, and 40% more than DSpark. At 512 concurrent requests, EAGLE-4 reaches 28,340 aggregate tokens/s, compared with 21,203 for conventional decoding, 16,641 for DFlash 2, and 14,229 for DSpark.

The difference between the two workloads matters. At 512 concurrent requests, 28,340 tokens/s describes the combined output across the server, not the speed experienced by any individual user. EAGLE-4 delivers a 34% improvement over conventional decoding in that setting, while both competing speculative methods fall below the non-speculative baseline.

The launch announcement also highlights 961.6 tokens/s through TokenSpeed, but this result comes from a separate optimized integration. In that configuration, TokenSpeed runs EAGLE-4 approximately 15% faster than a reference SGLang implementation of the same method, which achieves 837.6 tokens/s. The two headline throughput figures therefore come from different measurements and should not be treated as interchangeable. Improvements on Nemotron and AMD Accelerators

The gains extend beyond Qwen. Running NVIDIA Nemotron 3.5 Super VL on four B300 GPUs, EAGLE-4 achieves 647 tokens/s for a single request, compared with 236 using conventional decoding. At 512 concurrent requests, aggregate throughput increases from 19,823 to 33,480 tokens/s, representing speedups of 2.74 and 1.69 times, respectively.

AMD also evaluated EAGLE-4 on a single Instinct MI350X accelerator. For one request, throughput increases from 75.27 to 262.89 tokens/s, a 3.49-fold improvement. At 32 concurrent requests, aggregate output reaches 2,360.83 tokens/s, compared with 1,390.52 for conventional generation, or 1.70 times the baseline.

These