AngelSpec speeds up responses by having AI write a draft

Tencent's AngelSpec framework accelerates text generation by having draft tokens verified. An optimization for production.

To produce a response, a large model usually generates tokens one after the other. AngelSpec aims to shorten this mechanism through speculative decoding: a lighter model proposes several tokens in advance, and then the main model verifies them all at once. Validated proposals are kept, while the others are replaced, without altering the expected output of the main model.

The framework combines several methods for creating these drafts. MTP favors short sequences and is better suited for open-ended conversations, where multiple formulations remain possible. DFly produces more tokens in parallel and proves more suitable for code or mathematics, where the continuations are generally more predictable. AngelSpec allows for training, evaluating, and comparing these different architectures within a single environment.

A second mechanism, dubbed D-cut, adjusts the amount of text to verify based on server load and the confidence placed in each proposal. Rather than systematically checking the entirety of the drafts, it prioritizes keeping the passages with the highest probability of being accepted, in order to avoid unnecessary computations.

On Hy3-A21B, Tencent indicates that DFly multiplies generation speed by 1.98 to 2.40 compared to token-by-token generation, depending on the number of requests processed simultaneously. Its average throughput is also said to exceed that of DFlash by 10.5% to 11.8%. The training code, several draft models, and the tools necessary for their integration are published as open source.