Generating text without advancing word by word: the path explored by Stanford

Avec Next-Concept Modeling, des chercheurs de Stanford génèrent du texte par niveaux de détail, jusqu'à 31 fois plus vite que certains modèles de diffusion.

A language model that begins by constructing a general representation of its text, then progressively refines its details, rather than drafting each word in order. This is the principle behind Next-Concept Modeling (NCM), a method developed by a team of researchers from Stanford and Together AI. In their experiments, this approach achieves generation up to 31 times faster than certain diffusion language models, at comparable quality, while requiring less training data.

The work, led notably by Hermann Kumbong and Ishan S. Khare, is documented in a scientific publication and accompanied by public source code, pretrained models, and datasets. They examine another way of producing language, situated between traditional autoregressive generation and models that build their responses through successive refinements. Moving Beyond Word-by-Word Generation

The majority of large language models operate in an autoregressive manner. They predict a token, add it to the sequence, and then use this new context to predict the next one. This sequential dependence constitutes a constraint when responses become long: each new step awaits the result of the previous one.

Diffusion language models explore another method. They can predict or correct multiple tokens simultaneously, but must often repeat this operation over the entire sequence. Reducing the number of iterations speeds up generation, at the risk of degrading the quality of the resulting text.

NCM alters the problem at its source. Instead of directly generating the final tokens, the system constructs a hierarchy of discrete representations, from the most general information to the finest details. Each level is produced after the previous one, but all codes belonging to the same level are predicted in parallel.

This organization reduces the number of successive steps and avoids processing a complete sequence from the very first operations. The text no longer appears as a linear sequence of words, but as a progressively reconstructed representation. An Architecture Borrowing from Image Models

The method relies on two training phases. In the first, a multiresolution VQ-VAE autoencoder learns to convert a text into several levels of discrete codes. The first levels are short and contain a coarse representation of the content. The subsequent ones add the information necessary for a more precise reconstruction.

This mechanism is notably inspired by work on image generation, specifically Visual Autoregressive Modeling (VAR), where the composition of an image is constructed progressively, from major structures to fine details.

In the second phase, the researchers freeze this encoder and train a Transformer to predict the different levels. During generation, it produces all the codes of a given level in parallel, then moves to the next. The decoder reconstructs the text from the complete representation.

The team illustrates this operation with a translation from German to English. In the initial steps, the output already captures certain elements of the sentence's meaning, but retains repetitions and errors. Subsequent levels progressively correct the phrasing until a coherent sentence is obtained. This evolution is visible in the interactive demo published by Stanford.

The term "concept" here does not, however, designate an explicitly formulated idea or symbolic reasoning. It refers to the information encoded in the early levels of representation learned by the model. Up to 31 Times Faster in Evaluations

The researchers tested NCM across several families of tasks, notably free-form text generation, translation, summarization, and Sudoku solving.

On One Billion Word (LM1B), NCM achieves lower generative perplexity than the best diffusion model selected as a baseline, while generating text 21 times faster. On OpenWebText, it attains comparable quality with 10 times faster generation.

The most significant gap appears on XSum, a dataset dedicated to text summarization. At a comparable ROUGE-L score, NCM proves to be 31 times faster than the baseline diffusion model. Time measurements were conducted on an 80 GB NVIDIA H100 SXM GPU under the conditions described by the authors.

These speedups stem from two factors. NCM requires fewer successive model evaluations, but each of the early steps also processes a shorter representation, which is therefore less computationally expensive to calculate.

The results do not constitute a general comparison with commercial conversational large language models. The primary experiments involve models of relatively modest size, around 100 million parameters depending on the task, and targeted benchmarks. The reported gains pertain to specific architectures and comparison protocols. Measured Results on Reasoning as Well

Sudoku serves as another evaluation ground, as a solved grid must simultaneously satisfy multiple constraints: each row, column, and block must contain the correct digits.

On the hardest grids, NCM completely solves 86.1% of the puzzles, compared to 81.3% for FMLM+, the top-performing competing model in this comparison. It achieves this result with 12 model evaluations, versus 16 for FMLM+.

Performance also remains superior on easy and intermediate grids. The researchers attribute these results to the model's ability to work on a global representation of the solution, rather than fixing digits sequentially without being able to revisit prior decisions.

However, the study does not demonstrate that NCM possesses superior general reasoning capabilities. It establishes results on a structured task with precisely defined constraints. Fewer Training Tokens, More Possible Generations

The experiments also indicate a reduction in the volume of data processed during training. On LM1B, the tokenizer and generator of NCM together use approximately 4.9 times fewer training tokens than the compared diffusion models. On OpenWebText, this ratio is close to four.

The team posits a possible explanation: unlike diffusion models, which must learn to