After diffusion, another path opens up for generating images with AI

A team from MIT, Harvard, and UIUC presents TEM, an energy-based generative architecture capable of competing with certain diffusion models.

What if the next evolution in image generation did not rely solely on diffusion models? Researchers from MIT, Harvard, and the University of Illinois Urbana-Champaign have just introduced an architecture named Transformer Energy-Based Model (TEM), capable of producing high-quality images using a different probabilistic approach. Their results show that energy-based models, long limited by training difficulties and scaling challenges, can now compete with several diffusion architectures under comparable experimental conditions.

Presented in the study Scaling Transformer Energy-Based Models for High Quality Image Synthesis, published on October 22, 2026, this research is authored by Jungsoo Lee, Alexi Gladstone, and Yilun Du. It is accompanied by an interactive demo page and an accessible implementation on GitHub. Learning to recognize a good image rather than reconstructing it

Diffusion models generally generate an image by starting from a noisy representation and then following a sequence of steps designed to recover a coherent structure. Energy-based models, or EBMs, adopt a different approach: they learn to assign a numerical value to each visual configuration.

This value, called energy, is lower for images that match the characteristics of the training data and higher for those that deviate from them. Generation then consists of starting from random noise and progressively modifying this representation to reach a low-energy region.

The distinction is important. Rather than exclusively learning a generation trajectory, the model learns a function that allows it to evaluate possible configurations. This separation enables different generation procedures, inference-time adjustments, and the combination of multiple visual constraints.

The principle has existed for several decades, but EBMs encountered major difficulties when producing complex images. Their training could become unstable, while searching for low-energy configurations required significant computation. TEM specifically aims to overcome these obstacles. A Transformer architecture to scale up

The proposed solution relies on a vision Transformer capable of processing the latent representations of an image—that is, compressed information that preserves its essential structure.

The originality of TEM stems notably from the use of multiple energy functions distributed across different depths of the network. The early layers contribute to learning local textures, the intermediate layers to the organization of objects, while the final layers capture broader semantic characteristics.

These evaluations are then added together to form a joint energy function. The model can thus take into account multiple levels of coherence during generation.

The researchers also introduce a method called Positive Representation Alignment (PRA). It consists of aligning certain internal representations of TEM with those produced by a pretrained vision model, DINOv2, exclusively for real images used during training.

This restriction plays an important role. Imperfect synthetic images are not subject to the same alignment, in order to preserve the contrastive mechanism that distinguishes real data from generated samples.

Ablation experiment results show the value of this technique. On an intermediate configuration, the FID score drops from 42.8 without alignment to 19.9 with PRA after 50,000 training steps. In this test, a lower value corresponds to better statistical proximity between the generated images and the reference images. Competitive results against diffusion models

On ImageNet, a dataset commonly used to evaluate vision models, the TEM-XL version achieves a FID-50K score of 3.82 in class-conditional generation at a resolution of 256 × 256. At 512 × 512, it achieves a score of 8.24.

The model contains approximately 609 million parameters and uses 60 generation steps in the main evaluations. It operates without classifier-free guidance, a technique used by many diffusion systems to enhance adherence to generation conditions.

In the comparisons published by the researchers, TEM-XL achieves a better unguided FID score than several reference architectures, notably DiT-XL, SiT-XL, and SiT-XL with REPA. The presented curves also indicate favorable progression as training resources increase.

However, these performances do not mean that TEM surpasses all existing diffusion models. Certain compared architectures achieve superior results when using guidance mechanisms. The experiments also focus on class-conditional ImageNet image generation, rather than a general-purpose system for creation from complex text descriptions.

The significance of the work therefore lies less in setting a new absolute record than in demonstrating that a family of models historically difficult to train can reach a competitive level at higher resolutions. Composing images by adding constraints

One of the most interesting properties of energy-based models involves compositional generation. If a model knows how to represent different visual categories through energy functions, it becomes possible to combine them by addition to search for an image that satisfies multiple conditions simultaneously.

On the project demonstration, the researchers notably combine the concepts of a zebra and a beach, or a shark and a car rearview mirror. The functions corresponding to the two categories are added together and then used during generation, without retraining the model for each combination.

A more structured evaluation was performed with CLEVR, a synthetic dataset used to measure compliance with spatial constraints. For two objects at defined positions, TEM achieves a success rate of 76.6%, compared to 59.8% for the Composable Diffusion method selected as a baseline. With three objects, the gap becomes much smaller: 27.4% versus 26.2%.

These results illustrate both the potential of the approach and its current limits when the number of constraints increases.

The same energy function also allows for inpainting,