Halcyon questions image models' language pretraining

Radical Numerics presents Halcyon, a family of image generation models trained without language pretraining, using up to 275x less compute.

Should an artificial intelligence be taught to understand language before being taught to generate images? With Halcyon, Radical Numerics challenges this widely used method in visual generation systems. The laboratory introduces a family of multimodal models trained directly on 100 million image-text pairs, bypassing a pre-trained language model. In its comparisons, Halcyon outperforms several benchmark architectures with up to 275 times fewer compute resources dedicated to pre-training. The researchers are also releasing ten models along with their training methods. Generating images without pre-training a language model

Most modern image generation systems use linguistic knowledge acquired prior to their visual training.

A common method consists of using a pre-trained text encoder to interpret user prompts, and then feeding its representations into a model tasked with producing the image.

This setup, known as late fusion, initially separates language processing from image processing.

Other architectures combine both modalities earlier, but retain a prior training phase on large amounts of text.

Halcyon eliminates this language pre-training step.

The models are trained from scratch on pairings between images and descriptions. They simultaneously learn to process textual and visual information, without relying on the knowledge of an already established language model.

The research therefore addresses a specific question: how much of the resources traditionally invested in language learning is truly necessary to achieve a high-performing image generator? A shared architecture for text and images

Halcyon relies on an autoregressive Transformer utilizing a decoder-only architecture.

Text and images are represented in a shared sequence of tokens, which the model learns to predict progressively.

This approach is called early fusion, as the different modalities are integrated directly into the same system rather than processed by two independent models.

Each 256 × 256 image is converted into a grid of 1,024 visual tokens, drawn from a codebook of 64,000 entries.

Text has its own tokenization system, comprising 16,384 units. Textual information, visual information, and special tokens are then merged into a shared vocabulary of 80,640 entries.

Halcyon uses an architecture inspired by Qwen3, but the parameters are randomly initialized. The laboratory does not reuse the pre-trained language weights from Qwen3.

The visual encoder, the text representation system, and the generative model are developed within the same training program. One hundred million image-text pairs for a controlled comparison

The researchers trained five sizes of Halcyon models, ranging between 100 million and 4 billion parameters, using the GPIC dataset.

Each model goes through the 100 million image-text pairs once, with images resized to a resolution of 256 × 256.

The protocol consists of 97,632 training steps, using batch sizes of 1,024 examples.

To compare Halcyon against other approaches, Radical Numerics also re-trained several variants of LlamaGen and JiT-T2I on this same dataset.

This precaution helps limit a frequent issue in evaluations: comparing models that were trained on different datasets, and subsequently attributing performance discrepancies solely to their architecture.

In particular, the researchers adapted competing models to align visual representation lengths and certain training methods.

The total cost factored into the comparisons includes training the language component, the image representation system, and the generator itself.

This accounting is essential for interpreting the reported differences. Up to 275 times less compute in experiments

Radical Numerics compares the models using the gFD score, which measures the statistical proximity between generated images and those of a reference set. Lower scores indicate better results.

Halcyon-0.52B achieves a score of 48.0 compared to 63.3 for LlamaGen-XL/8, with a reported total pre-training cost 18.2 times lower.

The difference is even more pronounced in another comparison: Halcyon-1.6B reaches 35.3 compared to 41.0 for JiT-T2I-XXL/8++, using 275 times less compute dedicated to pre-training.

This factor accounts for the resources used to pre-train the Qwen3 language model leveraged by JiT-T2I.

It does not mean, however, that each individual generation operation is 275 times faster or less expensive.

Nor does the result constitute a direct comparison with all recent commercial models. The experiments focus on specific architectures, trained or adapted under a common protocol. Superior results also in prompt adherence

The statistical quality of images is not enough to evaluate their utility. A model can produce visually plausible images while failing to accurately reflect the initial prompt.

Radical Numerics therefore used two additional evaluations.

GenEval evaluates, among other things, the ability to follow object counts, colors, and spatial positions.

DPG-Bench measures compliance with longer, more detailed descriptions.

On GenEval, Halcyon-0.52B reaches 44.91 compared to 30.48 for LlamaGen-XL/8. On DPG-Bench, the scores are 76.84 and 67.08, respectively.

When pitted against JiT-T2I-XXL/8++, Halcyon-0.52B also achieves a better result on GenEval, scoring 44.91 against 43.17, and a near-identical score on DPG-Bench, at 76.84 against 76.85.

The Halcyon-1.6B version advances to 53.48 on GenEval and 77.90 on DPG-Bench.

At four billion parameters, GenEval reaches 56.39, while DPG-Bench remains at 77.90.

These results indicate that the observed gains are not restricted to the visual distribution of images, but extend to alignment between prompts and generated content. Three experiments to measure the utility of pre-trained language

The researchers isolated the contribution of language pre-training through three series of experiments.

The first compares text encoders of different sizes. Replacing FLAN-T5-Base, which has 110 million parameters, with FLAN-T5-XL, which has 1.22 billion, improves the