Z1T adapts the Transformer to a chip that computes with randomness

Extropic introduces Z1T, a family of sparse models designed for its Z1 probabilistic chip, with energy efficiency gains over the H100 still based on projections.

Up to 139 times less energy than an H100 GPU to generate one token. Extropic’s figure immediately grabs attention, but it describes neither a complete large language model nor a measurement performed on an available product. Z1T remains a co-design experiment involving an extremely sparse architecture, a probabilistic chip, and a conventional coprocessor.

Introduced on September 4, 2026, the Z1T family is Extropic’s first public attempt to adapt transformer-like models to Z1, its upcoming thermodynamic computing chip. The company is also releasing the Z1T-0 weights on Hugging Face and its JAX training recipes on GitHub.

The project starts from a simple observation: today’s transformers have been shaped by the hardware on which they run. GPUs excel at large, dense matrix multiplications, where most values participate in every operation. Models have consequently adopted layers, attention mechanisms, and formats suited to this organization.

Z1 operates under almost the opposite constraint. Its physical structure connects each unit to only a limited number of neighbors. Extropic is therefore not attempting to reproduce a dense transformer directly on its chip. Instead, it modifies the model architecture to match the connections already present in the silicon.

The chip uses “pbits,” or probabilistic bits. Unlike a digital bit maintained in a stable state, a pbit is deliberately allowed to fluctuate between two values under the influence of thermal noise. Adjustable biases and connections modify the probability of each state.

The system does not interpret a single fluctuation as a definitive answer. It observes several samples, then calculates their average to produce a value that a neural network can use. Increasing the number of samples can improve the precision of that estimate, at the cost of additional processing time.

This approach is not quantum computing. Z1 uses no qubits, superposition, or cryogenic cooling. The chip relies on CMOS circuits operating with thermal noise at extremely low power. Its distinguishing feature is that it treats this physical instability as a computing resource instead of attempting to eliminate it.

According to Extropic, one Z1 chip contains eight cores, 269,568 pbits, and slightly more than 2.1 million physical connections. Each pbit has 16 adjustable couplings. Its internal sampling rate exceeds 50 MHz, while claimed power consumption remains below one watt.

This limited connectivity is both the hardware’s advantage and its constraint. A dense matrix cannot be mapped onto it directly. Z1T’s layers are therefore trained with a fixed number of connections from the outset instead of being reduced after training.

In the configuration most directly designed for Z1, each output receives only four input values. Each value is represented by four pbits, using all 16 connections available around the output unit. The weights use a four-bit representation.

This design substantially reduces the work performed during each pass. It also limits how much information each layer can exchange. To compensate, a sparse model may sometimes need to be wider or deeper than a dense model offering comparable performance.

Extropic’s experiments reflect this trade-off. On OpenWebText, Z1T models would require approximately ten times more training FLOPs to reach the same loss as a dense GPT-2 model. The expected advantage therefore does not come from more economical training, but from the much lower physical cost attributed to sparse operations during inference.

The comparison with GPT-2-small also relies on an extrapolation. The published curve estimates that approximately 9.5×10199.5 \times 10^{19} FLOPs would be required to match its validation loss. It does not show a Z1T model that actually reached this level through a completed training run.

Extropic also studies variants closer to GPT-2 to isolate the effect of connectivity. The tests vary the number of connections per output between 4, 16, 32, 64, and 128, alongside a dense version. At an equal amount of compute, dense operations remain more effective at reducing loss. The company is therefore betting that the difference in physical energy cost will be large enough to reverse this hierarchy at the hardware level.

Z1T also modifies several familiar transformer components. RMSNorm is replaced by Dynamic Tanh, a function better suited to Z1’s probabilistic cells. Conventional softmax attention gives way to a form of gated convolutional attention.

This mechanism combines a short local window with a cumulative summary of older information. It avoids directly comparing each token with every token that precedes it, as conventional dense attention does. The projections used to calculate its gates are also highly sparse.

Not every operation can run on Z1. Extropic therefore envisions a hybrid architecture in which tokens circulate between several Z1 chips and an FPGA, a reprogrammable digital component.

Z1 handles sparse projections and transformations based on the tanh function. The FPGA retains responsibility for orchestration, certain additions, residual connections, pooling operations, and the final vocabulary projection. This division makes it possible to use probabilistic hardware without requiring it to perform computations for which it was not designed.

For now, the company intends this configuration for decoding, when the model generates successive tokens. Initial prompt processing, known as prefill, would remain better suited to a GPU. Z1T would therefore not replace the H100 throughout the entire pipeline: both types of hardware could coexist within the same server or data center.

The main energy estimate concerns a small experimental model with four layers, an internal dimension of 512, a context length of 1,024 tokens, and approximately 11.55 million parameters in its body. This scale is far removed from commercial models containing tens or hundreds of billions of parameters.

Extropic estimates its energy use at 294.52 nanojoules per token