4-bit computation from the start of Inkling 2 Small’s pretraining

The expert layers in Inkling 2 Small combine 4-bit and 8-bit computation. Thinking Machines reports speed gains on GB300 hardware.

From the first pretraining step to the final reinforcement learning rollout, Inkling 2 Small uses the same low-precision computing recipe. In a post by Jack Cook, Thinking Machines details its use of NVFP4 for the forward pass through expert layers: the stage where the model computes its predictions. The backward pass, which calculates the corrections needed for learning, uses MXFP8.

The phrase “end-to-end 4-bit training” therefore describes continuity across training stages while using several numerical precisions. The final sixth of the layers still runs in MXFP8, while some activations remain in BF16 and master weights in FP32. According to the lab, early experiments that also applied NVFP4 to the backward pass degraded learning efficiency too much.

In its largest training runs, the lab measures an 8% pretraining speedup and a 20% reinforcement learning speedup compared with an MXFP8 configuration. Those figures cover the complete recipe, including its changes to computation and communication. They should be distinguished from the theoretical matrix multiplication speedups offered by the hardware.

The 68 hours highlighted by Mira Murati refer to a different scope. The model announcement specifies that this duration covers the initial H1 phase, processing 8.4 trillion tokens of natural language, code, and image data on GB300 NVL72 racks. Additional phases followed before reinforcement learning. The model has 594 billion total parameters, with 17 billion active. The reduction from several weeks to 68 hours is attributed to multiple improvements, and the post does not specify the total number of GPUs used.

Using fewer bits reduces storage requirements and accelerates certain operations at the cost of more aggressive rounding. NVFP4 combines four-bit values with a scale factor shared across sixteen values, averaging 4.5 bits per value when that factor is included. The published table reports peak computational throughput up to six times higher than BF16 on GB300. That local ratio does not automatically translate into training an entire model six times faster.

The difficulty becomes particularly apparent during the backward pass. Matrices used in the forward pass must be transposed, regrouping the values that share a scale factor. Quantizing them again can then produce different values and distort gradient calculations. Rounding that consistently favors one direction introduces a bias that can accumulate during training.

To limit that effect, the team dequantizes the values used in the forward pass, transposes them at higher precision, then requantizes them into MXFP8 using stochastic rounding. This method probabilistically chooses the higher or lower representable value based on their distance. The aim is to preserve the forward-pass value on average, rather than require an identical representation in every operation. It does not recover all the information lost during the initial quantization.

Once computation becomes faster, communication between GPUs becomes another limiting factor. In a mixture-of-experts architecture, data is sent to the processors hosting the selected experts, then their results are combined. The lab dispatches inputs in NVFP4. For the return path, a Hadamard transform spreads out extreme values before quantization into S2F6 with scale factors. The transmitted data volume then falls from sixteen to 8.25 bits per value.

GPU computing programs, or kernels, developed internally under the name BBBK combine several of these operations to reduce waiting between computation, conversion, and data transfer. The team describes fusing the transform and quantization into the final stage of matrix multiplication. The result therefore depends on how work is organized on GB300 hardware as much as on the numerical format.

The tradeoff retains a learning cost. During pretraining, the lab reports approximately a 2% loss in compute efficiency: MXFP8 training would have reached the same predictive loss with roughly 2% fewer operations. In reinforcement learning, the team finds no discernible reward difference in its comparisons. Computing the backward pass in FP4 remains a research effort, separate from the recipe used for Inkling 2 Small.