Kyutai measures the hidden randomness behind FID in image generation.

Kyutai and UC Berkeley find that training randomness causes a 2.1-point variation in Fréchet Inception Distance (FID), making single-seed scores unreliable.

Kyutai, with UC Berkeley, published The FID Lottery, a paper that quantifies a rarely measured element of randomness in the evaluation of generative image models. Its target: FID, or Fréchet Inception Distance, the metric that acts as an arbiter, where half a point reorders a ranking and a few decimals guide GPU-hour budgets.

Behind every reported FID, the authors remind us, lie five sources of randomness: model initialization, data order, noise injected at each training step, hardware non-determinism, and generation noise at the time of sampling. Current practice only accounts for the last one, by re-sampling multiple seeds; the paper studies all five.

The finding: across hundreds of models trained on ImageNet, training randomness weighs significantly more heavily than generation randomness. By rerunning the same training twenty-five times on the same machine, with only the seed changing, the FID ranges from 33.59 to 35.69, a 2.1-point difference due to pure chance, and the dispersion from one retraining to another exceeds that of a simple re-sampling by approximately three times. None of the tested levers—guidance, compute, model size, or learning rate—closes the gap, such that a single-seed FID relies on a noise floor below which published recipe gains often fall. The authors note, however, that fine-tuning guidance halves this relative floor, and that a lucky training seed achieves the same score with up to half the computation of an unlucky one.