Eighteen years of screen recordings, and not a single annotated click

Induction Labs' Photon-1 model learns to use a computer by predicting the next frames of screen recordings. A simple and efficient approach.

The internet is teeming with videos of people using computers, without anyone ever having recorded which click was made at what moment. Induction Labs proposes to tap into this resource anyway, using an architecture they call imagination models. The principle consists of training a model to predict the next state of a video, not in pixels but in a representation space. Its first specimen, Photon-1, was fed eighteen years of screen recordings, without any actions being pointed out to it.

The core result lies in a learning process that no one taught. By dint of predicting what will be displayed, the model implicitly learns to act. After a light fine-tuning, designed to make it adopt the format of keyboard and mouse commands, Photon-1 first imagines the next screen, then produces the action that leads to it. Reinforcement learning conducted on virtual machines, with results verified automatically, subsequently improves its competence.

None of this would hold up without the in-house visual encoder, which compresses each image into 960 discrete tokens, about 2.2 kilobytes—roughly a hundred times less than the usual representations of multimodal models—while preserving text, layout, and state changes. On its own computer-use benchmarks, the laboratory places Photon-1 ahead of Gemini 1.5 Flash, with training it estimates to be thirty times less computationally intensive and a serving cost three times lower.

Two effects draw attention beyond the score. Although it has never seen anything other than screens, the model transfers its representations elsewhere: fine-tuned on games of checkers and then on billiards simulations, it outperforms baselines trained on the same data, both in anticipating the evolution of a board and in rendering a ball's trajectory. Above all, it has inherited the human habits contained in its videos, to the point of querying ChatGPT, proofreading its response, and redirecting it until it gets what it is looking for. The team, consisting of two people based in San Francisco, plans to extend the approach to other unlabeled videos, including physical-world gestures and skilled labor.