One video is enough to learn a ten-minute robotic task

S1 learns a robotic task from a single video, without fine-tuning, and performs it for up to ten minutes, according to Skild AI.

Repotting a plant, making pour-over coffee, flipping a pancake, or assembling a kit after watching a person perform the task only once. The S1 model introduced by Skild AI is designed to let a robot acquire this type of behavior from a video demonstration, without changing its parameters or starting a new training process.

The demonstration serves a role similar to that of a prompt. Instead of describing the task with text or supplying hundreds of executions recorded through teleoperation, the user simply films the desired behavior. S1 then analyzes the sequence to determine the objective, identify relevant objects, track task progress, and translate the observed movements to the robot’s own body.

The video does not need to be recorded with the robot that will perform the action. Skild AI notably uses human demonstrations filmed from a first-person perspective. The setting, camera angle, and demonstrator’s body can differ from the execution environment. The model must therefore understand that a human hand, robotic gripper, or articulated arm can serve equivalent functions despite their physical differences.

This approach relies on in-context learning, which is already widely used by language models. Under this framework, a system adapts its behavior to information supplied in the context without changing its permanent parameters. S1 applies the same principle to robotics: the video temporarily specifies the desired behavior, but it does not permanently add a new skill to the model’s parameters.

This distinction separates S1 from systems that require fine-tuning for every new task. A traditional method may need dozens or hundreds of demonstrations performed with the target robot, followed by a task-specific training phase. Skild AI says a single video is enough for S1 to begin execution.

The published demonstrations involve tasks the model had not encountered during training and that can last up to ten minutes. They include several dozen successive steps, with object changes, movements, and manipulations requiring different actions. Repotting a plant, for example, involves preparing the container, moving the plant, handling the soil, and watering the result.

For this experiment, Skild AI says eleven minutes elapsed between recording the demonstration and beginning the autonomous execution. Most of that time was reportedly spent physically preparing the scene. No additional training took place between the two sequences.

Every demonstration shown was generated using the same set of model parameters. Skild AI therefore did not build a separate model for cooking, gardening, or assembly. The robot’s behavior is conditioned by the video placed in its context at execution time.

S1 must also maintain its understanding of a long task. The robot needs to recognize which steps have already been completed, identify the next action, and avoid repeating an operation performed several minutes earlier. This becomes important as soon as an activity can no longer be reduced to picking up an object and moving it.

The model was trained using several categories of data. Skild AI mentions teleoperation sessions, recordings made with universal manipulation interfaces, first-person human videos, and simulated environments. The company says it produces this data internally, but has not yet disclosed the exact size of the final training corpus, its detailed composition, or the share represented by each source.

The demonstrations also undergo human quality control. Skild AI says it spends three dollars on verification for every dollar spent collecting data. The goal is to remove incorrect sequences, inaccurate annotations, and recordings that do not clearly represent the task.

The company compared its approach with a vision-language-action model controlled through text instructions. Both systems were trained using the same data, a comparable architecture, and the same compute budget. The datasets used in the experiment ranged from 1,000 to 100,000 hours.

With 1,000 hours of data, the language-conditioned model scored 53% on previously encountered tasks, compared with 43% for the demonstration-based system. At this scale, directly learning the relationship between an instruction and an action remained more effective.

The trend reversed as the amount of data increased. With 100,000 hours, Skild AI reported a score of 96% for in-context learning on known tasks, compared with 89% for the text-controlled model. The gap became especially pronounced on unseen tasks, where S1 scored 66%, compared with 9% for its experimental counterpart.

These figures require careful interpretation. The score represents average per-step success across internal evaluations, not the percentage of tasks completed fully and autonomously. When a failure prevents the robot from continuing, a person may intervene and return it to a state where the remaining steps can be evaluated. A score of 66% therefore does not mean that two out of three long tasks were completed without assistance.

Skild AI has not disclosed the full number of trials behind its most impressive demonstrations either. The plant repotting, cooking, and coffee-making videos show that the system can complete these behaviors, but they do not reveal the failure rate, the spread of results, or how much selection occurred before publication.

Another experiment compares the single demonstration with fine-tuning a language-conditioned model. According to the internal results, approximately 380 task-specific demonstrations are required to exceed S1’s 66% score from one video. For activities lasting more than four minutes, Skild AI estimates that collecting these demonstrations would require between 50 and 100 hours of teleoperation.

Fine-tuning still retains an advantage when a large amount of specialized data is available. With 2,000 demonstrations, the task-adapted model reached 86%. S1 is therefore less about replacing every form of specialized training than reducing the initial cost when a robot