Skilled manual work as training data for robotics: the HubEgo-100K project

The first HubEgo-100K release contains more than 10,000 hours of recorded manual work to study robot learning across varied settings.

Microsoldering phone circuit boards, repairing watches, and bricklaying are among the activities recorded in HubEgo-100K. This first-person video dataset documents people working in workshops, on construction sites, and at home. The project aims to provide robotics researchers with observations of manipulation involving varied tools, objects, and environments.

The announced 100,000 hours will be released in stages. The first release lists 10,124 hours across 72,754 episodes from 12,227 contributors. The remainder is scheduled to arrive in batches over the coming weeks. The dataset’s name therefore describes the scale of the overall program, rather than the volume published in this initial version. Environmental diversity as a collection priority

The dataset focuses on skilled manual tasks, with more than 62,000 distinct descriptions across 56 task types and 30 work categories. A second classification identifies 35 action families, such as cutting, assembling, and cleaning. The same action therefore appears with different tools and materials, rather than in a single repeated setup.

According to the documentation, 83% of the footage comes from workplaces. The preprint counts 15,548 environments, each defined as one contributor’s setting of a particular type. Collection spans workshops, commercial kitchens, warehouses, and farms, alongside activities performed in homes.

To support this approach, Hub cites a study by Fanqi Lin and coauthors, presented at ICLR 2025. On the manipulation tasks studied, systems’ ability to operate with unfamiliar objects and in new locations improves primarily with the diversity of training situations. Beyond a certain threshold, adding demonstrations in the same context provides less benefit. This study supports a collection strategy; it does not measure the performance of robots trained on HubEgo-100K. A head-mounted camera and synchronized sensors

The recordings combine 1080p color video at 30 frames per second with accelerometer and gyroscope measurements at 100 Hz. These sensors capture the movement of the head-mounted device. The data follows the LeRobot v3 format, with native sensor streams preserved in MCAP.

Hand visibility is a central criterion. The preprint estimates that at least one hand appears in approximately 95% of sampled frames. It also reports that 96.9% of the retained footage is spent performing the task. These measurements come from automated analysis using a vision-language model.

Each episode receives a description, instructions, an action family, a difficulty level, and a recording context. The dataset card specifies that these automatically generated annotations have not been checked by people. The first version of the preprint documents collection and diversity; the authors plan to add benchmarks in later versions. An open license allowing commercial use, with access through a form

The dataset is offered under CC BY 4.0, which permits reuse and adaptation, including commercially, with attribution, a link to the license, and an indication of any changes. Access nevertheless requires a Hugging Face account and a form requesting information including affiliation and intended use. Free access therefore does not mean an immediate anonymous download.

According to Hub, contributors consented to their recordings being used to train and evaluate AI and robotics systems. Before publication, the company says it removed segments detected as containing minors, other people’s faces, or private text. Audio, location data, and information identifying contributors were also removed from the released dataset.