Fold, grasp, feel: a World Model learns to simulate the world in 3D

At over 50 frames per second on an RTX 5090, 3D World Simulator predicts motion, geometry, and tactile sensations to train robots.

Folding a napkin, grasping an object, or inserting a part into a mechanism requires more than a good understanding of images. A robot must anticipate movements, recognize contact, and take into account the deformations caused by its actions. A team of researchers from Columbia University, UC Berkeley, the University of Michigan, NVIDIA, and Analog Devices proposes a new approach capable of simulating these interactions in three dimensions, at more than 50 frames per second on a GeForce RTX 5090.

Called 3D World Simulator, this World Model does not merely predict the visual appearance of a scene. It simultaneously generates its geometry, its evolution in space, and tactile signals associated with the objects being manipulated. The whole system operates in a shared 3D representation, making it possible to change viewpoint or observe contacts without having to reconstruct each piece of information separately. The researchers present this architecture in their work on multimodal 3D simulation, accompanied by an interactive demo accessible from a browser. A 3D scene that evolves at the pace of actions

The system’s distinctive feature lies in its hybrid architecture, combining particles and voxels. The latter divide space into elementary volumes and provide a stable structure for predicting how a scene evolves. The particles are then used to reconstruct geometric details, especially when objects deform or come into contact.

The researchers use an autoencoder to convert information captured by multiple stereo cameras into a compact 3D representation. A second model learns to predict its next state from previous states and the actions commanded to the robot. Each new prediction can feed the next one, allowing extended simulations without imposing a predefined trajectory.

Unlike approaches that track the individual points of an object, this method does not need to preserve their identity from one frame to the next. This distinction becomes especially important when a robotic hand obscures part of the object or when a piece of fabric is constantly changing shape. The simulation maintains spatial consistency while allowing camera changes, including to viewpoints absent from the initial observations.

Tactile signals are also represented as 3D particles, aligned with the geometry of the objects. A robot can thus make use of simulated contact information, in addition to what it observes visually. From fabric folding to precision assembly

The interactive demos cover six manipulation scenarios: fabric folding, box packing, parts assembly, moving a T-shaped object, marker insertion, and rope threading.

These situations involve different physical properties. Folding requires tracking the deformations of a flexible surface, while inserting a part requires handling precise contacts between rigid objects. Manipulations performed directly in a robot gripper present an additional challenge, since the cameras do not always have an unobstructed view of the contact zones.

The interface makes it possible to control the robot arms with the keyboard or by moving their on-screen controls, and then observe the result through several representations. The user can display the rendered scene, the geometric particles, tactile information, or intermediate states of the model.

This flexibility also applies to the sets and the robots. The researchers can associate the simulated interactions with environments reconstructed by Gaussian Splatting, or replace the mechanical arms with other kinematic models. Elements outside the manipulation zone are then handled separately, which avoids recalculating the entire scene for every variation. Training robots without multiplying physical trials

One of the main advantages of this architecture concerns robotic learning. Typically, training a manipulation policy requires many demonstrations or interactions in a physical environment, with the hardware, time, and repeatability constraints that entails.

The 3D simulation makes it possible to generate visual and tactile observations consistent with a robot’s actions. The researchers used these data to train manipulation policies, then executed them on real machines. In the experiments presented, some policies trained exclusively on synthetic data achieve performance comparable to that of policies trained exclusively on real data.

The team also evaluates policies directly in the World Model, notably for assembly, fabric folding, and object movement. The simulations reproduce both successful situations and failures related to misalignment, slipping, or a missed grasp. The authors report a strong correlation between simulated success rates and those observed physically on the tasks studied.

These results remain tied to bounded manipulation environments. The system does not claim to freely reproduce the entire physical world: complex interactions take place within a defined area, while certain elements of the set and the robots are represented by separate components. More than 50 frames per second on a single graphics card

Speed is another important result. The system exceeds 50 frames per second on an RTX 5090, which allows real-time interaction with the generated scenes. According to the researchers’ technical notes, training the autoencoder and training the dynamics model each required about 16 hours on this same graphics card, starting from about two hours of manipulation data collected per task.

To examine the generalization of their method, the researchers also used Deform360, a dataset bringing together 198 everyday objects, 1,980 interaction sequences, and more than 215 hours of observations. The demonstrations show predictions on napkins, ropes, clothing, plush toys, and other deformable objects.

Part of the code is available on GitHub, along with the training instructions and the expected data structure. The repository nevertheless indicates that the data, checkpoints, and inference code still need to be added. The public demo therefore remains, for