A single designated pixel is sufficient to reconstruct the 3D trajectory of an object.

ProxyPose, a method by the University of Toronto and Vector Institute, tracks an object's 3D trajectory from a single clicked pixel in a standard video.

Researchers from the University of Toronto and the Vector Institute present ProxyPose, a method that tracks the motion of an object in a video from a single clicked point. The input is an ordinary video and a user-selected pixel; the output is the complete six-degree-of-freedom (position and orientation) trajectory of the target object.

The trick lies in a detour through generation. Rather than estimating the pose directly, ProxyPose first translates the source video into a "proxy video" where a colored cube exactly replicates the local rigid motion of the designated point. This translation relies on a generative video model fine-tuned via LoRA. The trajectory is then recovered using classic, training-free geometric methods, specifically the Perspective-n-Point algorithm.

The approach targets situations where existing methods fail. It tracks objects with challenging materials, whether transparent or reflective, and holds up under heavy occlusions. Although trained solely on rigid objects, it also manages to track deformable surfaces like a human face, and extends to camera tracking where classic photogrammetric reconstruction stalls due to a lack of texture. Without retraining, it even applies to other sensors, such as event cameras or single-photon imaging.

The authors note the limitations of the process: pose drift on reflective or textureless surfaces undergoing complex motion, inconsistencies when the motion is poorly defined (the surface of a fluid), and blurry images beyond the video model's capabilities during very fast movements. Code and paper are public.