Meta opens its OpenXR layer to agents, but not in production

Meta XR Operator exposes Unity session data to an external agent. An experimental method to automate testing on Quest headsets.

A software layer sandwiched between an XR application and its runtime is what Meta XR Operator, an experimental component of the Meta XR Core SDK, offers. It intercepts the calls that a Unity application already sends to the OpenXR runtime and exposes what it finds there—session state, tracking data, input control—to any MCP-compatible agent. No modification of the application's code is required, as the integration occurs at a lower level.

The agent then gains two capabilities that did not previously go together: perceiving and acting. On the perception side, it queries the session state and the data produced frame by frame, traverses the Unity scene graph, inspects interfaces and their clickable elements, captures composite views blending the real environment and virtual content, and retrieves the list of recognized spatial entities—walls, floors, tables, and other furniture—along with their position and semantic label. On the action side, it reads and controls the position of the head and controllers, using interpolated movements, and operates buttons, triggers, grips, and joysticks.

The setup relies on a local MCP server launched within the application's process, connected to the agent via a proxy. The same sequence applies in the Unity editor, in the in-house simulator, or on a Quest headset, with the latter case requiring port forwarding for the host machine to reach the embedded server. Windows benefits from full support, while macOS on Apple silicon is limited to the simulator. Custom tools written in C# can be added to those provided.

The highlighted use cases fall under development and testing: reproducing a bug by describing the maneuver in natural language, running test campaigns without writing every scenario, and examining the state of a session while it is running. Meta accompanies the whole package with a clear caveat: the component is experimental, its tool names, APIs, and behavior may change, and the publisher advises against relying on it in a published application.