Runway Characters win tool calling in real-time video

Runway Characters can now use tool calling to perform actions like web navigation and cart checkout in real-time video streams using JSON function calls.

Runway's Characters, its conversational video agents, can now call external tools during a call. Previously limited to verbal interaction, these avatars can perform concrete actions: navigate a website, query a database, click within an interface, add items to a cart, or finalize an order, all while continuing the discussion with the user.

The official demonstration features Vivi, a virtual stylist, in a video call. At the user's request, she searches an online store, displays products, navigates to a denim jacket, adds it to the cart, and finalizes the order, without interrupting the verbal exchange.

The mechanism adopts the tool calling pattern already widespread with OpenAI or Anthropic, transposed into a continuous video stream. The model analyzes the request, decides if a tool is necessary, and returns a function call in JSON format (for example, `checkorderstatus` accompanied by an `orderid`). The developer then executes the tool, which can be a Client tool (browser action: overlay, click, navigation) or a Server tool (HTTP call to a backend). The result is returned to the Character, which continues the conversation by integrating it.

The documentation is published on Runway's developer portal.