An interface generated as you use it: Runway Labs’ experimental approach
The Diffusion-UI prototype combines structured data, state management, and generated controls to create interactive interfaces in real time.
Moving or resizing a bounding box to change a scene is one of the interactions Runway Labs describes in its introduction to Diffusion-UI. This research application explores software whose visual output is generated by a model in real time as the user interacts. The team says it can incorporate structured data, maintain state, and generate controls.
The concept centers on the ongoing use of a generated interface. Information supplied to the system, visual composition, and user actions all contribute to what appears on screen. The project remains an exploration, with no public access announced in the posts shared. Data to guide the rendering
One of the highlighted capabilities is the integration of external information. JSON objects can pass data from an API to the model. The examples include precise map routes and live weather information for the system to render in a dynamic interface.
This gives the visual output an explicit foundation: the model receives specific values or instructions to represent. Maintaining state is also among the announced capabilities. However, the thread does not explain how that state is stored or how the system checks that the rendering remains faithful to the data. Layered composition and identifiable controls
Creation combines several layers: data sources, reference layouts, styles, and text prompts. A written request can transform a scene, add objects, or modify its contents during use. Moving and resizing a bounding box provides another way to adjust the composition.
Identifying controls is also part of the experience described. As the pointer hovers, white outlines highlight nearby buttons, sliders, and other interactive elements. These visual cues show where users can act within a display that is itself generated.
The proposed uses extend beyond information panels: the team also cites complex 3D interfaces and immersive game scenes. More prototypes and tools are expected to be shared soon, without a specific timeline.