Source-linked AI summary
Solaris: Towards Interfaces That Are Generated, Not Coded
Yuval Alaluf, Omri Avrahami, Guy Bukchin Leshem, Michal Geyer, Kfir Goldberg, Elad Richardson, Diego Alarcón, Alejandro Alvarez, Cole Garry, Anastasis Germanidis, Tenaya Goldsen, Corina Gurau, Robin Kahlow, Joel Kwartler, Kathleen Lewis, Alejandro Matamala Ortiz, Eugene McMahon, Thon Prom, Sarah Saltonstall-Wurm, Jamie Umpherson, Hudson Yeo
TL;DR
Existing interfaces require visual designs and behaviors to be translated into predefined intermediate representations, limiting the interaction space. Solaris instead combines a visual world model with a language model to generate interfaces and responses frame by frame, with interactive-speed generation and user-directed behavior. Participants preferred Solaris over a coded website for both requested interaction and natural behavior, while the authors identify unresolved challenges in real-time generative modeling.
Problem
Traditional interfaces translate visual designs into code and predefine behaviors, limiting possible interactions and sacrificing visual fidelity.
Method
Solaris generates frames autoregressively, uses few-step distillation and self-output training for real-time coherence, and pairs a visual world model with an LLM for interaction reasoning.
Results
Participants preferred Solaris in 62% of comparisons for following requested interactions and 72% for natural behavior, versus 25% and 21% for the coded website.
Takeaways & Limitations
Solaris generates interfaces directly without an intermediate representation, preserving the visual and semantic state from the first frame.
Takeaways & Limitations
Solaris is strongest at ambient motion, click-and-drag interactions, and scene transitions, while several challenges remain.
Abstract
from arXiv · showhide
Digital interfaces are traditionally implemented through intermediate representations such as code, requiring their appearance and behavior to be specified in advance. We introduce Solaris, an interface world model that instead generates an interactive UI directly, frame by frame, in response to user actions. Solaris treats mouse interactions as conditioning signals and autoregressively synthesizes the resulting visual state at interactive speeds. To enable real-time generation while maintaining visual coherence over extended interactions, we combine autoregressive frame generation with few-step distillation and training on the model's own outputs. A language model complements the visual world model by interpreting user intent and specifying how interactions should affect the generated environment, separating high-level reasoning from visual rendering. By generating both the appearance and behavior of an interface dynamically, Solaris enables open-ended interactions that need not be explicitly programmed in advance. We view interface world models as a step toward a new paradigm for software, where interfaces are generated and adapted continuously around user intent rather than implemented as fixed collections of predefined states and
1 Introduction
Solaris proposes interface world models that generate visual interfaces directly, frame by frame, rather than translating designs into fixed code. The resulting environments support direct, continuously evolving, and open-ended interaction around user intent.
- Concept: Solaris introduces interface world models that generate apps and websites as users interact with them.The model renders the interface itself rather than relying on a separate implementation beneath the visuals.
- Motivation: Traditional software translates visual designs into intermediate representations such as code, requiring behaviors to be specified before users arrive.This translation can limit possible interactions and sacrifice visual fidelity.
- Implications: By jointly generating frames and responses, Solaris removes the intermediate representation and treats the entire visual frame as the interface.The paper also positions this approach as useful for training agents in more dynamic environments.
- Capabilities: Solaris is entirely visual: users manipulate the scene directly, with objects and environments responding continuously to their actions.Examples include trying on clothing, rearranging rooms, building salads, experimenting with combustion, exploring landmarks, and inspecting products.
- Capabilities: Solaris is open-ended, allowing the same scene and interaction to produce different behaviors rather than only developer-anticipated workflows.The same drag can zoom in to inspect a wrist or measure a hand.
2 Motivation
Traditional interfaces separate systems that understand user intent from systems that render interactive experiences. Solaris proposes an interface world model that combines these roles, without a reference implementation defining all behaviors.
- The interface gap: Digital interfaces traditionally divide knowledge-bearing systems from real-time systems that create interactive experiences.An interface world model must understand intent while continuously rendering an interactive world.
- Limits of prior systems: Prior interface world models reproduce existing software, so available interactions remain bounded by the original implementation.NeuralOS, ViMo, and CUWM are trained against reference implementations.
- Solaris’s premise: Solaris removes the reference implementation: a language model determines behavior during the session, while a world model renders context-dependent interactions.The same drag can zoom into a wrist or measure a hand, depending on context.
- Engineering challenges: Building an interface world model requires real-time speed, coherence across an entire session, and visual quality at 720p.Interactions feel non-interactive around half a second of delay, motivating sequential frame generation.
3 Method
Solaris generates an interface continuously from user actions rather than selecting predefined screens. It combines interaction-conditioned frame generation with language-model reasoning about how the scene should evolve.
- Learning interaction: Solaris treats clicks, drags, and other user inputs as conditioning signals for generating the next frame.The model learns relationships between actions that have already occurred and their visual outcomes.
- Running in real time: To run in real time, Solaris generates frames autoregressively, distills denoising into a few steps, and trains on its own outputs.These stages target interactive speed while stabilizing visual quality.
- Reasoning and rendering: A language model interprets requests and determines how interactions modify the current scene or transition to a new one.It defines behaviors and produces prompts that guide the world model’s rendering.
- Continuous generation: Starting from an initial state, Solaris streams frames and incorporates clicks, drags, or typing into subsequent generated frames without predefined screens or templates.Natural-language prompts specify what interactions mean in each scene.
- Redefining the mouse: Natural-language interaction descriptions let scene objects become new tools whose effects are not fixed in advance.Examples include transferring a cat’s fur properties or drawing in a painting’s style.
4 Results
Translation through code loses visual information as interfaces become more complex, while Solaris operates directly on the visual interface. In a user study, participants preferred Solaris over coded interfaces for both instruction following and natural behavior.
- The Cost of Translation: The reconstruction benchmark evaluates 30 interfaces spanning plain webpages, image-heavy webpages, and natural images.It uses SSIM for in-place appearance and DINOv3 features for content preservation despite layout changes.
- The Cost of Translation: Every evaluated language model loses information during reconstruction, with natural images affected most and losses increasing with visual complexity.Figure 10 summarizes decreasing reconstruction fidelity as complexity increases.
- The Cost of Translation: Solaris operates directly on the visual interface and preserves its complete visual and semantic state from the first frame.This avoids translating the interface into an intermediate representation and reconstructing it.
- Solaris vs. Coded Interfaces: The user study compares Solaris with Claude Opus 5 using 250 participants, 30 interaction examples, and nearly 7,500 pairwise judgments.Both systems started from the same image and received the same interaction requests.
- Solaris vs. Coded Interfaces: 62% preferred Solaris for following requested interactions versus 25% for coded results, while 72% preferred Solaris for natural behavior versus 21% for coded websites.The remaining comparisons were rated equivalent: 13% for instruction following and 7% for natural behavior.
- Solaris vs. Coded Interfaces: Interface world models can produce interactions coherent with objects, materials, and environments rather than treating each action as an isolated update.This broader distinction explains the larger preference margin for natural behavior.
5 Limitations
Solaris remains strongest in ambient motion, click-and-drag interactions, and scene transitions, while several challenges constrain its current scope. These include real-time text generation, grounding and trust, long-session coherence, and accessibility integration.
- Current scope: Solaris is strongest at ambient motion, click-and-drag interactions, and scene transitions, while the listed challenges reflect the frontier of real-time generative models.The paper expects these challenges to improve alongside the underlying models.
- Text: Fully real-time generated text remains an open challenge because stable, legible text is difficult for video generation despite interfaces’ heavy dependence on it.The paper suggests hybrid systems as one practical path when brief pauses are acceptable.
- Trust: Solaris must be grounded in richer verified context to support trustworthy instructional or commercial experiences.The paper identifies reference images, product data, and documents as active research directions for conditioning generation during sessions.
- Long sessions: Maintaining visual and semantic coherence over extended, open-ended interactions remains an active research area.This limitation concerns the persistence of coherent interface state across long sessions.
- Accessibility and integration: Generated interfaces still need integration with software infrastructure, including screen readers and accessibility APIs, so flexibility does not reduce usability.Accessibility and integration remain practical requirements beyond visual generation itself.
6 Conclusions
Solaris presents interface world models as a new operating layer in which generated environments adapt around user intent rather than fixed app boundaries. The paper illustrates this direction through coherent interactions, personalized interfaces, and context-adaptive tutorials, while expecting future models to become faster, more coherent, controllable, and capable.
- 6 Conclusions: Solaris is presented as an early step toward a new operating layer with emerging interaction patterns.The conclusion frames generated software as an ongoing direction rather than a finished interface paradigm.
- 6 Conclusions: Generated interfaces could make the app no longer the unit of interaction by surfacing customized experiences around what users want to do.The paper contrasts this with fixed catalogs of apps for shopping, news, or reservations.
- 6 Conclusions: Solaris preserves scene coherence while responding to the same interaction request, producing interactions described as more natural and physically grounded.Figure 12 compares the systems on coherence of the entire scene.
- 6 Conclusions: A single generated interface can support diverse interior-design interactions, including changing wall colors, adjusting lighting and time of day, changing viewpoints, and modifying artwork.Figure 14 presents these interactions within one generated environment.
- 6 Conclusions: Interface world models remove translation between visual ideas and intermediate representations, allowing visual concepts to become interactive interfaces.The paper describes this as an alternative to UI frameworks, components, and code.
- 6 Conclusions: Generated storefronts can preserve brand identity while adapting products, layouts, colors, materials, and recommendations to individual intent in real time.The conclusion describes recognizable brands within hyper-personalized experiences.
- 6 Conclusions: Generated tutorials can render the next step in a user’s context, adapt as progress occurs, and recover when the user goes off script.This differs from replaying one fixed sequence for everyone.
- 6 Conclusions: The paper expects interface generation to become faster, more coherent, more controllable, and more capable across model generations.It characterizes obstacles to generated interfaces as increasingly solvable engineering problems.