Source-linked AI summary
Generative UI: LLMs are Effective UI Generators
Yaniv Leviathan, Dani Valevski, Matan Kalman, Danny Lumen, Eyal Segalis, Eyal Molad, Shlomi Pasternak, Vishnu Natchu, Valerie Nygaard, Srinivasan, Venkatachary, James Manyika, Yossi Matias
TL;DR
LLMs commonly present generated content through static markdown interfaces, motivating Generative UI, which generates custom interfaces alongside content. The paper combines tools, detailed instructions, and post-processors to produce complete interactive web pages, finding strong user preference over markdown and comparable quality to expert pages in half of cases. It also releases PAGEN and identifies generation speed and occasional rendering errors as limitations.
Problem
LLM outputs are commonly presented through static, markdown-based interfaces rather than interfaces generated for the content and prompt.
Method
The implementation combines accessible tools, detailed system instructions, post-processors, and expert-built PAGEN examples to generate complete web pages and assets.
Results
Generative UI is preferred over Markdown UI in 82.8% of comparisons and matches expert-page quality in 50% of cases.
Takeaways & Limitations
Robust Generative UI emerges with newer models and provides custom interactive interfaces for prompts, while PAGEN supports future evaluation.
Takeaways & Limitations
Generation often takes a minute or two, and JavaScript, CSS, or HTML errors can occasionally occur.
Abstract
from arXiv · showhide
AI models excel at creating content, but typically render it with static, predefined interfaces. Specifically, the output of LLMs is often a markdown "wall of text". Generative UI is a long standing promise, where the model generates not just the content, but the interface itself. Until now, Generative UI was not possible in a robust fashion. We demonstrate that when properly prompted and equipped with the right set of tools, a modern LLM can robustly produce high quality custom UIs for virtually any prompt. When ignoring generation speed, results generated by our implementation are overwhelmingly preferred by humans over the standard LLM markdown output. In fact, while the results generated by our implementation are worse than those crafted by human experts, they are at least comparable in 50% of cases. We show that this ability for robust Generative UI is emergent, with substantial improvements from previous models. We also create and release PAGEN, a novel dataset of expert-crafted results to aid in evaluating Generative UI implementations, as well as the results of our system for future comparisons. Interactive examples can be seen at https://generativeui.github.io
1 Introduction
Generative UI extends AI output from content to complete, custom interactive experiences rather than static interfaces or markdown walls of text. The paper presents an implementation whose results are strongly preferred over standard markdown and can approach expert-crafted quality.
- Generative UI generates both content and the entire user experience, enabling custom interactive experiences with rich formatting, images, maps, audio, simulations, and games.
- Generative UI aims to provide a custom experience for each prompt, replacing extended product, UX, and engineering work with an AI-generated interface.
- The prevalent LLM interface is markdown-based chat, while the paper’s implementation produces results that users overwhelmingly prefer over standard markdown.
- Human experts’ pages are broadly preferred, but the implementation achieves comparable results on a large fraction of custom prompts.
2 Method
The implementation generates complete web pages and assets using tools, detailed system instructions, and post-processors. Prompt changes can also impose consistent visual styles across generated results.
- The system outputs a fully generated web page and accompanying assets that are rendered directly in the user’s browser.
- The system overview organizes the implementation into three main components.
- The implementation combines tool endpoints, carefully crafted system instructions, and post-processors to improve generation quality and address common issues.
- 2.1 Consistent Styling: Detailed “Classic” and “Wizard Green” style instructions specify colors, fonts, and other properties, and generated results follow those styles.
3 Results
The evaluation compares Generative UI with human-made websites, search results, plain text, and Markdown UI using human preferences. Generative UI is strongly preferred over standard formats, while newer models and full prompting improve results.
- The evaluation uses pairwise human preferences across five result formats on 100 sampled LMArena prompts, with generation time excluded.Each result was sent to two raters using pre-cached generations.
- Generative UI achieves an ELO score of 1736.2 on LMArena, indicating strong user preference except against human experts.
- 82.8% of the time, raters prefer Generative UI to Markdown UI on LMArena prompts.On the Information Seeking prompt set, preference for the implementation reaches 90.5%.
- 3.1 Emergent Capability: Newer Gemini models produce stronger user preference and fewer errors, supporting Generative UI as an emergent capability.
- 3.2 Prompt Ablations: Raters significantly prefer interfaces generated with the full prompt over those produced with a minimal prompt, although the minimal prompt still performs reasonably.The minimal prompt specifies image search, image generation, and valid HTML output.
4 The PAGEN Dataset
The paper introduces PAGEN, a publicly released dataset of expert-made websites paired with prompts. It was constructed to support clear and consistent evaluation of Generative UI systems.
- PAGEN is a publicly available dataset of human-expert-made websites for prompts from the LMArena and Info-Seeking evaluation sets.
- The dataset enables comparisons between Generative UI results and expert-human-made websites.
- The authors contracted highly rated independent web developers to create paired prompt-and-website examples with consistent timing and conditions.
5 Related Work
Related work spans natural-language UI generation, LLM code generation, and AI interaction paradigms. Generative UI differs from static markdown and fixed templates by generating bespoke interfaces directly.
- UI Generation from Natural Language: Earlier UI-generation systems used structured inputs or constrained languages to target specific platforms.
- UI Generation from Natural Language: Deep-learning approaches later translated visual inputs such as sketches and screenshots directly into interface code.
- Large Language Models for Code Generation: Advances in LLM code generation provide the capabilities underlying systems that translate natural language into functional code.
- Interaction Paradigms for AI: Generative UI moves beyond static markdown and fixed widget templates toward bespoke, dynamic experiences tailored to each prompt.Examples include games, simulators, and custom data visualizations.
6 Discussion
The paper presents Generative UI as a capable but still imperfect way to create custom interfaces, with strong user preference and expert-level quality in many cases. It also identifies slow generation and occasional code errors as important limitations.
- Results: 83% of evaluated cases preferred the Generative UI implementation over standard markdown when generation speed was ignored.The comparison used human user preferences.
- Results: Newer models increased user preference and reduced generation errors compared with previous models.The paper characterizes robust Generative UI as an emergent capability of newer, more capable models.
- Results: 50% of cases matched the quality of expert-built pages, although expert humans remained better overall.The comparison is based on the PAGEN dataset of expert-built websites.
- Limitations: Generation often takes a minute or two, while occasional JavaScript, CSS, and HTML errors remain important limitations.Streaming can reduce the initial waiting time by about half.
- Future directions: The authors envision an infinite catalog of interfaces generated on demand for individual user needs.This future direction is presented alongside acknowledged limitations of early Generative UI.
A.1 Selected Examples
The paper provides interactive access to all presented examples through the Generative UI website.
- Interactive access: All examples presented in the paper can be viewed interactively.The examples are available through the project website.
- Interactive access: The examples are hosted at generativeui.github.io.The passage identifies the project website directly.
- Interactive access: The website supplements the paper’s presentation with access to the generated examples.Readers can inspect the examples beyond the static paper text.
A.1.1 Fractal Explorer
For the fractals prompt, the system generated an immersive interactive webpage called “Fractal Explorer” that combines historical explanation with mathematical exploration tools.
- Prompt: The prompt asked for an in-depth explanation of fractals covering the topic in detail.The request explicitly emphasizes depth and completeness.
- Fractal Explorer: The generated “Fractal Explorer” page explains self-similarity and the history of fractal discovery.Its historical account runs from Weierstrass’s “monsters” to Benoit Mandelbrot’s modern definitions.
- Fractal Explorer: Interactive tools include a Dimension Calculator that visually demonstrates fractal concepts.The page combines explanatory content with interactive mathematical exploration.
- Fractal Explorer: Figure 5 presents the generated web app for “Explain fractals.”The figure documents the example as a generated web application.
A.1.2 History of Time Keeping Devices
For the timekeeping-history prompt, the system generated “Chronos,” a dark-themed interactive timeline spanning ancient methods through quartz and atomic precision.
- Prompt: The prompt requested a history of timekeeping devices.The example uses a concise historical topic as its input.
- Chronos: The generated “Chronos: A History of Timekeeping” page uses a visually immersive dark theme and scroll-animated timeline.The timeline organizes the historical material into six distinct eras.
- Chronos: The timeline progresses from Egyptian obelisks and water clocks through mechanical and pendulum innovations to quartz and atomic precision.Christiaan Huygens is identified among the innovators in the mechanical and pendulum stages.
- Chronos: Figure 6 presents the generated web app for “History of Time Keeping Devices.”The figure documents the timekeeping example as a generated web application.
A.1.3 Memory Game
The system generated a responsive memory-matching game from a prompt requesting large cards with humorous faces and props.
- The prompt requested a match-up memory game with large cards showing people making funny expressions and wearing funny props.
- The generated “Funny Faces Memory Match” uses a responsive grid of flip-cards to test recall.The interface includes 3D transform effects and real-time progress tracking.
- Figure 7 presents the generated “Memory Game” web-app.
A.1.4 Basketball Math
The system generated a basketball-themed educational application for teaching arithmetic and binary representation to a young child. Its interface organizes learning into playful game modes centered on basketball.
- The prompt requested teaching addition, subtraction, multiplication, and binary representation to a five-year-old using cute basketball-playing creatures.
- The generated “Little Ballers Math Academy” provides four game modes: addition, subtraction, multiplication, and binary counting.The modes are named “Passing Practice,” “Taking Shots,” “Team Drills,” and “Alien Scoreboard.”
- Figure 8 presents the generated “Basketball Math” web-app.