PaperlayerView exact revision on arXiv

Source-linked AI summary

Macaron-A2UI: A Model for Generative UI in Personal Agents

Fancy Kong, Congjie Zheng, Murphy Zhuang, Rio Yang, Sueky Zhang, Hao Fu, Gene Jin, Song Cao, Kaijie Chen, Andrew Chen, Pony Ma

arXiv:2605.24830v1cs.HC

TL;DR

Static text-only chat is limited for personal-agent tasks requiring structured interaction, motivating Generative UI. Macaron-A2UI builds an A2UI corpus and benchmark, trains models with schema-light supervised fine-tuning and reward-driven reinforcement learning, and achieves 75.6 overall while surpassing the strongest full-prompt frontier baseline.

  • Problem

    Generative UI lacks grounded dialogue supervision, evaluations separating protocol validity from interaction quality, and evidence of capability without long schema prompts.

  • Method

    Macaron-A2UI constructs a heterogeneous A2UI corpus and benchmark, then trains assistants with schema-light LoRA supervised fine-tuning followed by reward-driven reinforcement learning.

  • Results

    75.6 overall: the best model surpasses the strongest full-prompt frontier baseline, with improvements in protocol correctness, interaction quality, and user experience.

  • Takeaways & Limitations

    Generative UI capability can be learned and internalized through training without depending on heavy schema prompting at inference time.

  • Takeaways & Limitations

    The evolving A2UI v0.8 protocol, model limitations for complex multi-turn interaction, and real-time generation, validation, and rendering latency constrain the system.

Abstract

from arXiv · show

As personal agents evolve to handle complex, user-centric tasks, static plain-text chat is rapidly becoming a bottleneck. Generative UI emerges as the necessary new interface layer, dynamically synthesizing the right controls, options, and state from the interaction context in real time. We present Macaron-A2UI, a model for Generative UI in personal agents. Our goal is to move beyond text-only interaction by enabling agents to generate natural language together with lightweight, executable UI actions for information collection, preference refinement, confirmation, and multi-goal organization. We build a large-scale Generative UI corpus from heterogeneous dialogue sources, introduce A2UI-Bench for controlled evaluation, and train 30B, 235B and 754B models with parameter-efficient LoRA-based supervised fine-tuning followed by reward-driven reinforcement learning. The best Macaron-A2UI model reaches 75.6 overall on A2UI-Bench without explicit schema hints, surpassing the strongest full-schema frontier baseline. We release the models, benchmark, and evaluation protocol to support future work on Generative UI for personal agents.

1 Introduction

Generative UI is presented as a necessary interface layer for personal agents, enabling executable, lightweight controls alongside natural language for structured interaction. Macaron-A2UI addresses the missing learning formulation with a validated corpus, A2UI-Bench, and schema-light two-stage training, reaching an overall score of 75.6.

  • Generative UI dynamically creates personalized interfaces during interaction, matching the user’s goal, context, and next actions.
  • Executable lightweight interfaces reduce the reading burden and cognitive load of providing information, comparing options, confirming decisions, or organizing multiple goals.
  • Generative UI for personal agents remains underexplored, lacking a unified formulation, large-scale UI-grounded supervision, and benchmarks separating protocol validity from interaction quality.
  • The proposed formulation generates a unified response containing natural language and executable UI actions from instructions, dialogue history, and the current user message.
  • Over 14,000 samples from four heterogeneous dialogue sources form the Generative UI corpus, while A2UI-Bench evaluates protocol validity, interaction quality, and visual metrics.
  • 75.6 overall score is achieved by the best model through LoRA-based supervised fine-tuning followed by reinforcement learning that improves executable interaction quality without long schema prompts.

2 Related Works

Related work has explored both foundation models that generate task-specific interfaces and agents that operate over existing web, screen, and GUI environments. These directions motivate Generative UI as an interface layer beyond linear text and advance multimodal interaction with digital interfaces.

  • Interface Generation: Generative UI research enables foundation models to synthesize rich, task-specific interfaces instead of returning only linear text.Chen et al. further studies proactively generated interfaces for information-dense tasks, evaluating functional, interactive, and emotional dimensions.
  • Research Directions: Together, these works broaden agent interaction beyond plain text through generated interfaces and operation over existing digital interfaces.The cited literature spans both proactive interface synthesis and multimodal interaction with web and desktop environments.
  • Interface-Operating Agents: A related research line develops agents for web browsing, screen-grounded GUI interaction, and scalable task or trajectory construction.Studies examine persistent, increasingly multimodal browsing and desktop GUI automation under diverse initial states.

3 Problem Formulation and A2UI Primer

The paper formulates Generative UI as producing natural language with structured A2UI messages when appropriate, rendered by clients through a trusted component catalog. It identifies protocol validity, interaction construction, and user-facing quality as the central challenges of A2UI generation.

  • A2UI generation takes system instructions, dialogue history, and the current user message as input, producing a unified response with natural language and, when appropriate, structured messages.
  • A2UI is declarative: models emit structured messages, while clients render them through a trusted component catalog instead of executing generated HTML, JavaScript, or framework-specific code.This separation is described as safer and more portable across rendering environments.
  • A2UI v0.8 uses four message types: surfaceUpdate, dataModelUpdate, beginRendering, and deleteSurface.They respectively update components, update application state, trigger rendering, and remove an existing surface.
  • The main challenges are protocol validity, interaction construction, and user-facing quality, including constraints on references, typing, widget choice, text grounding, and cross-turn state handling.
  • This formulation motivates a corpus, benchmark, and two-stage training pipeline in which supervised fine-tuning stabilizes response format and basic text–UI grounding before reinforcement learning.

4 A2UI Corpus Construction

The A2UI corpus combines four heterogeneous dialogue sources, normalizes their annotations, and uses hybrid rule-and-LLM generation to teach when and how to produce executable UI. The resulting 14,245-sample corpus emphasizes component diversity and achieves 99.2% final renderability after validation and retries.

  • Corpus sources: The corpus combines MultiWOZ 2.2, SGD, ESConv, and AnnoMI to cover task-oriented assistance, emotional support, and motivational interviewing.These sources differ in annotation schema, dialogue length, and interaction style, requiring normalization into a unified sample format.
  • Data representation: Each training sample pairs full dialogue history with an assistant response containing natural language and an optional A2UI payload.Consecutive same-speaker utterances are merged to enforce strict user–assistant alternation before annotation.
  • A2UI annotation: Hybrid generation uses rules for annotation-constrained task-oriented dialogues and a two-stage Editor–Author LLM process for under-specified open-domain dialogues.Deterministic post-processing corrects structural issues such as enum normalization, data-binding types, incomplete fields, and layout constraints.
  • Component augmentation: 4,165 augmented samples account for 29.2% of the final training set and target under-represented layout, interactive, and multimedia components.The augmentation expands coverage of rows, slider-like controls, icons, images, date/time inputs, modals, tabs, check boxes, video proxies, and audio proxies.
  • Validation: 99.2% final renderability is achieved across UI-turns, with 91.3% passing initially, 7.6% recovered through error-feedback retries, and 85 failures after three attempts.Validation checks format, structure, data binding, and semantic consistency.
  • Corpus composition: 14,245 assistant-turn samples comprise 10,210 UI-turns and 4,035 text-only turns, producing an overall UI ratio of 71.7%.Most no-UI samples are natural: 3,277 (81.2%) come directly from source dialogues, while 758 (18.8%) are augmentation-generated.

5 A2UI-Bench

A2UI-Bench is a controlled benchmark for evaluating Generative UI across task behaviors, quality levels, and user experience. It uses fixed task composition and shared protocols to compare models through language-side and visual-side evaluation.

  • Benchmark design: A2UI-Bench targets UI triggering, UI suppression, cross-turn consistency, and compositional organization while separating protocol correctness, functional quality, and user experience.Its fixed task composition and shared evaluation protocol support direct model comparison.
  • Task structure: The benchmark organizes evaluation into atomic single-turn tasks, depth multi-turn episodes, and width tasks combining multiple information needs.These task structures test turn-level UI decisions, state consistency, and unified multi-goal interaction planning.
  • Task coverage: 300 tasks cover diverse interaction intents, including explicit no_ui_chat cases that test whether models suppress unnecessary UI.Negative examples and structurally broad width tasks prevent the benchmark from becoming a simple single-turn form-generation test.
  • Evaluation metrics: Language-side evaluation scores protocol correctness with L1, task construction quality with L2, and user experience quality with L3.L1 checks parsing, schema, references, required fields, and value formats; L2 evaluates trigger and component alignment; L3 evaluates value addition, naturalness, and cognitive load.
  • Evaluation metrics: Visual-side evaluation renders model-produced A2UI and scores visual integrity, task alignment, and action clarity from screenshots.Only successfully parsed, non-empty, successfully rendered targets are included; depth tasks produce step-level visual targets.

6 Experiment

The experiment trains A2UI assistants with LoRA-based SFT followed by GRPO, using rewards centered on executable UI quality. Under schema-light evaluation, training substantially improves performance across model scales and outperforms untuned frontier models, while L1 improves earliest during RL.

  • Training pipeline: SFT teaches unified responses that jointly generate fluent text and protocol-compliant, executable UI actions.GRPO subsequently refines this behavior using interaction-oriented rewards and group-relative candidate advantages.
  • Reward design: Hard structural gates assign zero reward to malformed JSON, missing required UI, protocol failures, and render-critical errors.Passing responses receive weighted rewards for structural quality, task construction, and user-level utility.
  • Main results: 74.2 overall: Qwen-235B rises from 21.6 at base to 63.6 after SFT and 74.2 after RL in the schema-light regime.Qwen-30B improves from 19.8 to 37.2 after SFT and 58.8 after RL.
  • Main results: 25.5 overall: GPT-4o mini, DeepSeek-V3.1, and GPT-5.4 remain weak without schema hints, scoring 25.5, 21.9, and 23.9, respectively.With complete schema prompts, DeepSeek-V3.1 reaches 63.8 overall, Gemini-3.1 Pro reaches 71.0, and GPT-5.4 reaches 74.1.
  • Dataset and task analysis: 3.83: Macaron-A2UI-235B is strongest overall, outperforming GPT-5.4 at 3.75 and untuned Qwen3-235B at 3.37 across balanced datasets.MultiWOZ, SGD, ESConv, and AnnoMI scores range from 3.82–3.84.
  • RL training dynamics: L1 reward increases first and most rapidly during GRPO for both model scales, indicating structural executability improves before L2 and L3.Malformed outputs, schema violations, and render-critical errors are directly penalized.

7 Conclusion

The paper frames A2UI-based Generative UI as unified production of natural language and executable UI actions, supported by a corpus, benchmark, and two-stage training pipeline. Results show strong schema-light performance, while evolving protocols, model capability, and real-time latency remain limitations.

  • Contributions: The work unifies natural-language generation and executable UI actions under a fixed declarative protocol, supported by an A2UI-grounded corpus, A2UI-Bench, and schema-light two-stage training.The training pipeline combines supervised fine-tuning with subsequent optimization, and experiments cover 30B and 235B models.
  • Results: The best 235B model slightly surpasses the strongest full-prompt frontier baseline in overall score under minimal prompting.Training improves protocol correctness, interaction quality, and user experience by a large margin without requiring heavy schema prompting at inference time.
  • Limitations: The approach remains limited by the evolving A2UI protocol, currently version 0.8, bottlenecked model capability for complex multi-turn interaction, and real-time generation, validation, and rendering latency.These limitations are identified as barriers to bringing Generative UI into production environments.

A Appendix · A.1 A2UI Rendering Implementation

The appendix describes a Flutter Web renderer that converts model-generated A2UI JSON messages into interactive widget trees for production use and automated visual evaluation. It specifies the message protocol, reactive data handling, component catalog, and dedicated rendering route.

  • A.1 A2UI Rendering Implementation: The renderer is implemented as a Flutter Web application serving both the production chat client and a headless backend for automated visual evaluation.It transforms raw JSON message streams into fully interactive widget trees.
  • A.1 A2UI Rendering Implementation: A2UI outputs are JSON arrays whose messages use exactly four action types: beginRendering, surfaceUpdate, dataModelUpdate, and deleteSurface.beginRendering declares a surface and root tree; surfaceUpdate patches subtrees; dataModelUpdate writes reactive key-value state; deleteSurface removes a surface.
  • A.1 A2UI Rendering Implementation: beginRendering creates a UI surface with a surfaceId and root component tree, while surfaceUpdate replaces, inserts, or removes subtrees at specified hierarchy paths.These operations define surface creation and incremental component-tree modification.
  • A.1 A2UI Rendering Implementation: dataModelUpdate enables components to bind through JSON pointer paths, supporting pre-filled form values, selection defaults, and conditional visibility.The message writes key-value pairs into a reactive data model.
  • A.1 A2UI Rendering Implementation: 23 component types are registered in the Macaron design catalog across four functional categories.Each CatalogItem maps a component type and optional extensionType to a Flutter widget builder, JSON Schema data contract, and child-acceptance metadata.
  • A.1 A2UI Rendering Implementation: The dedicated /render route accepts a complete A2UI message array as a URL query parameter and processes it with the production renderer and message processor.It extracts a surfaceId from the first message containing one, falls back to a default identifier, and displays the resulting widget tree in a PreviewStage.

A.2 VLM-Based Visual Evaluation

The auxiliary VLM-based visual evaluation assesses whether structurally valid A2UI outputs render into usable interfaces. It complements protocol and LLM-judge evaluations by judging the end-to-end appearance users actually see.

  • Motivation: The visual layer addresses cases where schema-valid JSON still renders as a visually broken or misleading interface.It evaluates actual rendered appearance rather than relying solely on structural validity.
  • Render Check Gate: Seven deterministic render checks gate samples before visual evaluation, detecting renderer-critical defects not covered by schema validation.Checks include missing literalArray bindings, invalid widget layouts, multiple surface IDs, malformed action contexts, invalid selection values, incorrect date fields, and incomplete data-model updates.
  • Screenshot Capture and Cropping: Passing samples are rendered in Flutter Web, captured with Playwright and Chromium after readiness, and automatically cropped to preserve relevant page content.The cropping step reduces excessive whitespace for short interfaces while retaining scrollable content for long ones.
  • VLM Judge: A VLM judge scores each screenshot on three dimensions using integer scores from 1–5, conditioned on the task, scenario, dialogue, user message, and assistant response.The judge evaluates strictly what is visibly shown, without treating hidden or off-screen content as acceptable.
  • Relationship to L1–L3 Evaluation: The visual layer complements L1–L3 by evaluating the end-to-end rendered result and exposing rendering-specific defects that raw JSON evaluation may miss.L1 checks protocol and schema correctness, while L2 and L3 judge task construction and interaction experience from raw A2UI JSON.

B Experiments · B.1 Training Details · C A2UI Prompts

The paper documents a two-stage training pipeline and specifies A2UI generation and evaluation prompts around structured JSON messages, schema-constrained components, and interaction-quality judgments. Qualitative examples illustrate how dialogue is handed off to concise, actionable interfaces across motivational, emotional-support, and task-oriented settings.

  • B.1 Training Details: The training pipeline uses supervised fine-tuning followed by GRPO, with LoRA adaptation and grouped-rollout optimization.The reported hyperparameters focus on optimization, sampling, and reward composition while omitting engineering-specific options.
  • C A2UI Prompts: The A2UI corpus documentation includes minimal/full system prompts, L2/L3 LLM-judge prompts, and a V1–V3 VLM-judge prompt.Runtime placeholders remain unchanged, while long schema blocks are abbreviated in the paper and complete files are included in the released evaluation package.
  • C.1 System Prompts: Without an A2UI schema, the system prompt permits natural-language text with optional A2UI messages, while requiring valid JSON containing text_response and a2ui.The schema-guided prompt instead requires strict field-name compliance and provides 15 available components.
  • C.1 System Prompts: A2UI messages must contain exactly one action key, using beginRendering, surfaceUpdate, dataModelUpdate, or deleteSurface for rendering, component updates, data writes, and surface removal.Components belong inside surfaceUpdate.components, with beginRendering establishing the root and dataModelUpdate carrying typed values when needed.
  • C.1 System Prompts: The prompts enforce one surface per reply, matching beginRendering and surfaceUpdate structures, valid child-component references, and required selection bindings for interactive lists.They also specify absolute versus relative data-binding paths and require relative paths inside templated nested contexts.
  • C.2 LLM Judge Prompts: The L2 judge evaluates task-construction quality across trigger appropriateness, component-intent alignment, text-UI grounding, data-model utilization, and action completeness.The L3 judge separately evaluates experience quality through value-add over text, conversational naturalness, and cognitive load, using strict scoring criteria.
  • C.3 VLM Judge Prompt: The V1–V3 visual judge scores rendered screenshots from 1 to 5 for visual integrity, task alignment, and action clarity, explicitly penalizing visible defects.The prompt requires judging only visible content and reporting issues such as clipping, overflow, cramped layouts, misleading content, or unclear controls.
  • B Experiments: Four qualitative examples span motivational interviewing, emotional support, and task-oriented assistance, presenting sufficient dialogue context alongside the final utterance and rendered interface.The examples show reminder, planning, decision, and booking interfaces that make next steps, options, cognitive load, or transaction state easier to act on or verify.
Loading 2605.24830v1…