When Interfaces Speak: Data-Aware Generative UI Harness for Active Interaction

1 INTRODUCTION

• Text interaction can impose cognitive overload, ambiguity, information chaos, and slow input for complex, goal-directed tasks.Generated interfaces consolidate retrieved candidates and unresolved fields into a structured canvas for review and submission.Source: §1 · pp. 1–2
• GENUI-Harness places a GUI Coder Agent between the Tool Agent’s read-only retrieval and its downstream database action execution.The coder uses retrieved evidence to identify ambiguities and generate executable front-end code for user input.Source: §1 · p. 2
• Dynamic UX supports parallel, session-isolated execution and reward collection, while Reward Auditor uses task outcomes to diagnose and revise training rewards.Reward Auditor looks for failures receiving high scores and successful interactions undervalued by the criteria.Source: §1 · pp. 2–3
• 58.00% Pass@3 for GENUI-4B on Lite exceeds Claude Opus 5’s 46.67% Pass@3, after training raised the 4B backbone from 9.33% Pass@3.The paper also reports that GENUI-Harness has higher Pass@3 than smolagents, mini-swe-agent, and Pi on 8, 7, and 8 of the 9 shared backbones, respectively.Source: §1 · p. 3
• UI-TAU Bench evaluates generated-UI interaction on 10 real-world domain databases, with Lite and Full splits of 300 and 1,000 tasks.The benchmark tests whether interfaces support user input and the downstream action needed to reach a target database state.Source: §1 · p. 3

2 TASK DEFINITION

• Each instance specifies a user request, initial database state, query and action tools, and domain rules.Source: §2 · p. 3
• The agent uses read-only tools to retrieve relevant records, then generates a task-specific UI to obtain missing information or confirm the intended action.Source: §2 · p. 3
• The agent must use an action tool to update the database while satisfying the user’s requirements and applicable rules.The goal is to produce the requested state through tool use.Source: §2 · p. 3

3 UI-TAU BENCH

• UI-TAU Bench uses 10 real-world domain databases built from public sources, covering flight booking, hotel booking, e-commerce, and retail.Source: §3 · p. 4
• Domain data is normalized into user-facing, resource, and transaction tables so query tools can retrieve related records and action tools can update state under explicit rules.Source: §3 · p. 4
• Query tools are validated for error-free, non-empty, argument-consistent results, while action tools must obey domain rules and produce the intended database state.Tools failing checks are revised and revalidated at most twice, then cross-checked by a second annotator.Source: §3 · p. 4
• Task synthesis masks internal identifiers and additional arguments, requiring agents to recover missing information through database exploration, user interaction, or both.Moderate-ambiguity tasks mask 2–3 additional arguments; high-ambiguity tasks mask 4–5.Source: §3 · p. 4
• Pass@3 measures success in at least one of three sessions, while Avg@3 averages success across all three.A session counts as successful only when execution produces exactly the reference database state.Source: §3 · p. 4

4 GENUI-HARNESS

• The Tool Agent retrieves records and executes the final action, while the GUI Coder Agent turns retrieved evidence and unresolved arguments into a user-operated interface.Separating roles makes the interface component independently trainable and pairable with a different tool-reasoning model.Source: §4 · p. 5
• Before UI generation, the Tool Agent uses read-only tools to retrieve records and the user profile, grounding known arguments and gathering candidate options.Source: §4 · p. 5
• The GUI Coder selects controls for unresolved arguments and renders retrieved candidates with readable labels, while resolved arguments appear as context or prefilled values.The user’s submission returns as a JSON object keyed by action arguments.Source: §4 · p. 5
• After receiving user inputs, the Tool Agent may make up to five further query turns, completes the action arguments, and executes the action tool.Source: §4 · p. 5

5 TRAINING

• Training environments use 40 synthesized databases, split into 25 for supervised fine-tuning and 15 for reinforcement learning.Retained successful trajectories match target database states and render with valid ARIA snapshots; the two agents are fine-tuned separately, then only the GUI Coder is trained with GRPO while the Tool Agent stays fixed.Source: §5 · pp. 5–6

5.1 DYNAMIC UX

• Dynamic UX isolates each rollout in its own browser context while sharing one rendering runtime and browser process.It initializes the server, router, browser, and component library once, then returns execution status and an initial ARIA snapshot; session routing prevents state crossing between samples.Source: §5.1 · p. 6

5.2 REWARD AUDITOR

• Judge rewards can give identical zero scores to different execution failures or high scores to rendered interfaces that cannot submit intended values.Static checks cannot establish rendering or interaction success, while verifying task completion requires user interaction, action execution, and checking the resulting database state.Source: §5.2 · p. 6
• Reward Auditor compares reward predictions with exact database-state outcomes, diagnoses mismatches, and revises shared judging criteria and scoring rules.It inspects internal rollouts, replays candidate rewards on stored interfaces, and holds generated code fixed so score changes reflect reward revisions; two revision rounds produce the final training reward.Source: §5.2 · p. 7, Figure 4 · p. 7
• The final reward distinguishes code-extraction, structural, rendering, and empty-interface failures before quality and accessibility scoring.The initial rubric scores functional equivalence, feature completeness, implementation correctness, and task alignment without requiring exact code agreement with a reference.Source: §5.2 · pp. 6–7

6.1 SETUP

• The four harnesses share tasks, databases, tools, user simulator, evaluator, and backbone per row, while GENUI-Harness uses generated UI and baselines retain text interaction.The Lite comparison uses three sessions and reports Pass@3 and Avg@3.Source: §6.1 · p. 7, Table 2 · p. 8

6.2 MAIN RESULTS

• 4.48 percentage points: GENUI-Harness improves macro-averaged Pass@3 over smolagents and leads on seven of nine backbones.The task-paired bootstrap 95% confidence interval for the gain is [2.37, 6.59]; the comparison measures complete system performance, not the interface channel alone.Source: §6.2 · p. 7, Table 2 · p. 8
• 36–43% fewer agent turns: GENUI-Harness occupies the favorable higher-Avg@3, lower-turn frontier among successful episodes.The authors suggest the Tool Agent handles retrieval and validation while the UI collects multiple grounded choices in one structured interaction.Source: Figure 5 · p. 8, §6.2 · p. 8
• 58.00% Pass@3 and 36.00% Avg@3: GENUI-4B with the final audited reward achieves the best Figure 6 result and exceeds the strongest Table 2 baseline on both metrics.Role-specific SFT raises Avg@3 from 3.22% to 27.67% before RL.Source: Figure 6 · p. 8, §6.2 · p. 8

6.3 ANALYSIS

• 52.0% of failures involve unsuccessful UI interaction and 44.5% involve mismatched action arguments across four models on Lite.Examples include a single-select control that cannot express a required multi-selection and an immutable cabin-class field that submits the wrong value.Source: §6.3 · pp. 8–9
• 3.4 to 1.2 average dialogue rounds: generated UIs reduce rounds in a reviewer comparison on 100 tasks with valid interaction in all three sessions.UI latency was judged acceptable on 85% of tasks by a majority of UI reviewers.Source: §6.3 · p. 9
• GUI-Coder replacement increases Lite Avg@3 for all five tested reasoning models when the pre-agent output and post-agent backbone are held fixed.This intervention changes only interface construction.Source: §6.3 · p. 9
• GENUI-4B remains strongest on both ambiguous and non-ambiguous Lite variants, with tools, databases, simulator, and evaluation protocol fixed.The non-ambiguous variants reveal user-facing arguments masked in the original requests.Source: §6.3 · p. 9

• Tool-using agents and state-based benchmarks study planning, API execution, and task completion while generally retaining language-based interaction.Source: §7 · p. 9
• Clarification and mixed-initiative systems use questions, structured options, or generated interfaces, but typically separate interface generation from database-backed action execution.Source: §7 · p. 9
• Execution-based UI-code methods test functional correctness but treat code generation as the endpoint rather than part of an agent loop.Source: §7 · p. 9

8 CONCLUSION

• The evaluated system outperforms all evaluated baselines; among successful episodes, GENUI-Harness uses 36–43% fewer agent turns, while generated UIs reduce reviewer dialogue from 3.4 to 1.2 rounds.Source: §8 · p. 10

A.1 FORMAL TASK DEFINITION AND EVALUATION

• Each task combines a user query, initial database, domain-specific tools, and a goal; success requires the terminal action to produce the target database state.Read-only queries gather evidence, the generated interface collects input, and one terminal action updates the database.Source: §A.1 · p. 17

A.2 DETAILED AGENT WORKFLOW

• The Tool Agent explores before and after UI generation, while the GUI Coder produces one interface; the fixed simulator operates it without direct tool or database access.The authors use the simulator as a controlled capable-user proxy, not as a reproduction of the full distribution of human interaction behavior.Source: §A.2 · p. 17

B RELATED WORK

• Unlike dialogue-based Tau-Bench, this setting elicits user choices through an executable interface grounded in retrieved records and the action schema.Related work also distinguishes this database-state-checked task from interface revision and outcome-based code-feedback settings.Source: B RELATED WORK · pp. 17–18

C UI-TAU BENCH CONSTRUCTION DETAILS

• The appendix documents UI-TAU Bench’s schema design, database population, domain rules, tool generation and validation, task synthesis, and quality control.These are the stated topics of the benchmark-construction details.Source: C UI-TAU BENCH CONSTRUCTION DETAILS · p. 18

C.1 DETAILED CONSTRUCTION PIPELINE

• Benchmark construction proceeds from public-domain data through rule-based tool design to task synthesis, with tool and task execution checks before retention.Tasks mask internal identifiers and additional arguments; retained actions are checked against intended database states and task details for alignment and realism.Source: §C.1 · pp. 18–20

C.2 BENCHMARK STATISTICS AND COMPARISON

• UI-TAU Bench uses generated front-end code to elicit task information before tool execution updates an objectively evaluated database state.Its Lite and Full splits maintain comparable query length, ambiguity density, and annotation agreement, while Full broadens task and action-tool coverage.Source: §C.2 · p. 20, Table 5 · p. 20, Table 6 · p. 21

D.1 TRAINING DATA CONSTRUCTION

• Training uses 40 databases disjoint from the 10 evaluation domains, split into 25 SFT and 15 RL databases with no overlap between training sets.Trajectories are retained only when the terminal action executes and reaches the target database state.Source: §D.1 · pp. 21–22, Table 7 · p. 21

D.2 MODEL AND OPTIMIZATION DETAILS

• The Tool Agent and GUI Coder Agent start from separate Qwen3.5-4B checkpoints; during GRPO, the Tool Agent remains fixed and only the coder is optimized.GRPO uses sampled interface groups and their relative rewards to compute advantages, without a separate value model.Source: §D.2 · p. 22

D.3 EVALUATION INFERENCE CONFIGURATION

• Evaluation uses the listed open-weight and closed-source baselines, with Lite harness comparisons at temperature 0.7 and isolated database copies for each complete session.Browser rendering and user-simulator execution each have a 60-second timeout; structured generations allow at most three extraction attempts.Source: §D.3 · p. 23

D.4 BOOTSTRAP CONFIDENCE INTERVALS FOR LITE RESULTS

• All 95% paired-bootstrap confidence intervals remain above zero for both metrics across 10,000 resamples of the 300 Lite tasks.Differences are macro-averaged across nine backbones, using the same sampled task identifiers and retaining all three sessions for each sampled task.Source: §D.4 · p. 23, Table 10 · p. 23

D.5 BASELINE HARNESS ADAPTATION

• The three baselines retain their original text-based agent loops and share the tasks, tools, hidden goal, simulator model, and final-state evaluator.Framework-specific changes adapt tool and termination boundaries; format adapters do not alter generated arguments, observations, database state, or grading.Source: §D.5 · pp. 23–24

D.6 FULL SPLIT RESULTS

• 40.90% Success Rate: role-specialized GENUI-4B on the 1,000-task Full split under the single-session protocol.Table 11 reports State EM Pass@1 (single-session Success Rate, %).Source: Table 11 · p. 24, §D.6 · p. 24

D.7 DYNAMIC UX SYSTEMS BENCHMARK

• At concurrency 16, Dynamic UX improves throughput by 37.5%, lowers median and P95 latency by 29.2% and 41.3%, and reduces peak RSS by 48.1%.It uses 76.0% fewer Chromium processes than fresh-process rendering at concurrency 16; at concurrency 32, it retains 11.3% higher throughput, 50.9% lower peak RSS, and 79.7% fewer processes. In a separate 1,000-session concurrency-32 test, all sessions complete without timeout or rendering failure, with no cross-session markers.Source: §D.7 · p. 24

D.8 INITIAL REWARD SPECIFICATION

• The initial judge scores functional equivalence, feature completeness, implementation correctness, and task alignment against a successful reference trajectory.Direct RL uses this judge score after extraction, evidence, structural, rendering, and response checks; failed checks receive zero reward.Source: §D.8 · pp. 24–25

D.9 META-REWARD REVISION PROTOCOL

• Reward Auditor compares complete rewards with terminal database-state outcomes, diagnoses mismatches from trajectories, and replays proposed revisions before accepting them.Automated revisions consolidate task-specific diagnoses into a shared specification that remains fixed during each RL session.Source: §D.9 · pp. 25–26
• The final reward applies ordered failure checks before combining the quality score with a source-based accessibility multiplier.Three weighted soft checks cover non-string labels, overlong labels, and Select controls without a linked label; label-collision checks have zero weight.Source: §D.9 · pp. 26–27

E DETAILED ERROR ANALYSIS

• Across 800 failures, 52.0% had no valid UI interaction, 44.5% involved correct-tool calls with mismatched arguments or state, and 3.5% were rejected during execution.The partition covers one session on each of four models across the 300 Lite tasks; the columns are mutually exclusive.Source: Table 15 · p. 27, E DETAILED ERROR ANALYSIS · p. 27
• A single-select hotel control could not represent two requested hotels, while an immutable cabin field submitted “Economy” instead of “Premium Economy.”These examples attribute obstruction to generated controls and submission logic, even when the user’s goal is fully specified.Source: E DETAILED ERROR ANALYSIS · p. 28
• Generated identifiers, list ordering, and free-text differences count as failures under strict state comparison, but mismatches alone do not establish their cause.Attribution requires the corresponding execution trace and domain semantics.Source: E DETAILED ERROR ANALYSIS · p. 28

• 100 tasks were sampled from the four models’ common pool after all three sessions met interface-validity and usable-submission checks.This comparison concerns communication on consistently usable interfaces; benchmark state-EM results separately include unsuccessful executions and interactions.Source: F COMMUNICATION-CHANNEL SURVEY PROTOCOL · p. 28
• Six reviewers were randomly split into equal text and UI groups, with each reviewer seeing each task in only one condition.Task-level pairing prevents reviewers from carrying answers learned in one version into the other version of the same task.Source: F COMMUNICATION-CHANNEL SURVEY PROTOCOL · pp. 28–29
• Both conditions used the same query, retrieved options, and task state; a fixed converter derived text from each interface’s ARIA snapshot.This holds available task information fixed so the comparison concerns presentation rather than independently retrieved content.Source: F COMMUNICATION-CHANNEL SURVEY PROTOCOL · p. 29
• Reviewers could ask follow-up questions in either condition, and interactions ended when an action produced the target database state.The evaluation allowed the exchanges needed for completion instead of enforcing the benchmark’s fixed interaction sequence.Source: F COMMUNICATION-CHANNEL SURVEY PROTOCOL · p. 29
• 3.4 rounds for text versus 1.2 for generated UI, averaged across 300 task–reviewer interactions per condition through successful completion.A round counts a user message or interface submission and the corresponding agent response; multiple UI edits in one submission count once.Source: F COMMUNICATION-CHANNEL SURVEY PROTOCOL · p. 29
• 85/100 tasks (85%) passed the latency-acceptability criterion, based on at least two of three UI reviewers judging latency acceptable.These outcomes describe selected tasks with consistently valid interfaces.Source: F COMMUNICATION-CHANNEL SURVEY PROTOCOL · p. 29

G LIMITATIONS

• Benchmark evaluation may overstate user capability because its Playwright simulator sees generated source and the goal, while the survey covers only consistently usable interfaces.Code execution and accessibility-tree evidence may miss visual problems such as overlapping or confusing layouts, and strict database-state equality may reject semantically equivalent outcomes with different identifiers, ordering, or free text.Source: G LIMITATIONS · p. 29
• Source-blind evaluation and adaptive interface regeneration after clarification are identified as future directions.Source: G LIMITATIONS · p. 29

H PROMPT TEMPLATES

• The prompt appendix describes Tool Agent exploration and execution templates, an initial four-criterion reward judge, and a Claude Opus 5-based meta-reward revision and review protocol.It also points to Appendix D.9 for the final reward composition and revision protocol.Source: H PROMPT TEMPLATES · p. 29
• The Playwright template requires an exactly named `interact_with_page(page: Page)` function, compatible with version 1.53.0, and assumes the UI is already loaded.The function should use the UI to complete the task and include the necessary Playwright imports.Source: H PROMPT TEMPLATES · pp. 33–34
• The evaluator compares predicted and ground-truth code using execution results and accessibility snapshots across functionality, feature completeness, implementation correctness, and task alignment.The rubric prioritizes functional equivalence and asks judges to assess what actually renders, while focusing on functionality rather than implementation details.Source: H PROMPT TEMPLATES · p. 34
• Each rubric dimension receives an integer score from 0 to 5, with instructions to use the full range rather than clustering around 3–4.The examples distinguish excellent, adequate, poor, and failed implementations by functionality and missing or broken features.Source: H PROMPT TEMPLATES · p. 35