InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?

1. Introduction

• Natural-language website generation lowers participation barriers, but non-expert users’ ambiguous, low-code instructions create a semantic gap between their intent and model understanding.The introduction identifies this gap as a major bottleneck to real-world deployment.Source: §1 · p. 1
• Existing benchmarks use standardized, well-structured instructions, while non-expert users may provide ambiguous, fragmented, redundant, or conflicting requirements.The paper argues that these differences make idealized static evaluation insufficient for real-world interactive scenarios.Source: §1 · p. 1
• The paper defines blind execution as passively implementing incomplete or conflicting requests without inferring intent or proactively seeking clarification.It links this behavior to task failures, UI rendering anomalies, and functional hallucinations.Source: §1 · p. 2
• Escaping blind execution requires dynamic intent recognition, with agents interleaving clarification and visual verification in an executable frontend environment.The introduction argues that text-only dialogue is insufficient for this visually intensive, interdependent task.Source: §1 · p. 2
• InteractWeb-Bench is introduced as the first multimodal interactive benchmark for non-expert website generation, using persona-driven instruction mutation and Clarify, Implement, Verify, and Submit actions.Agents can switch among eliciting requirements, writing code, and visually inspecting rendered pages before submitting them for constraint–slot-based evaluation.Source: §1 · pp. 2–3
• The authors report extensive experiments on frontier MLLM-based agents and analyses of interaction, intent recognition, and task-solving behavior.They frame the findings as identifying factors for better alignment with non-expert users’ needs.Source: §1 · p. 3

Website Generation Benchmarks

• Existing website-generation benchmarks move from static screenshot-to-code tasks to multi-page apps, but assume comprehensive instructions; InteractWeb-Bench targets realistic ambiguous and contradictory inputs.Examples include Design2Code [Si et al., 2025], Web2Code [Yun et al., 2024], and WebGen-Bench [Lu et al., 2025].Source: §2 · p. 3

Intent Clarification and User Interaction

• ClarifyCoder [Wu et al., 2025], HumanEvalComm [Wu and Fard, 2025], ClarifyMT-Bench [Luo et al., 2025], and Persona2Web [Kim et al., 2026] address ambiguity but lack active language elicitation integrated with visual verification.Persona2Web infers intent passively from browsing histories; InteractWeb-Bench combines language requirement elicitation with iterative visual feedback.Source: §2 · p. 3

Interactive Agents and Execution Environments

• Interactive coding frameworks and benchmarks use execution feedback across turns, but focus on predefined tests rather than eliciting unstated user needs.InteractWeb-Bench adds Clarify, Implement, Verify, and Submit actions for requirement clarification, code synthesis, and GUI-based visual debugging.Source: §2 · p. 3

3.1. Persona-Driven User Agent Module

• Four persona mutations model sparse, rambling, aesthetic or ambiguous, and contradictory users, testing elicitation, noise filtering, cross-modal alignment, and conflict resolution.The design draws on Requirement Engineering defect taxonomies [Van Lamsweerde, 2009, Wu and Fard, 2025] and Grice’s Maxims; the mutations apply persona-specific operators to golden instructions [Chen et al., 2020].Source: §3.1 · pp. 4–5
• A two-stage retrieval-and-containment pipeline answers only details explicitly requested, then renders them in persona-specific style to prevent golden-instruction leakage.The P-RAM persona adds irrelevant conversational noise to an answer, while P-CON briefly defends a contradiction before yielding the accurate baseline truth.Source: §3.1 · p. 5

3.2. Interactive Agent Framework and Execution Environment

• Agents autonomously choose Clarify, Implement, Verify, or Submit; verification uses a checklist and fused interaction, code, and audit memory to inspect pages and return visual and diagnostic feedback.To bound exploration, task turn limits are 15, 20, or 25, and consecutive verification-error limits are 6, 8, or 10, depending on task complexity.Source: §3.2 · pp. 5–6

3.3. Constraint-Slot-Based Evaluation Metric and Dataset Statistics

• Evaluation uses complexity-weighted atomic oracle slots for task completion and separately tracks unrequested UI elements through hallucination rate.An independent visual evaluator using WebVoyager [He et al., 2024] with Set-of-Mark prompting verifies passed slots; slot weights reflect component complexity and constraint count.Source: §3.3 · p. 6
• The benchmark expands 101 seed websites, each with 7–12 atomic constraints, into 404 test cases using four persona-driven mutations.Tasks are clustered into Easy, Middle, and Hard difficulty levels using K-Means with k=3.Source: §3.3 · pp. 6–7

4.1. Experimental Setup

• Agents were built on bolt.diy, tested in a Playwright-based interactive browser environment, and evaluated by GPT-5-mini using WebVoyager augmented with Set-of-Mark prompting.DeepSeek-V3.2 simulated persona-specific users, and tasks began from textual instructions alone.Source: §4.1 · p. 7

4.2. Main Results

• Qwen3.6-Plus achieved the best TCR at 38.78%, yet completion fell as tasks became harder and varied substantially by persona, with higher scores under P-RAM than P-MIN.The authors interpret this as sensitivity to instruction perturbations, particularly when key information is missing.Source: Table 2 · p. 7, §4.2 · p. 7

Finding 1: Although agents achieve strong

• Qwen3.6-Plus and Kimi-K2.5 produced over 1,000 Avg. LoC while maintaining HR above 60%, indicating code over-generation rather than clarification to compensate for missing requirements.The authors note that hallucinated features, redundant components, and misaligned designs can degrade usability and increase maintenance overhead.Source: Table 3 · p. 8, Finding 1: Although agents achieve… · p. 8
• GPT-4.1-mini asked more questions and had lower SR, while Qwen3.6-Plus and Kimi-K2.5 asked fewer questions, generated larger outputs, and submitted more often after shorter trajectories.These patterns reflect a contrast between exploration and early commitment, which can risk hallucinated or misaligned components.Source: Table 3 · p. 8, Finding 1: Although agents achieve… · pp. 9–10
• VBR was typically below 10% but visual defects persisted; Kimi-K2.5 scored highest on aesthetics, and MLLM judges correlated moderately with human judges (Kendall’s τ=0.4490).Human experts showed moderate inter-rater agreement (Kendall’s τ=0.5675), and Kimi-K2.5 also ranked best with human judges.Source: Table 3 · p. 8, Figure 5 · p. 10, Finding 1: Although agents achieve… · p. 10
• Execution steps ranged from 5.02 to 12.87 and VCI from 0.35 to 1.51 without clear task-completion gains; Gemini-3.1-Flash-Lite scored 25.72 TCR.The authors find that agents tend to apply local fixes rather than reconsider incomplete or incorrect requirements.Source: Finding 1: Although agents achieve… · pp. 8–9, Figure 3 · p. 9
• Representative cases across the golden instruction and four personas show agents prematurely implementing instead of clarifying underspecified or conflicting requirements, trapping them in blind execution.The paper also presents generated website cases across models.Source: Finding 1: Although agents achieve… · p. 10

• InteractWeb-Bench evaluates agents’ handling of ambiguous, underspecified instructions and reveals blind execution, with plausible websites often deviating from users’ actual needs.The benchmark targets realistic website-generation scenarios.Source: §5 · p. 10
• Website-generation agents need to recognize missing information, elicit clarifications, and update assumptions from feedback; InteractWeb-Bench can support future user-aligned agents.Source: §5 · pp. 10–11

A. More Details about Evaluated Models

• Table 4 reports model sizes, maximum context and output lengths, and access methods; all evaluated models accept images, up to five per input.The details of API-model costs on InteractWeb-Bench are summarized in Table 5.Source: A. More Details about Evaluated Models · p. 13
• Closed-source models and large open-source models were evaluated through official APIs for a fair, consistent comparison.The section also reports total costs incurred by API-based models on InteractWeb-Bench.Source: A. More Details about Evaluated Models · p. 13
• Smaller open-source models were deployed with vLLM [Kwon et al., 2023] on 8 NVIDIA A800 GPUs using the same hyperparameters.Source: A. More Details about Evaluated Models · p. 13

B. More Details about Human Evaluation

• Three computer science PhD students evaluated aesthetic quality using the MLLM judge-prompt criteria, with average scores reported across models.The detailed averages are presented in Table 6.Source: B. More Details about Human Evaluation · p. 13, Table 6 · p. 14

C. Detailed Case Study

• Persona-conditioned cases expose failures in intent recovery, replanning, verification, and completion of key website requirements.The cases include repeated repair loops, premature submission after local success, verification deadlock, and ignored feedback.Source: C. Detailed Case Study · p. 14, Figure 6 · p. 14, Table 7 · p. 16
• Kimi-K2.5 favored harmonious color palettes, Qwen3.6-Plus added visual assets, GPT-4.1 used textured backgrounds, and Gemma-4-31B-it showed structured image layouts.Figure 7 presents website cases generated by these four models.Source: C. Detailed Case Study · pp. 14–15, Figure 7 · p. 16
• The post-verification repair loop repeatedly fixed a title emoji but missed broken sorting logic, preventing score, feedback, leaderboard, and rewards from working.The model did not clarify or reassess the overall task state.Source: C. Detailed Case Study · p. 15
• P-MIN clarified and fixed a runtime error, then submitted a basic sorting game without recovering the leaderboard, feedback, rewards, instructions, or requested colors.The case shows reliance on one local verification success instead of checking all requirements against the original task.Source: C. Detailed Case Study · p. 15
• Another persona-conditioned case became trapped repairing drag-and-drop logic, leaving item removal, scoring, and success feedback broken while other high-priority requirements remained unmet.Only superficial visual style and color were satisfied.Source: C. Detailed Case Study · p. 15
• P-INT failed to translate abstract sensory instructions into concrete features, ignored verification errors, and submitted a static page with broken interactions and missing requested colors and feedback.The passage reports a low score and hallucination penalty.Source: C. Detailed Case Study · p. 15

Prompts for Website Generation Agent

• The website-generation agent’s system prompt defines its action space and output constraints, while a GUI-based prompt guides Verify.The prompts are shown in Figures 9 and 8, respectively.Source: D. Prompts · p. 16, Figure 8 · p. 17, Figure 9 · p. 18

Prompts for User Agent and Webvoyager-based Evaluator

• The section provides prompts for the user agent and WebVoyager-based evaluator, plus persona rules for the user agent.These materials are shown in Figures 10, 12, and 11, respectively.Source: D. Prompts · p. 16, Figure 10 · p. 19, Figure 11 · p. 19

Prompt and Criteria for Aesthetic Quality Evaluation

• Aesthetic-quality evaluation uses detailed criteria that also serve as the basis for human evaluation.The criteria are illustrated in Figure 13.Source: D. Prompts · p. 17, Figure 13 · p. 21