Source-linked AI summary
InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
Qiyao Wang, Haoran Hu, Longze Chen, Hongbo Wang, Hamid Alinejad-Rokny, Yuan Lin, Min Yang
TL;DR
Existing website-generation benchmarks assume clear instructions and static execution, leaving a gap in evaluating ambiguous non-expert requirements. InteractWeb-Bench addresses this gap with persona-driven perturbations and an interactive Clarify–Implement–Verify–Submit environment; experiments show current agents remain prone to blind execution.
Problem
Existing benchmarks use idealized, well-structured inputs and static settings, whereas non-expert users provide ambiguous requirements that challenge intent recognition.
Method
InteractWeb-Bench combines four persona-driven instruction perturbations with an interactive environment supporting Clarify, Implement, Verify, and Submit actions.
Results
Current agents remain prone to blind execution, and the best-performing model achieves a Task Completion Rate of 38.78% in realistic interactive settings.
Takeaways & Limitations
Improving website-generation agents requires recognizing missing information, eliciting clarification, and updating assumptions from feedback in addition to coding capability.
Takeaways & Limitations
Evaluation trajectories are forcibly terminated when they exceed task-scaled maximum-turn or consecutive-verification-error limits.
Abstract
from arXiv · showhide
With the advancement of multimodal large language models (MLLMs) and coding agents, the website development has shifted from manual programming to agent-based project-level code synthesis. Existing benchmarks rely on idealized assumptions, especially for well-structured, information-rich inputs and static execution settings. In contrast, real-world development is constrained by a critical bottleneck: the semantic misalignment between ambiguous, low-quality instructions from non-expert users and model understanding, which results in a failure mode that we term blind execution. To address this gap, we introduce InteractWeb-Bench, the first multimodal interactive benchmark for website generation under non-expert low-code user conditions. InteractWeb-Bench introduces four types of user agents and persona-driven instruction perturbations to systematically simulate diverse user behaviors, including ambiguity, redundancy, and contradiction, grounded in requirement engineering defect taxonomies. We develop an interactive execution environment for agents, featuring a unified action space comprising Clarify, Implement, Verify, and Submit, enabling iterative intent refinement, code synthesis, and visual feedback-based validation. Extensive experiments and analysis reveal that frontier MLLM-based agents remain trapped in blind execution, exposing limitations in intent recognition and adaptive interaction.
1. Introduction
InteractWeb-Bench targets blind execution caused by ambiguous, fragmented, and conflicting non-expert instructions. It evaluates whether agents can recognize intent dynamically through clarification, implementation, verification, and submission.
- Non-expert users produce ambiguous, fragmented, redundant, or contradictory requirements that create a semantic gap between intent and model understanding.
- Existing website-generation benchmarks use standardized, well-structured instructions and static evaluation settings unlike real-world non-expert requirements.
- Blind execution occurs when agents bypass intent verification, directly synthesizing code from incomplete or conflicting instructions instead of requesting clarification.
- Frontend agents must interleave clarification and visual verification because user requirements are rarely complete at the outset.
- InteractWeb-Bench introduces persona-driven instruction perturbations and a unified action space of Clarify, Implement, Verify, and Submit for interactive website generation.
2. Related Work
Prior website-generation benchmarks emphasize static vision-to-code or standardized generation, while related interaction work studies clarification and execution feedback in other settings. InteractWeb-Bench builds on these strands for interactive website generation.
- Early website-generation benchmarks formulate the task as static vision-to-code translation from screenshots or design specifications to HTML/CSS.
- Recent clarification benchmarks evaluate agents under incomplete, ambiguous, noisy, or persona-perturbed requirements.
- Interactive-agent benchmarks evaluate multi-turn use of execution feedback such as compiler errors and test outputs.
3. InteractWeb-Bench
InteractWeb-Bench combines persona-driven requirement perturbations with an autonomous multimodal execution environment. Agents clarify, implement, verify, and submit under bounded trajectories, with constraint-slot-based evaluation.
- Persona-Driven User Agent Module: The benchmark uses four persona-driven user agents grounded in requirement-engineering defects to create targeted intent traps.
- Persona-Driven User Agent Module: Persona-specific metamorphic operators transform an unambiguous instruction I∗ into perturbed instructions Imutated.
- Persona-Driven User Agent Module: User-agent responses retrieve only details explicitly requested by the execution agent and filter them through the assigned persona.
- Interactive Agent Framework and Execution Environment: Agents autonomously select Clarify, Implement, Verify, or Submit from a discrete action space while iterating through multimodal feedback.
- Interactive Agent Framework and Execution Environment: Verification fuses interaction history, code context, and prior visual audits, then supports GUI exploration and structured failure feedback.
- Interactive Agent Framework and Execution Environment: Evaluation bounds trajectories with maximum-turn and consecutive-verification-error limits scaled to task difficulty.
- Constraint-Slot-Based Evaluation Metric and Dataset Statistics: Oracle slots encode target components, expected results, and assertion types, while an anti-hallucination slot measures unrequested interface elements separately.
- Constraint-Slot-Based Evaluation Metric and Dataset Statistics: Slot weights reflect component complexity and testing density through a step-wise technical complexity metric.
4. Experiments
Experiments show that current website-generation agents achieve limited task completion in realistic interactive settings and exhibit distinct, often ineffective interaction and verification behaviors.
- Main Results: 38.78% TCR is the best overall result, achieved by Qwen3.6-Plus, while performance decreases as task difficulty increases.
- Main Results: Models vary substantially across user personas, with higher TCR under P-RAM and significantly lower performance under P-MIN.
- Behavioral Findings: Models over-generate code instead of seeking clarification, with some producing over 1000 Avg. LoC alongside hallucination rates above 60%.Aggressive generation introduces unwanted functionality, redundant components, and misaligned designs that degrade usability and increase maintenance overhead.
- Behavioral Findings: Verification intensity and execution length vary widely, but GUI-based feedback does not produce clear task-completion improvements.Models often apply local fixes without reconsidering incomplete or incorrect underlying requirements.
- Behavioral Findings: Models trade off exploration and decisiveness: GPT-4.1-mini asks more questions with lower submission rates, whereas Qwen3.6-Plus and Kimi-K2.5 submit more quickly with larger outputs.The more decisive pattern carries a risk of hallucinated or misaligned components.
- Aesthetic Evaluation: Aesthetic quality shows a ceiling: websites are generally structurally complete, yet slight visual defects persist and layout and creative-alignment scores vary little across models.Kimi-K2.5 achieves the highest aesthetic quality, while GPT-4.1-mini is relatively weaker.
5. Conclusion
InteractWeb-Bench evaluates MLLM-based agents on ambiguous, underspecified instructions in realistic website-generation scenarios. Experiments reveal blind execution and motivate stronger intent recognition, clarification, and feedback-based assumption updates.
- InteractWeb-Bench evaluates whether MLLM-based agents can handle ambiguous and underspecified instructions in realistic website-generation scenarios.
- Experiments observe a blind-execution tendency: generated websites often appear superficially plausible while deviating from users’ actual needs.
- Future progress requires recognizing missing information, actively eliciting clarifications, and updating assumptions based on feedback, beyond stronger coding capabilities.
- The benchmark is envisioned as a foundation for more interactive and user-aligned website generation by future MLLM-based coding agents.
A. More Details about Evaluated Models
The evaluation reports model sizes, context and output limits, access methods, hardware settings, and API costs to support consistent comparison across multimodal systems.
- Evaluated models are documented by size, maximum context length, maximum output length, and access method.
- All models support image inputs, with a maximum of five images per input.
- Closed-source and large open-source models are evaluated through official APIs for fair and consistent comparison.
- The study reports total API costs for models accessed through APIs.
- Smaller open-source models are evaluated on eight NVIDIA A800 GPUs with identical hyperparameters.
B. More Details about Human Evaluation
Human experts evaluate aesthetic quality using the same criteria as the MLLM judge prompts, with average scores reported across models.
- Aesthetic quality is evaluated by three PhD students specializing in Computer Science.
- Human evaluation uses the same criteria as the MLLM judge prompts in Appendix D.
- The results report detailed average human evaluation scores across models in Table 6.
- Tables 4 and 5 provide the evaluated-model overview and API-based model costs, respectively.
C. Detailed Case Study
The case studies compare golden and persona-conditioned instructions across models, showing distinct behavioral deviations, repair failures, premature submission, and model-specific visual preferences.
- Case-study setup: Table 7 and Figure 6 compare website cases generated from golden instructions and four persona-conditioned variants.
- Case-study setup: Persona-conditioned instructions produce distinct behavioral deviations and final webpage outcomes.
- Cross-model comparison: Models show different visual preferences: Kimi-K2.5 favors harmonious palettes, Qwen3.6-Plus detailed assets, GPT-4.1 textured backgrounds, and Gemma-4-31B-it structured image layouts.
- Golden instruction: The golden instruction specifies a children’s recycling game with competition, cartoon visuals, sorting feedback, rewards, and recognition.
- Persona case studies: A post-verification repair loop repeatedly addressed a superficial title-emoji issue while missing broken score and feedback updates in drag-and-drop gameplay.
- Persona case studies: P-MIN implemented basic gameplay but omitted the leaderboard, feedback, rewards, introduction, and specified color scheme before premature submission.
- Persona case studies: Another case repeatedly cycled between implementation and verification without enabling item removal, score updates, or success feedback.
- Persona case studies: A P-INT case downgraded to a static webpage, ignored verification errors, and failed to translate abstract sensory instructions into concrete UI properties.
D. Prompts
The appendix documents prompts governing website-generation agents, user personas, evaluators, and aesthetic-quality assessment. Together, these prompts specify agent actions, persona behavior, web interaction, and evaluation criteria.
- Website Generation Agent: The website-generation agent prompt defines its action space and output-format constraints.A GUI-based verification prompt is provided when the agent invokes Verify.
- User Agent: User-agent prompts and persona rules specify how simulated users provide instructions and respond to clarification.The Minimalist persona supplies only the minimum requested information and gives exact values in the shortest form when asked.
- Aesthetic Quality Evaluation: Aesthetic-quality evaluation uses detailed criteria that also govern human evaluation.The appendix provides the associated prompt and criteria.
- WebVoyager-based Evaluator: The WebVoyager-based evaluator instructs a robot-like browser agent to act from webpage screenshots and labeled web elements.Available actions include clicking, replacing textbox content, scrolling, waiting, going back, and answering once the task is solved.