Source-linked AI summary
MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation
Yan Li, Zezi Zeng, Yifan Yang, Yuqing Yang, Ning Liao, Weiwei Guo, Lili Qiu, Mingxi Cheng, Qi Dai, Zhendong Wang, Zhengyuan Yang, Xue Yang, Ji Li, Lijuan Wang, Chong Luo
TL;DR
Existing webpage-generation pipelines often produce inconsistent or globally incoherent multimodal pages because assets are generated independently. MM-WebAgent coordinates native multimodal generation through hierarchical planning and iterative self-reflection, and experiments show stronger performance than code-generation and agent-based baselines, especially for multimodal elements and integration.
Problem
Existing webpage-generation benchmarks and pipelines inadequately assess or coordinate native multimodal assets with global page semantics, style, and layout.
Method
MM-WebAgent combines global and local hierarchical planning with three-level self-reflection to coordinate multimodal content, geometry, and webpage integration.
Results
MM-WebAgent outperforms code-generation and code-only agent baselines, with particularly strong gains in multimodal element generation and integration.
Takeaways & Limitations
The framework provides a structured plan-and-refine paradigm and a benchmark with multi-level evaluation for coherent multimodal webpage generation.
Takeaways & Limitations
The approach depends on external AIGC tools and fixed invocation patterns, and its training-free orchestration does not optimize agent behavior through learning.
Abstract
from arXiv · showhide
The rapid progress of Artificial Intelligence Generated Content (AIGC) tools enables images, videos, and visualizations to be created on demand for webpage design, offering a flexible and increasingly adopted paradigm for modern UI/UX. However, directly integrating such tools into automated webpage generation often leads to style inconsistency and poor global coherence, as elements are generated in isolation. We propose MM-WebAgent, a hierarchical agentic framework for multimodal webpage generation that coordinates AIGC-based element generation through hierarchical planning and iterative self-reflection. MM-WebAgent jointly optimizes global layout, local multimodal content, and their integration, producing coherent and visually consistent webpages. We further introduce a benchmark for multimodal webpage generation and a multi-level evaluation protocol for systematic assessment. Experiments demonstrate that MM-WebAgent outperforms code-generation and agent-based baselines, especially on multimodal element generation and integration. Code & Data: https://aka.ms/mm-webagent.
1 Introduction
MM-WebAgent addresses the difficulty of generating webpages whose multimodal assets remain coherent with global layout, style, and semantics. It combines hierarchical planning, multimodal generation, iterative reflection, and systematic evaluation.
- Existing pipelines often retrieve or independently insert assets, causing style inconsistency, geometry mismatch, and global incoherence.
- MM-WebAgent uses global layout plans and context-conditioned local element plans to coordinate multimodal asset generation.Global plans specify hierarchy, ordering, spatial organization, and page-level style; local plans encode roles, constraints, and style guidance.
- Hierarchical self-reflection refines assets locally, patches HTML/CSS contextually, and revises the full page globally using code and rendered screenshots.These levels jointly optimize content, geometry, and aesthetics rather than treating multimodal elements as loosely coupled add-ons.
- MM-WebGEN-Bench and its multi-level protocol assess global webpage quality alongside local image, video, and chart quality.The protocol covers layout correctness, style coherence, aesthetics, and embedded multimodal elements.
- Experiments show MM-WebAgent outperforms code-generation and code-only agent baselines, especially on multimodal element generation and integration.
- The paper contributes a multimodal web-agent paradigm, hierarchical self-reflection, and a benchmark with systematic evaluation.
2 Related Work
Prior webpage-generation systems improve code or layout reconstruction but generally rely on static or externally supplied visual assets. MM-WebAgent instead targets native multimodal asset creation aligned with page semantics and design.
- Visual code-generation studies reconstruct webpages from screenshots or augment generation with externally retrieved visual assets.
- These approaches improve layout fidelity or code correctness but limit novel, semantically aligned, and stylistically coherent multimodal content.
- Existing code agents use planning, tools, and environmental interaction to orchestrate complex design or software workflows.
- Their hierarchies remain focused on reasoning or code granularity, whereas MM-WebAgent defines hierarchy at the design-abstraction level for cross-modal refinement.
- Existing benchmarks emphasize HTML/CSS correctness, screenshot reconstruction, or placement of static assets rather than native asset generation aligned with global semantics.
3 Method
MM-WebAgent models webpage generation as a hierarchical structured process: it plans globally, generates elements individually, and iteratively refines the result through hierarchical self-reflection.
- The framework organizes webpage generation into hierarchical planning, element-wise generation, hierarchical self-reflection, and multi-level evaluation.
Multi-level Evaluation
MM-WebAgent’s evaluation combines a diverse benchmark with global and local quality criteria, using structured scoring to measure webpage composition and embedded multimodal elements.
- Framework Overview: The framework overview comprises task planning, hierarchical generation, multi-level evaluation, and iterative reflection.
- Evaluation Dataset: MM-WebGEN-Bench is constructed through controlled scenario generation, automatic validation, rendering, and manual quality inspection.
- Evaluation Dataset: The final evaluation set contains 120 webpages spanning 11 scene categories, 11 visual styles, 4 video types, 8 image types, and 17 chart types.
- Multi-level Evaluation: The multi-level scheme decomposes webpage assessment into global page quality and local embedded multimodal-component quality.
- Multi-level Evaluation: Global evaluation measures layout correctness, style coherence, and aesthetics, while local evaluation examines images, videos, and charts in context.
- Scoring: Penalty-based scoring identifies rule violations, weights them by severity, and computes a bounded score using the aggregate penalty.
- Scoring: Graded dimensions use scores from 0 to 1 in increments of 0.2, with benchmark and model scores obtained by averaging across samples and dimensions.
4 Experiments
MM-WebAgent is evaluated against code-generation and agent-based baselines on multimodal and conventional webpage benchmarks, with ablations, cost analysis, user preferences, and qualitative comparisons. Results show strong global and local webpage quality, while hierarchical planning, reflection, and native AIGC tools support multimodal integration.
- Main Results: MM-WebAgent achieves the best global and local performance on MM-WebGEN-Bench, with an average score of 0.75.The six dimensions cover layout, style, aesthetics, image, video, and chart quality and integration.
- Main Results: MM-WebAgent remains highly competitive on WebGen-Bench despite not being explicitly designed for backend code generation.This benchmark primarily tests functional backend code, logic, and component completeness, while its prompts lack specific visual instructions.
- Ablation Studies: Disabling local planning reduces overall performance from Avg: 0.75 to 0.69, with pronounced degradation on Image and Video metrics.The ablation attributes the decline to removing context-aware local planning from the full system.
- Computational Cost: MM-WebAgent’s average execution time is 155.8s, compared with 182.4s for Openhands, although its monetary cost is higher than code-only agents.Image, video, and chart generation are executed in parallel in the implementation.
- User Study: MM-WebAgent achieves a winning rate of 78.99% in a pairwise study with 50 annotators evaluating layout, relevance, asset quality, and embedding quality.The annotators had backgrounds in web design, computer science, or multimodal content creation.
- Qualitative Results: Qualitative comparisons show more coherent layouts, consistent visual styles, and better-aligned multimodal content than representative baselines.The method more reliably integrates images and charts into the overall page structure and semantic requirements.
- Qualitative Results: Hierarchical reflection iteratively refines global layout, context, and local elements, while local reflection can propagate corrections to the global page structure.The visualization includes global layout refinement, context refinement, local element refinement, and local-to-global correction.
5 Conclusion
MM-WebAgent combines structured planning, hierarchical multimodal generation, and iterative self-reflection to produce coherent webpages. MM-WebGEN-Bench evaluates diverse layouts, styles, and multimodal compositions, and experiments show improvements over code-generation and agent-based baselines.
- MM-WebAgent integrates structured planning, hierarchical generation, and iterative self-reflection for multimodal webpage generation.
- The planning stage organizes global layouts and local multimodal elements, while hierarchical reflection adjusts elements and layouts to improve consistency and visual quality.
- MM-WebGEN-Bench covers diverse webpage layouts, visual styles, and multimodal compositions for evaluating multimodal webpage generation.
- Experiments show that MM-WebAgent outperforms code-generation-based and agent-based baselines in generating well-integrated multimodal webpages.
6 Limitation and Future Work
MM-WebAgent depends on external AIGC tools and fixed invocation patterns, while using a training-free orchestration-based formulation. Future work could optimize planning, tool use, and reflection through learning-based methods.
- External AIGC tools make webpage quality susceptible to instability, bias, safety filters, and changes in tool availability.
- The framework assumes a fixed set of tools and invocation patterns, restricting flexibility in dynamic tool selection and composition.
- MM-WebAgent uses a training-free orchestration-based formulation rather than learning-based optimization of agent behaviors.
- Reinforcement learning or other learning paradigms are proposed as future directions for optimizing planning, tool usage, and reflection over long-term interactions.
A More Qualitative Results
The paper provides additional qualitative examples of webpages generated by MM-WebAgent. These examples supplement the benchmark’s qualitative assessment of multimodal webpage generation.
- Additional generated webpage examples are presented in Fig. 6.
B Prompt Templates
The prompt templates specify structured planning and tool-call formats for generating webpages and their image, video, and chart assets. They encode contextual, stylistic, geometric, and data-specific requirements for multimodal integration.
- B.1 Planner Prompt: The planner prompt asks a webpage-generation planning agent to decompose a user request into a structured plan for tool calls.
- B.1 Planner Prompt: HTML layout planning requires descriptions of webpage sections and visual elements, including references for images, videos, and charts.
- B.1 Planner Prompt: Image extraction records each asset’s path, section, functional role, webpage style, visual style, color tone, composition, and lighting.
- B.1 Planner Prompt: Video extraction specifies context, role, visual style, motion intensity, camera behavior, loopability, duration, and size for dynamic webpage elements.
- B.1 Planner Prompt: Data-visualization extraction requires chart references, contextual roles, chart types and styles, color palettes, visual emphasis, prompts, and complete source data.
- B.1 Planner Prompt: The output format requires valid JSON containing separate structured entries for image, video, and chart tool calls.
- B.1 Planner Prompt: Image and video templates express each asset’s webpage context and role, while attribute templates specify visual or motion properties.
- B.1 Planner Prompt: The layout agent must output directly usable HTML, preserve referenced image paths, and specify file paths and layout dimensions for visual assets.
C User Study
The user study evaluation uses representative questions about multimodal asset quality, webpage layout, and chart quality, alongside rendered webpage examples comparing MM-WebAgent with baseline methods.
- Figure 7 presents representative questions used in the user study evaluation.
- Figure 6 shows rendered webpage examples generated by MM-WebAgent and baseline methods.
- The survey questions assess coherence and attractiveness of multimodal assets, layout aesthetics and elegance, and chart accuracy and readability.