Source-linked AI summary
IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?
Yang Chen, Minghao Liu, Yufan Shen, Yunwen Li, Tianyuan Huang, Xinyu Fang, Tianyu Zheng, Wenxuan Huang, Cheng Yang, Daocheng Fu, Jianbiao Mei, Rong Wu, Yunfei Zhao, Licheng Wen, Xuemeng Yang, Song Mao, Qunshu Lin, Zhi Yu, Yongliang Shen, Yu Qiao, Botian Shi
TL;DR
Existing webpage-to-code benchmarks emphasize static reconstruction or simplified interactions, leaving stateful real-world web workflows underrepresented. IWR-Bench evaluates reconstruction from interaction videos and complete static assets with automated functional and visual assessment, and experiments show a substantial gap between visual fidelity and functional correctness across 28 LVLMs.
Problem
Existing benchmarks focus on static reconstruction or simplified interactions and lack the complete interaction and asset context needed for real-world web applications.
Source: §1 · p. 2Method
IWR-Bench provides interaction videos and complete crawled static assets, then uses an agent-as-judge protocol with IFS and VFS to evaluate generated webpages.
Source: §1 · pp. 3–4, §4.2 · p. 7Results
Across 28 LVLMs, visual fidelity substantially exceeds functional correctness; GPT-5 reaches 64.25% VFS, 24.39% IFS, and a 36.35% Final Score.
Source: §1 · pp. 3–4, §5.2 · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
The webpage-to-code task requires models to understand visual representations of webpages and generate corresponding code. However, existing benchmarks primarily focus on static screenshot-to-code tasks, thereby overlooking the dynamic interactions fundamental to real-world web applications. To address this limitation, this paper introduces IWR-Bench, a novel benchmark for evaluating the capabilities of Large Vision-Language Models (LVLMs) in interactive webpage reconstruction from video. IWR-Bench comprises 113 meticulously curated tasks from 100 real-world websites, with 1,001 actions and featuring diverse interaction complexities (e.g., web games), visual styles, and domains. Aligning with standard web development practices, each task includes not only user interaction videos but also all crawled static assets (e.g., images, videos). This benchmark evaluates models on two fundamental challenges: comprehensive multi-modal reasoning to infer interaction logic from video and assets, and advanced code generation to translate this logic into functional code. An agent-as-a-judge framework with a comprehensive metric system automatically assesses the functional correctness and visual fidelity of generated webpages. Extensive experiments on 28 LVLMs reveal a significant challenge: the best model achieves an overall score of only 36.35%, as functional correctness (24.39% IFS) lags significantly behind visual fidelity (64.25% VFS). These results highlight critical limitations in current models' ability to reason about temporal dynamics and synthesize event-driven logic, establishing IWR-Bench as a challenging frontier for vision-language research. The benchmark and evaluation code will be made publicly available at https://github.com/SIGMME/IWR-Bench.
1 INTRODUCTION
IWR-Bench addresses the gap between static webpage reconstruction and real-world interactive applications by pairing stateful interaction videos with complete static assets and evaluating both functionality and visual fidelity. Across 28 LVLMs, results show that visual replication substantially outpaces event-driven logic synthesis.
1 INTRODUCTION
- The task requires multimodal reasoning to infer latent interaction logic and match dynamic elements to static assets, followed by code generation for complex stateful behavior.Examples include interactive web games such as 2048 and Minesweeper.Source: §1 · p. 2
- Each task supplies a complete stateful interaction video and all crawled static assets, creating a realistic reconstruction context.The setup addresses prior benchmarks’ limited interaction settings and incomplete resource coverage.Source: §1 · pp. 2–3
- An agent-as-judge executes ground-truth action sequences to assess functionality and uses screenshot comparison to evaluate visual fidelity through IFS and VFS.IFS measures operational and logical correctness, while VFS measures rendered visual quality.Source: §1 · pp. 3–4
- GPT-5 achieves a Final Score of 36.35%, while its functional correctness is 24.39% IFS versus 64.25% VFS.The results expose a substantial gap between reproducing appearance and implementing interactive behavior.Source: §1 · pp. 3–4
- IWR-Bench formalizes interactive webpage reconstruction from video with 113 curated real-world tasks spanning domain, visual complexity, and interaction logic.The benchmark provides a structured testbed for reconstructing stateful web applications rather than static layouts alone.Source: §1 · pp. 3–4, Figure 3 · p. 3
2 RELATED WORK
Related work establishes strong LVLM capabilities for multimodal webpage understanding and static visual-to-code generation, while benchmark development has progressed from single-image tasks toward multi-image and video reasoning. IWR-Bench extends this trajectory to interactive webpage generation.
2 RELATED WORK
- LVLMs support unified webpage understanding across visual and textual content, including element grounding, screen question answering, and visual webpage-to-code generation.These capabilities motivate extending webpage reconstruction beyond static layouts.Source: §2 · p. 4
- LVLM benchmarks have advanced from single-image comprehension to multi-image reasoning and video understanding as model capabilities expanded.This progression provides the broader evaluation context for interactive reconstruction from video.Source: §2 · p. 4
3 IWR-BENCH
IWR-Bench is a video-based benchmark for reconstructing interactive webpages from complete user workflows and crawled assets. It standardizes diverse real-world tasks through curated annotations, taxonomy axes, and comparisons with existing benchmarks.
Benchmark task
- IWR-Bench reconstructs webpages from interaction videos, static assets, and structured action trajectories rather than static screenshots alone.Each task supplies temporal interaction evidence and the assets needed to reproduce visual appearance and behavior.Source: §3.1 · pp. 4–5
Benchmark construction
- The benchmark is curated from real websites through expert sourcing, deduplication, diversity balancing, recording, asset collection, annotation, cross-verification, and filtering.The process reduces 200 candidate tasks to 161 annotation candidates and finally 113 verified tasks.Source: §3.2 · pp. 5–6
Taxonomy and statistics
- Tasks span domain coverage, visual complexity, and interaction logic, ranging from static displays to complex workflows and algorithmic games.The taxonomy covers 5 major and 16 subcategories, with visual complexity from minimalist layouts to data-dense dashboards.Source: §3.3 · p. 6
- The final benchmark contains 113 videos, 1,001 actions, 620 visual-evaluation checkpoints, and 403 assertion checks, averaging 8.9 actions per video.It also includes mobile-interface tasks, reflecting varied web usage patterns.Source: §3.3 · p. 6, §3.4 · p. 6, Table 1 · p. 6
Comparison to other benchmarks
- Compared with prior benchmarks, IWR-Bench combines live-web sourcing, stateful video trajectories, comprehensive interaction evaluation, and complete static assets.It targets code generation from interactive workflows, unlike static reconstruction, stateless single-step interaction, or general video-understanding benchmarks.Source: §3.4 · p. 6, Table 2 · p. 6
4 EVALUATION AND METRICS
IWR-Bench evaluates generated webpages by deterministically replaying ground-truth actions and measuring both successful interaction execution and visual fidelity at reachable checkpoints. Its final score combines these metrics with greater weight on functionality.
Evaluation protocol
- A deterministic browser executor replays each ground-truth action sequence, isolating evaluation from high-level task planning for stable, reproducible assessment.Actions fail when they are operationally infeasible or when their associated logical assertions are unsatisfied.Source: §4.1 · pp. 6–7
Metrics
- Interactive Functionality Score (IFS) is the ratio of successfully completed actions to total actions, requiring error-free execution and satisfied logical assertions.It measures whether generated code implements the required interactive behavior.Source: §4.2 · p. 7
- Visual Fidelity Score (VFS) macro-averages weighted low-level and high-level visual scores over successfully reached checkpoints with visual evaluation enabled.Low-level scoring combines OCR-based Levenshtein similarity and DINO cosine similarity, while high-level scoring comes from an MLLM judge; the weight is 0.5.Source: §4.2 · p. 7, §4.1 · p. 7
- The Final Score combines IFS and VFS with α = 0.7, giving functionality substantial weight while reflecting unreachable states through the functionality component.An alternative success-ratio weighting scheme was explored but was ineffective for differentiating model performance.Source: §4.2 · p. 7
5 EXPERIMENTS
Experiments on 28 LVLMs show that IWR-Bench remains difficult: models reproduce visual appearance more successfully than interactive functionality, with reasoning enhancements yielding only moderate gains.
5.1 EVALUATION SETUP
- The evaluation uses 28 LVLMs with video and crawled assets as inputs, and reports results in a category-grouped table sorted by Final Score.The protocol includes metric-parameter alignment and validation of its agent-as-a-judge methodology.Source: §5.1 · pp. 7–8, Table 3 · p. 8, §5.4 · p. 9
5.2 MAIN RESULTS
- 36.35% Final Score is the highest result, achieved by GPT-5, while proprietary models generally outperform open-source and video-specialized models.Video-specialized models reach only 13.67 and 10.07, whereas GPT-4o (latest) scores 29.55.Source: §5.2 · pp. 8–9
- 64.25 VFS versus 24.39 IFS for GPT-5 exposes a large gap between visual fidelity and interactive functionality.The highest IFS remains below 25, indicating that event-driven webpage reconstruction is still largely unsolved.Source: §5.2 · p. 9
- Reasoning-enhanced variants provide consistent but moderate gains, such as Claude-Sonnet-4 thinking improving Final Score from 34.00 to 34.62 and IFS from 22.29 to 23.65.Similar improvements appear for Claude-Opus-4 and Gemini-2.5-Pro, but base-model capability remains the main performance ceiling.Source: §5.2 · p. 9
5.3 PERFORMANCE ANALYSIS ACROSS TASK DIMENSIONS
- Performance drops sharply from static L1 to interactive L2-L4 tasks, while highly structured layouts remain difficult and Entertainment & Media is comparatively stronger.The analysis identifies structured code generation and state management as key research directions.Source: §5.3 · p. 9
6 CONCLUSION
IWR-Bench introduces video-based evaluation of interactive webpage reconstruction using automated functional and visual metrics. Results show that models can reproduce appearance more successfully than event-driven behavior, motivating research on temporal reasoning, asset binding, and code synthesis.
6 CONCLUSION
- IWR-Bench is the first benchmark for interactive webpage reconstruction from video, evaluated with an automated agent-as-judge using IFS and VFS.The benchmark targets both visual replication and functional implementation of webpages.Source: §6 · p. 9
- Models achieve moderate visual fidelity but critically limited functional correctness, making complex interaction logic the primary bottleneck.The conclusion highlights temporal reasoning, dynamic asset binding, and robust code synthesis as future directions.Source: §6 · p. 9
A MULTI-DIMENSIONAL TASK TAXONOMY
IWR-Bench uses a three-dimensional taxonomy to analyze task difficulty across interaction complexity, visual complexity, and application domain. The axes support fine-grained diagnosis of model capabilities and failure modes.
A MULTI-DIMENSIONAL TASK TAXONOMY
- The taxonomy classifies tasks along orthogonal axes of Interaction Complexity, Visual Complexity, and Application Domain.This structure is intended to diagnose specific challenges and model failure modes rather than treat difficulty as monolithic.Source: A MULTI-DIMENSIONAL TASK TAXONOMY · p. 16
Interaction Complexity
- Interaction Complexity spans L1-L4 according to the depth of logical and temporal understanding required for reconstruction.L1 covers static content consumption and scrolling, while L2 covers local state manipulation such as filtering, tabs, and accordions.Source: A MULTI-DIMENSIONAL TASK TAXONOMY · p. 16
Visual Complexity
- Visual Complexity spans V1-V4, from minimalist layouts and standard grids to asymmetric modern designs and data-dense layouts.The axis focuses on the static challenge of rendering webpage appearance, including layout and styling.Source: A MULTI-DIMENSIONAL TASK TAXONOMY · p. 16
Application Domain
- Application Domain groups real-world web applications into five high-level domains, including Commerce & Services, Knowledge & Education, and Productivity & Tools.The taxonomy is designed to reflect the breadth of web applications.Source: A MULTI-DIMENSIONAL TASK TAXONOMY · p. 16
B DETAILED EXPERIMENTAL RESULTS
The detailed analysis reports Final Score across application domains, interaction logic complexity, and visual complexity. The supplied table description specifically identifies a breakdown by interaction logic complexity.
B DETAILED EXPERIMENTAL RESULTS
- Final Score is reported for each category across Application Domain, Interaction Logic Complexity, and Visual Complexity.These correspond to Tables 6, 4, and 5, respectively.Source: B DETAILED EXPERIMENTAL RESULTS · p. 16
Interaction Logic Complexity
- Table 4 breaks down Final Score by Interaction Logic Complexity.Source: Table 4 · p. 17
C METRIC PARAMETER VALIDATION
IWR-Bench standardizes evaluation through semantic action sequences and model-independent prompts, while validating visual-score parameters against human judgments.
Metric Parameter Validation
- Visual similarity and final-quality weighting coefficients are selected through a human-alignment study using 60 instances rated by five PhD-level students.The parameters are determined through grid search over candidate weights.Source: C METRIC PARAMETER VALIDATION · p. 17
Task and Action Representation
- Action sequences use natural-language element descriptions instead of coordinates, making evaluation robust to layout variation and testing semantic page understanding.The vocabulary includes Click, Type, Scroll, and Press actions.Source: D TASK AND ACTION REPRESENTATION · p. 17
Evaluation Protocol
- A standardized system prompt fixes task requirements, output format, and operational constraints across models and tasks.The design minimizes ambiguity and isolates code-generation capability.Source: E PROMPTS · p. 18
- Separate multimodal prompts evaluate webpage similarity and verify whether action assertions are correct.The similarity prompt combines quantitative and qualitative assessment, while the assertion prompt evaluates action correctness.Source: E PROMPTS · p. 18
F CASE STUDY
The case studies probe interactive reconstruction across e-commerce workflows, browser-game logic, long-page visual fidelity, and time-based mobile layouts. Results show that models often reproduce broad structure or interactions while failing fine-grained assets, state transitions, or algorithmic behavior.
Case Study Setup
- The benchmark presents each case through an operation video, static webpage resources, generated webpages, interaction results, and detailed analyses of representative challenges.Its case studies span interaction complexity, visual complexity, application domains, and prompt-based evaluation procedures.Source: F CASE STUDY · pp. 18, 20
E-commerce Workflow Simulation
- The e-commerce workflow exposes distinct failures: Claude-Sonnet-4 handles filtering and sorting but fails to add an item to the cart, while GPT-5 is affected by many static resources.Source: F CASE STUDY · p. 19
Algorithmic Logic Reconstruction
- Grok-4 reproduces the 2048 game logic, whereas Qwen2.5-VL-72B fails to merge blocks after an upward input, underscoring the challenge of algorithmic reasoning.The task requires reverse-engineering complete browser-game rules from observed behavior.Source: F CASE STUDY · pp. 19–20
Long-Context Fidelity and Fine-Grained Visual Detail
- Models must combine holistic page-structure understanding with precise matching of icons and product images to reconstruct long webpages successfully.Gemini-2.5-pro and GPT-5 restore relatively complete long pages but miss corresponding icons and matching product images.Source: F CASE STUDY · p. 21
Time-Based State Management in a Mobile Viewport
- GLM-4.5V implements the timed interactive operations in the Pomodoro task, while Kimi-VL-thinking cannot continue because required first-step elements are absent initially.The case combines time-driven state changes with a narrow mobile viewport requiring responsive or appropriately scaled layout.Source: F CASE STUDY · p. 21
Related papers
This paper cites
- Learning Transferable Visual Models From Natural Language Supervision2021
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?2023
- VizWiz Grand Challenge: Answering Visual Questions from Blind People2018
- Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding2022
Cited by
- CodeCap-Bench: Can Code Serve as a Good Video Caption?2026
- Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models2026
- Framework and Benchmark for Code-Driven Agentic Testing in Web Development2026
- UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation2026
- Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence2026