Source-linked AI summary
Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development
Yilong Guo, Hanqi Chen, Zixiao Ye, Guanzhong Wang, Chen Yu, Zeyu Chen
TL;DR
Webpage generation must address both visual fidelity and interaction correctness, yet runtime interaction failures can remain invisible to static inspection and screenshot comparison. RILA closes this gap by replaying reference interactions in a browser and using execution-aware scoring and critique to refine code; it also synthesizes execution-verified training data. With training and refinement, fine-tuned Qwen3.5-9B† reaches 57.52% Final score and surpasses the cited one-shot Kimi-K2.6 and GPT-5.5 results.
Problem
Runtime interaction failures can be invisible to static inspection or screenshot comparison, while regenerating a whole page after a failure can overwrite correct code and cause regressions.
Source: Overview · p. 3Method
RILA replays reference actions in a browser, scores interaction correctness and visual fidelity, and uses execution feedback to refine code; it also constructs execution-verified training data.
Source: Execution-Aware Rendering Score · p. 4, Conclusion · p. 7Results
57.52% Final score for fine-tuned Qwen3.5-9B† with Kimi-K2.6 critique surpasses one-shot Kimi-K2.6 (55.61%) and GPT-5.5 (55.74%).
Source: Main Results · p. 6Takeaways & Limitations
Execution-verified training and inference-time refinement address different aspects of webpage generation and combine effectively.
Source: Main Results · p. 6, Conclusion · p. 7
Abstract
from arXiv · showhide
Multimodal large language models have achieved remarkable progress in front-end web development, generating interactive webpages from multimodal references such as screenshots and interaction videos. However, existing work largely emphasizes visual metrics such as aesthetics and layout similarity, while overlooking the more critical validation of interactive functionality. We present RILA, an execution-driven agent that puts browser rendering in the loop, iteratively editing generated code from runtime interaction feedback. RILA introduces an Action Interaction Verification (AIV) module that replays the reference interaction trajectory on the generated webpage to collect grounded execution-aware observations, and an Execution-aware Rendering Score (ERS) that jointly measures interaction correctness and visual fidelity to guide iterative optimization. We further build an execution-verified data synthesis pipeline that produces diverse, high-quality training data, offering gains complementary to inference-time optimization. On IWR-Bench, RILA consistently improves both interaction and visual fidelity across foundation models. Notably, with our training pipeline, RILA lifts the compact Qwen3.5-9B backbone from 40.40% to 57.52%, surpassing far larger one-shot generators, including the 1T-parameter Kimi-K2.6 (55.61%) and the proprietary GPT-5.5 (55.74%).
Software Engineering Agents
RILA brings browser execution into webpage generation, unlike software engineering agents that repair codebases and web agents that operate sites without authoring their code. It uses runtime diagnosis and localized editing rather than repeatedly regenerating the whole page.
Software Engineering Agents
- RILA brings reference-interaction replay and browser feedback into code generation, unlike web agents that operate live sites without authoring their code.Software engineering agents resolve programming tasks through development environments, while RILA puts execution inside the generation loop.Source: Software Engineering Agents · p. 2
- RILA iterates from an initial code-model implementation, using browser observations and critique to refine the webpage while ERS retains the best implementation.The overview depicts the execution–critique–edit loop as progressively improving rendering quality and interaction correctness.Source: Figure 2 · p. 3
- Runtime execution exposes failures that static inspection or screenshot comparison miss, while full-page regeneration can overwrite correct code and cause regressions.RILA instead diagnoses observed failures and applies localized edits.Source: Overview · p. 3
- A task combines a text instruction, reference interaction video, and resource images; its reference demonstration supplies screenshots and the interaction trajectory for execution and evaluation.The code model first generates an initial frontend implementation from the task specification.Source: Overview · p. 3
- RILA couples browser-based Action Interaction Verification, Execution-Aware Rendering Score, execution-aware critique, and tool-assisted incremental code editing.The same execution-based verification also filters samples in the data-synthesis pipeline.Source: Overview · p. 3
Action Interaction Verification
Action Interaction Verification (AIV) runs the generated project in a real browser and replays the reference interaction trajectory. Its observations include verified action outcomes and rendered states captured during replay.
Action Interaction Verification
- AIV uses Playwright to run the generated project in a real browser rather than relying on static screenshot comparison.It grounds each reference action to a concrete target using the rendered structure and interactive elements.Source: Action Interaction Verification · p. 3
- AIV sequentially replays reference actions such as clicking, typing, or scrolling, interpreting page semantics as a real user would.Each action is grounded to a target in the rendered page.Source: Action Interaction Verification · pp. 3–4
- Playwright records each action's status and captures the page after successful interactions; after a failure, later actions are skipped and observations combine screenshots with execution logs.The reference trajectory is only AIV's replay script and is not exposed to the code-generation model.Source: Action Interaction Verification · p. 4
Execution-Aware Rendering Score
Execution-Aware Rendering Score (ERS) evaluates interaction correctness and visual fidelity from browser-execution observations. RILA uses the score to select the historically best implementation for the next editing round.
Execution-Aware Rendering Score
- ERS is computed from AIV's per-action verification results and replay screenshots, so it evaluates runtime behavior and rendering together.The score determines which implementation is selected for the next editing round.Source: Execution-Aware Rendering Score · p. 4
- Sint measures interaction correctness as Nsuccess/N, the proportion of expected interaction steps that AIV verifies as successful.N is the total expected steps and Nsuccess is the number verified successful.Source: Execution-Aware Rendering Score · p. 4
- Svis aggregates complementary visual metrics covering pixel-level similarity, textual consistency, and semantic structure similarity.Its measures include SSIM, OCR text similarity, and cosine similarity between image embeddings from a lightweight vision encoder.Source: Execution-Aware Rendering Score · p. 4
- RILA carries forward the historically highest-ERS implementation, rather than the latest candidate, to avoid degradation from unsuccessful edits.The structured critique, not ERS, drives the editing itself.Source: Execution-Aware Rendering Score · p. 4
Execution-Aware Critique
The execution-aware critique model compares the current best implementation's observations with the reference demonstration. It converts interaction and rendering discrepancies into structured, actionable fixes.
Execution-Aware Critique
- The critique model compares execution observations with the reference to assess action behavior and visual or design qualities, including layout, alignment, style, responsiveness, and accessibility.Its interaction review checks whether each action triggers the expected behavior.Source: Execution-Aware Critique · p. 4
- Rather than producing a full rewrite, the critique summarizes results as fix instructions containing the issue, supporting execution evidence, and actionable suggestions.This turns low-level browser observations into high-level editing objectives.Source: Execution-Aware Critique · p. 4
Tool-Assisted Code Editing
RILA uses a tool-assisted software engineering module to incrementally edit webpages from structured critiques, then re-executes each revision in a closed loop.
Tool-Assisted Code Editing
- Each round produces a candidate implementation by applying the structured critique to the current implementation with the SWE module.The candidate is defined as Ct = S(C∗, ∆t).Source: Tool-Assisted Code Editing · p. 4
- The SWE module incrementally edits webpages using semantic code search, file editing, shell execution, and runtime verification.Across tool-interaction rounds, it localizes faulty implementations, generates patches, and validates the edited webpage.Source: Tool-Assisted Code Editing · p. 4
- The AIV module re-executes each updated implementation, repeating the closed-loop optimization for K rounds.Source: Tool-Assisted Code Editing · p. 4
Execution-Verified Data Synthesis
The execution-verified synthesis pipeline builds interactive webpages from compatible combinations of domains, interaction capabilities, and visual styles, then retains pages that pass browser execution.
Execution-Verified Data Synthesis
- The pipeline samples compatible domain, interaction-capability, and visual-style combinations, expands them into recipes, fills image placeholders with local assets, and filters pages through browser execution.Its four stages are seed sampling, recipe generation, asset resolution, and execution verification.Source: Execution-Verified Data Synthesis · p. 4, Figure 3 · p. 5
- The taxonomy spans 32 content domains, 110 interaction capabilities, and 308 visual styles, with a compatibility matrix excluding implausible combinations.The taxonomy draws on web-interaction datasets, screenshots, and design systems.Source: Execution-Verified Data Synthesis · p. 4
- Execution verification replays each page’s full interaction sequence and retains only pages on which every intended action succeeds, rejecting roughly one-third of seeds.Resolved assets are cached locally and rewritten to relative paths so pages are self-contained and render deterministically offline.Source: Execution-Verified Data Synthesis · p. 4, Figure 3 · p. 5
- The retained data spans all three dimensions and has long-tailed visual styles, with the top 20 styles covering only 18%.The authors report that this diversity is important for generalization.Source: Execution-Verified Data Synthesis · pp. 4–5
Main Results
On IWR-Bench, execution-driven refinement and execution-verified fine-tuning complement one another: the fine-tuned 9B model with RILA surpasses larger one-shot generators, while RILA also improves stronger backbones.
Main Results
- 57.52%: fine-tuned Qwen3.5-9B† with RILA exceeds one-shot Kimi-K2.6 (55.61%) and GPT-5.5 (55.74%).The result combines execution-verified-data fine-tuning with RILA guided by a Kimi-K2.6 critique.Source: Table 1 · p. 6, Main Results · p. 6
- 65.85%: Kimi-K2.6 with RILA improves over its one-shot score of 55.61%, while GPT-5.5 rises from 55.74% to 63.00%.Both use a Kimi-K2.6 critique under RILA.Source: Table 1 · p. 6, Main Results · p. 6
- 50.34%: applying RILA to the 9B backbone raises its Final Score from 40.40% (+9.94) without additional training.The improvement simultaneously raises functional correctness and visual consistency.Source: Table 1 · p. 6, Main Results · p. 6
Ablation Study
The ablations isolate how component choices, critique strength, and iteration budget affect refinement, then test transfer to Interaction2Code.
Ablation Study
- Removing selection drops Final 50.34→48.56; removing execution feedback drops VFS 58.05→54.34; replacing tool edits yields 45.52 Final.Selection's loss is chiefly IFS (−2.41; VFS −0.33); feedback removal changes IFS by −0.46, while regeneration lowers IFS and VFS by 1.56 and 1.71.Source: Table 2 · p. 6, Ablation Study · pp. 6–7
- Stronger Kimi-K2.6 critique raises Final 45.07→50.34 for Qwen3.5-9B and 51.02→57.52 for fine-tuned Qwen3.5-9B†.Even compact critique improves over one-shot generation for both code models; stronger critique widens the gain.Source: Table 3 · p. 6, Ablation Study · p. 7
- Running-best ERS rises 0.500→0.671, but the first three iterations recover 95.3% of total improvement and the fourth adds about one-tenth of the first.The authors therefore use three SWE iterations throughout.Source: Figure 4 · p. 7, Ablation Study · p. 7
- On Interaction2Code, refinement improves every metric, including interaction implementation (+0.027) and full-page text fidelity (+0.025).The largest gains also include interaction-region text fidelity (+0.021); interaction position agreement rises +0.019. Similarity metrics use 301 paired tasks, while interaction implementation uses all 375.Source: Ablation Study · p. 7, Table 4 · p. 7