PaperlayerView exact revision on arXiv

Source-linked AI summary

ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Coding

Yuhang Li, Chenchen Zhang, Ruilin Lv, Ao Liu, Ken Deng, Yuanxing Zhang, Jiaheng Liu, Wiggin Zhou, Bo Zhou

arXiv:2510.11498v1cs.LGcs.CL

TL;DR

Front-end code generation is difficult because correctness depends on rendered appearance and interaction, which text-only methods cannot assess and existing approaches do not combine with iterative refinement. ReLook trains an agent to use an MLLM for visual feedback and reward, with safeguards for renderability and revision quality. It reports consistent gains over strong baselines across benchmarks and presents critic-free inference as a route to speedups.

  • Problem

    Front-end code quality depends on rendered appearance and interaction, but existing systems either generate without iterative refinement or refine without visual perception.

    Source: §1 · p. 1

  • Method

    ReLook trains an agent to invoke an MLLM for rendered-output feedback and perceptual reward, with zero reward for invalid screenshots and Forced Optimization for revisions.

    Source: §1 · pp. 1–2

  • Results

    Across three widely used benchmark datasets, ReLook significantly outperforms baselines, and the conclusion reports consistent gains over strong baselines.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The paper presents a perception-aligned evaluator inside the learning loop as a blueprint it expects to generalize beyond web UIs to other perceptual programming domains.

    Source: §5 · p. 8

    Using a large MLLM critic during training increases compute and monetary cost.

    Source: A Limitations and Future Work · p. 11

Abstract

from arXiv · show

While Large Language Models (LLMs) excel at algorithmic code generation, they struggle with front-end development, where correctness is judged on rendered pixels and interaction. We present ReLook, an agentic, vision-grounded reinforcement learning framework that empowers an agent to close a robust generate--diagnose--refine loop by invoking a multimodal LLM (MLLM) as a tool. During training, the agent uses the MLLM-in-the-loop both as a visual critic--scoring code with screenshots--and as a source of actionable, vision-grounded feedback; a strict zero-reward rule for invalid renders anchors renderability and prevents reward hacking. To prevent behavioral collapse, we introduce Forced Optimization, a strict acceptance rule that admits only improving revisions, yielding monotonically better trajectories. At inference, we decouple the critic and run a lightweight, critic-free self-edit cycle, keeping latency comparable to base decoding while retaining most of the gains. Across three widely used benchmarks, ReLook consistently outperforms strong baselines in vision-grounded front-end code generation, highlighting the benefits of agentic perception, visual rewards, and training-inference decoupling.

1 Introduction

Front-end code generation remains challenging because correctness depends on rendered appearance and interaction, while existing approaches either generate without refining or refine without visual perception. ReLook uses an MLLM-grounded generate–diagnose–refine loop with safeguards, and reports gains across benchmarks.

1 Introduction

  • Front-end quality depends on rendered pixels and interaction, but existing systems either generate without refinement or refine without seeing rendered outputs.Unlike binary unit tests, front-end quality lies on a continuum where even a single misaligned pixel can signify failure.Source: §1 · p. 1
  • ReLook closes the generate–diagnose–refine loop by having an agent invoke an MLLM to inspect rendered outputs and provide actionable feedback.During training, the MLLM also supplies a perceptual reward signal.Source: §1 · pp. 1–2
  • Offline evaluator validity is inherited from ARTIFACTSBENCH’s validation, and the authors conduct no additional human studies.The training critic and offline evaluator are decoupled to further mitigate on-policy judge overfitting.Source: §1 · p. 1
  • A strict zero-reward rule for answers without valid screenshots is designed to deter reward hacking and force renderable code.The reward system uses an MLLM to provide pixel-level training feedback.Source: §1 · p. 2
  • Forced Optimization accepts only strictly improving revisions, addressing behavioral collapse and producing monotonically improving rollout trajectories.Behavioral collapse occurs when a revision scores worse despite high-quality feedback.Source: §1 · pp. 1–2
  • Across three widely used benchmark datasets, ReLook significantly outperforms baselines and is compatible with different LLMs.The introduction reports broad applicability without providing benchmark-specific scores.Source: §1 · p. 2

2 Method

ReLook trains an agent to iteratively generate, inspect, and refine front-end code using rendered screenshots and MLLM feedback. Its training combines visual rewards with renderability safeguards and forced acceptance of improving revisions, while inference uses a lightweight critic-free self-edit cycle.

2.1 Problem Formulation

  • Front-end generation produces SVG, HTML, CSS, and JavaScript for a natural-language request, with correctness judged by rendered appearance and behavior rather than syntax alone.Source: §2.1 · p. 2

2.2 Overall Framework

  • ReLook trains an agent to invoke an MLLM to inspect rendered output, score revisions, and provide feedback within a generate–diagnose–refine loop.At inference, the framework can discard the external critic and use a lightweight self-edit cycle.Source: §2.2 · p. 2, Figure 1 · p. 2

2.3 Iterative Reflection Mechanism

  • Each reflection round can execute code in a sandbox, capture screenshots, obtain MLLM feedback, and feed the query, text, code, and feedback into the next round.Reflection stops when feedback is not requested or the round cap is reached.Source: §2.3 · p. 3

2.4 Reinforcement Learning Framework

  • Training uses GRPO with token-level policy gradients; advantages are computed relative to a group-mean baseline, while critic tokens are masked from policy updates.The framework also optionally distills critic feedback tokens toward frozen MLLM outputs for test-time self-reflection.Source: §2.4 · p. 3

2.5 Reward Design

  • The MLLM scores rendered pages using three temporal screenshots per reflection round, while any invalid required screenshot receives zero reward.A visual-output prompt increases the Valid Render Rate from about 40% initially to about 80% at convergence.Source: §2.5 · p. 3

2.6 Forced Optimization

  • Forced Optimization accepts a revision only when its reward exceeds the trajectory’s best-so-far, rejecting non-improving attempts and terminating after ten failed resamples.This addresses behavioral collapse and produces monotonically improving accepted trajectories.Source: §2.6 · p. 4

2.7 Efficient Inference

  • At inference, ReLook drops the external MLLM and runs a critic-free self-edit cycle of at most three rounds to reduce latency while preserving most gains.Screenshots and MLLM calls are disabled in this inference cycle.Source: §2.7 · p. 4

3 Experiment

The experiments evaluate ReLook on front-end benchmarks and compare it with base models and vision-grounded RL without agentic reflection. ReLook improves benchmark performance, while ablations, inference measurements, and task-level analyses examine its safeguards and scope.

3.1 Experimental Settings

  • The 3,000-task training corpus is deduplicated against evaluation benchmarks using lexical, DOM, and code similarity, with borderline cases manually reviewed.The evaluation includes ArtifactsBench, FullStack-Bench-Html, and Web-Bench; the authors use six ArtifactsBench subsets to manage costs.Source: §3.1 · p. 4

3.2 Evaluation Protocol

  • VisualScore uses three temporal screenshots and Gemini-2.5-Pro; FullStack-Bench-Html reports unit-test pass rate and Web-Bench reports pass@2.Scores are averaged over three runs; the evaluation protocol is inherited from ArtifactsBench, whose MLLM evaluators have reported human agreement above 90%.Source: §3.2 · p. 5, §3.1 · p. 5

3.3 Main Results

  • Across ArtifactsBench subsets, ReLook improves over Base Model and Web-RL, and ReLook-w/o-MLLM also outperforms Web-RL.The main results cover both Qwen2.5-7B and Llama-3.1-8B backbones; Figure 2 shows consistent improvements across subsets.Source: Table 1 · p. 6, §3.3 · p. 6, Figure 2 · p. 6
  • On ArtifactsBench-Lite, ReLook > Web-RL > Base Model; Figure 4 shows ReLook improving monotonically while the base model degrades after 2–3 rounds.The Figure 4 scores use the training-time Qwen2.5-VL-72B judge, and reflection is capped at three rounds for inference efficiency.Source: Figure 3 · p. 6, §3.3 · p. 6, Figure 4 · p. 7

3.4 Ablation Study

  • Ablations attribute +3.3 points to vision reward, +1.0 to the format constraint, and +2.0 to Forced Optimization on ArtifactsBench-Lite.Forced Optimization has the largest reported impact and is described as mitigating behavioral collapse.Source: Table 2 · p. 7, §3.4 · p. 7

3.5 Inference speed improvement after removing MLLM

  • ReLook averages 123.04 seconds per query, compared with 18.03 seconds for ReLook-w/o-MLLM.The measurement used 100 queries and a maximum of three reflection steps; removing screenshot and MLLM calls substantially improves inference efficiency.Source: §3.5 · p. 7

3.6 Error Analysis and Task-Level Performance

  • Visual-centric subsets show 3.2–6.1-point gains over base models, while Qwen2.5-7B improves from 3.00 to 4.20 Web-Bench pass@2, a 40% relative gain.The paper reports that absolute Web-Bench performance remains low and identifies long-horizon reasoning and cross-file dependencies as continuing challenges.Source: §3.6 · p. 7

3.7 Qualitative Analysis

  • Qualitative examples identify layout drift, broken interactions, and aesthetic inconsistency in non-vision baselines, which render-aware refinements mitigate.The cited examples include misaligned components, missing event listeners, and clashing colors.Source: §3.7 · p. 7

4 Related Work

Prior visual-code systems often target static similarity without iterative refinement, while code-RL feedback can miss pixel-level quality. ReLook combines temporal screenshot rewards, agentic refinement, and an online monotonic acceptance rule.

4.1 Visual Code Generation

  • Earlier vision-to-code systems generate from static screenshots, while later methods add structure but still struggle with dynamics and iterative refinement.ReLook instead trains with a renderer in the loop, temporal screenshot rewards, and RL to internalize refinement.Source: §4.1 · p. 8

4.2 Feedback-Driven Code Reinforcement Learning

  • Unit-test rewards are pixel-blind for front-end code; ReLook couples temporal screenshot rewards with Forced Optimization and zero reward for invalid renders.Forced Optimization requires monotonic improvement within one trajectory, unlike best-of-N or offline re-ranking.Source: §4.2 · p. 8

4.3 Multimodal UI Perception and Evaluation

  • ReLook places MLLM visual rewards in RL to internalize layout and interaction principles, then drops the critic at inference for lower latency.The paper positions this approach alongside multimodal UI perception and MLLM-as-judge evaluation.Source: §4.3 · p. 8

5 Conclusion

ReLook closes a generate–diagnose–refine loop for front-end code using visual rewards and safeguards. It reports consistent gains over strong baselines and critic-free inference speedups.

5 Conclusion

  • ReLook combines MLLM visual rewards, zero reward for invalid renders, and Forced Optimization, reporting gains over strong baselines and critic-free inference speedups.The paper expects perception-aligned evaluation in the learning loop to generalize beyond web UIs to other perceptual programming domains.Source: §5 · p. 8

A Limitations and Future Work

The authors identify costs, evaluation and environment constraints, remaining task difficulties, and limits to the current experimental scope. They outline these as directions for further work.

A Limitations and Future Work

  • Training with a large MLLM critic increases compute and monetary cost, despite critic-free inference; lighter judges or distillation remain future work.The experiments focus on 7B/8B models and front-end code; broader model scales and software stacks need further investigation.Source: A Limitations and Future Work · p. 11
  • The deterministic sandbox may under-represent real-world variation across devices, locales, and resource conditions, leaving robustness gaps.The authors also note that external-evaluator rewards and a length penalty may be biased by metric drift or prompt-template sensitivity.Source: A Limitations and Future Work · p. 11
  • Forced Optimization may reduce exploration in challenging cases, and complex multi-file projects remain difficult beyond single-artifact refinement.The authors identify long-horizon reasoning as requiring further architectural innovation.Source: A Limitations and Future Work · p. 11

B External validity and cross-judge analysis

The paper inherits ARTIFACTSBENCH’s evaluator-validity evidence and uses its evaluation protocol. It interprets scores relative to a difficult benchmark and reports them on a fixed scale with a multidimensional rubric.

B External validity and cross-judge analysis

  • Over 90% human–MLLM agreement and strong correlations with WebDev Arena preferences support the evaluator’s validity.The human study reported Cohen’s κ > 0.85; crowd-ranking correlations were ρ = 0.87 for Gemini-2.5-Pro and ρ = 0.83 for Qwen2.5-VL-72B.Source: B External validity and cross-judge… · p. 11
  • The test sets are strict subsets of ARTIFACTSBENCH’s 1,825 human-validated tasks, and the study reuses its scripts, prompts, and rubric without modification.The authors did not rerun human studies and separated the training-time Qwen2.5-VL-72B-Instruct critic from the Gemini-2.5-Pro offline evaluator.Source: B External validity and cross-judge… · p. 11
  • Scores of 20–30 for 7B/8B models reflect task difficulty; GPT-4o scores 33 and Qwen2.5-32B scores 26.The paper reports relative improvements over baselines on the 0–100 scale.Source: B External validity and cross-judge… · p. 11

D Evaluator rubric and scoring range

  • The [0, 100] VisualScore rubric covers specification adherence, layout and spatial fidelity, typography and color coherence, and interactive integrity when actions are specified.Dynamic-task rounds jointly score three temporal screenshots; trajectory reward averages round-level scores, without multi-judge averaging or smoothing.Source: D Evaluator rubric and scoring range · p. 11

E The loss function for Forced Optimization

The section describes a strict acceptance rule for revisions and a training loop that scores trajectories, updates the policy with GRPO, and can stop when improvement ceases. It contrasts this rule with an earlier penalty that reduced reflection frequency.

E The loss function for Forced Optimization

  • The training framework initializes task history and reflection state, extracts code, renders it, requests MLLM feedback, and computes a visual score for each proposal.The algorithm specifies a maximum of 10 resampling attempts per reflection round.Source: E The loss function for Forced… · p. 12
  • Only strictly improving revisions enter the trajectory; a rejected revision ends further reflections and preserves the best-so-far result.Forced Optimization resamples proposals, while the earlier negative-reward penalty was removed after it encouraged less reflection and reward hacking.Source: E The loss function for Forced… · pp. 11–13
  • After trajectory reward components are calculated, GRPO updates policy parameters; training exits when the convergence criterion is met.The loop then advances to the next training step if it has not exited.Source: E The loss function for Forced… · p. 12
  • A valid screenshot receives VisualScore, while an invalid screenshot receives zero reward.The trajectory reward also includes zero when its length exceeds Lend.Source: E The loss function for Forced… · p. 12

F Lightweight distillation loss on feedback tokens

The section describes lightweight distillation of critic feedback-token behavior from a frozen MLLM and distinguishes those tokens from the policy-controlled tokens used for RL optimization.

F Lightweight distillation loss on feedback tokens

G Reproducibility notes: GRPO and implementation details

The reproducibility notes specify optimization controls and decoding settings. They also report an increase in valid renders during training.

G Reproducibility notes: GRPO and implementation details

  • The frozen instruction-tuned backbone serves as the reference policy, with token-wise KL regularization weighted by β and no within-step annealing.Source: G Reproducibility notes: GRPO and… · p. 13
  • Trajectories mix policy and critic tokens; RL advantages are zeroed on critic tokens, with optional lightweight feedback distillation.Source: G Reproducibility notes: GRPO and… · p. 13
  • Forced Optimization caps resampling at 10 attempts per round and stops reflection when no proposal improves, retaining the best-so-far result.The notes describe this cap as balancing exploration and computational efficiency.Source: G Reproducibility notes: GRPO and… · p. 13
  • The length penalty uses Lstart = 12,000 and Lend = 14,000 tokens, and decoding uses temperature 1.0 and top-p 0.7 across three random seeds.Fixtures are cached to avoid network variance.Source: G Reproducibility notes: GRPO and… · p. 13
  • Valid Render Rate rises from approximately 40% at initialization to approximately 80% upon convergence.The authors attribute this to the visual-output constraint in prompts combined with the zero-reward penalty for invalid renders.Source: G Reproducibility notes: GRPO and… · p. 13

H Qualitative Analysis

Figure 6 compares baseline and ReLook websites across representative front-end tasks. ReLook reduces layout drift, improves interaction correctness, and achieves higher visual fidelity than the baseline.

H Qualitative Analysis

  • The baseline can render without error but often has layout drift, incomplete functionality, and weak visual coherence; ReLook examples include clickable cards, interactive forms, and a precisely aligned chessboard.ReLook outputs are described as visually faithful and functionally aligned with prompts.Source: H Qualitative Analysis · p. 14
  • ReLook reduces layout drift, strengthens interaction correctness, and achieves higher visual fidelity than the baseline across representative front-end tasks.Figure 6 presents the qualitative comparison.Source: H Qualitative Analysis · pp. 13–14, Figure 6 · p. 14

I Template prompt for ReLook rollout

Loading 2510.11498v1…