PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation

Tianyi Xiong, Zhengyuan Yang, Xiaofei Wang, Chung-Ching Lin, Ruichun Ma, Kevin Lin, Zhendong Wang, Linjie Li, Chenxi Liu, Ruibo Chen, Ramani Duraiswami, Heng Huang, Lijuan Wang

arXiv:2608.24138v1cs.CV

TL;DR

UI-to-code self-evolution is unstable because local code edits can propagate through coupled visual dependencies, degrading regions that were previously faithful. RubSE structures visual feedback as typed rubrics, selects focused repair targets, and preserves repair history. Across six VLMs and three benchmarks, it improves final-round and best-round performance over naïve self-evolution while supporting recovery from regressions and transfer to weaker code improvers.

  • Problem

    Test-time self-evolution for UI-to-code remains underexplored and does not reliably improve monotonically because visual feedback can trigger broad, coupled changes.

  • Method

    RubSE uses an EVOLVE–SELECT–HISTORY loop to represent visual mismatches as structured rubrics, prioritize one repair target, and preserve prior repair context.

  • Results

    15/18 final-round settings improve on both metrics, averaging +1.20 overall points and +0.11 aspect-mean score; best-round performance improves in 14/18 settings with +1.13 overall points on average.

  • Takeaways & Limitations

    Rubric-guided repair makes test-time UI-to-code refinement more structured and reliable, including recovery from severe visual regressions and transfer of guidance to weaker code improvers.

  • Takeaways & Limitations

    Evaluation is concentrated on selected frontier and open-source models generating HTML/CSS webpages, while broader model families and UI languages remain underexplored.

Abstract

from arXiv · show

Large vision-language models have shown strong progress in UI-to-code generation, yet their test-time self-evolution remains unstable. We first identify a fundamental obstacle, termed visual repair coupling: a local code edit may propagate through layout, style, and component dependencies, correcting one visual mismatch while degrading regions that were previously faithful. To address this issue, we present RubSE, a Rubric-guided Self-Evolution framework that uses rubrics to represent visual feedback as a structured visual-repair context. At each refinement round, RubSE generates typed candidate rubrics, selects one prioritized repair target, and stores previously selected rubrics as history, thereby steering each revision toward a well-scoped visual repair while discouraging repeated or over-broad changes. Evaluations across six VLMs and three UI-to-code benchmarks demonstrate that RubSE substantially outperforms naïve self-evolution in final-round and best-round settings, achieving more stable refinement trajectories and a higher trajectory-level performance ceiling. Further analysis shows that RubSE mitigates trajectory collapse by improving recovery from severe visual regressions, and that stronger rubric generators can transfer effective visual-repair guidance to weaker code improvers.

1 Introduction

UI-to-code self-evolution is unstable because local edits can propagate through coupled layout, style, and component dependencies. RubSE addresses this by structuring visual feedback into prioritized, persistent repair context and improves refinement outcomes across models and benchmarks.

  • Motivation: UI-to-code test-time improvement remains less explored, despite rendered outputs providing a natural source of visual feedback.Existing work primarily emphasizes training-time interventions, while self-evolution iteratively critiques and revises rendered code.
  • Problem: Visual repair coupling allows a local code edit to correct one mismatch while damaging previously faithful regions through non-local rendered changes.The coupling arises from dependencies among layout, style, and components, producing unstable refinement trajectories.
  • Approach: RubSE represents each visual failure as a typed rubric containing a specific mismatch, visual aspect, and targeted correction direction.This structured representation makes the repair objective explicit while preserving flexibility in how the code edit is implemented.
  • Approach: RubSE separates repair discovery, target selection, and context preservation through its EVOLVE–SELECT–HISTORY loop.EVOLVE generates candidates, SELECT chooses one prioritized repair, and HISTORY carries prior decisions forward to discourage repeated or over-broad revisions.
  • Results: 15/18 final-round settings improve on both metrics, with average gains of +1.20 overall points and +0.11 aspect-mean score over naïve self-evolution.RubSE also achieves stronger best-round performance in 14/18 settings, averaging +1.13 overall points and showing more stable trajectories and a higher performance ceiling.
  • Results: Further analyses show that rubric-guided repair accelerates recovery from severe visual regressions and transfers effective guidance from stronger rubric generators to weaker code improvers.These findings support the framework’s role in reducing trajectory collapse and providing actionable, well-scoped repair targets.

2 Related Work

UI-to-code research has progressed from task-specific screenshot translation toward flexible vision-language generation and larger benchmarks. Related work also improves models through data, staged pipelines, program filtering, decomposition, layout-aware reasoning, and rubric-based evaluation.

  • UI-to-Code Generation: Early UI-to-code methods translated screenshots into intermediate UI descriptions or HTML using encoder–decoder architectures.Recent vision-language models enable more flexible generation from visual inputs.
  • UI-to-Code Generation: Design2Code introduced a real-world UI-to-code benchmark, while related work also constructs benchmarks from synthetic or real-world webpages.These benchmarks support evaluation of screenshot-to-webpage generation.
  • UI-to-Code Improvement: Prior improvement methods use high-quality image–code data, stage-wise pipelines, compiler or multimodal filtering, hierarchy decomposition, and layout-aware reasoning.Recent studies also adapt reinforcement learning with verifiable feedback.
  • Rubrics in LLM/VLMs: Rubrics are used in LLM-as-a-Judge systems to decompose open-ended evaluation into criterion-level decisions and improve judgment reliability.This work positions rubrics as structured evaluation criteria in domains where holistic reward signals are insufficient.

3 Method

RubSE formulates UI-to-code self-evolution as sequential repair and uses structured, localized rubric context to constrain each revision amid non-local visual repair coupling. Its EVOLVE–SELECT–HISTORY loop generates typed candidates, prioritizes one repair, and carries prior targets forward.

  • Task Formulation: UI-to-code self-evolution is formulated as sequential repair from an initial generated code draft and rendering toward a screenshot-faithful reconstruction.At each round, the feedback interface constructs repair context for refining the previous code state.
  • Visual Repair Coupling: Visual repair coupling makes code edits propagate through layout, style, and component dependencies, so textual changes can produce non-local rendered-pixel differences.The rendering map can cascade effects through layout, flex/grid sizing, font metrics, and asset placement.
  • Motivation: Preliminary self-evolution experiments show unstable trajectories, including declines below initial generation and broad visual degradation after attempted local fixes.On Design2Code, GPT-5.4 declined below its initial generation by −0.20, while a representative repair introduced errors in page scale, bar length, and background color.
  • RubSE: RubSE represents feedback as a rubric naming a visual mismatch, specifying a correction direction, and focusing the next round on one prioritized repair target.Rubrics provide soft control: they make the objective explicit without imposing hard constraints on generated code.
  • RubSE: EVOLVE generates typed candidate rubrics, SELECT chooses one expected to maximize improvement under a targeted edit, and HISTORY passes prior selections forward as an avoid list.The rubric taxonomy spans five visual aspects, while history discourages repeated or over-broad revisions.
  • RubSE: Each selected rubric conditions the next HTML/CSS revision on the target screenshot, previous code and rendering, and accumulated repair context.Separating candidate generation from prioritization keeps selection distinct under a constrained edit budget.

4 Experiments

RubSE is evaluated across six VLMs and three UI-to-code benchmarks using overall and aspect-level visual-fidelity metrics. It improves stability and best-round performance over naïve self-evolution, while effects vary across model families.

  • On frontier executors, RubSE gains +1.90 judge and +0.16 aspect points, while nine Qwen settings gain +0.50 overall and +0.06 aspect points.
  • At selected best rounds, RubSE outperforms naïve self-evolution on both metrics in 14/18 settings.The best round is selected by overall score, with the aspect score taken from that same round.
  • RubSE reaches its best frontier-model checkpoint later than naïve self-evolution, at r=5.7 versus r=3.0, with larger gains over direct generation.Average gains over direct generation are +2.22 versus +0.77 overall points.
  • RubSE improves over direct generation in all nine Qwen settings, but Qwen3-VL-32B-Instruct shows mixed benchmark-specific results.Its model-specific rubric imbalance overemphasizes spacing-density adjustments and underweights completeness issues.
  • RubSE incurs 1.60× the API cost with GPT-5.4 and 2.5% higher inference latency with Qwen3.5-9B than naïve self-evolution.

5 Analysis

The analysis examines RubSE through ablations, trajectory-collapse metrics, recovery behavior, and rubric transfer. Results indicate that target selection and history improve repair control, while stronger rubric generators help weaker code improvers.

  • 5.1 Ablation Study: The full RubSE achieves the strongest and most consistent ablation gains across both evaluation metrics on UI2Code-Real with GPT-5.2.
  • 5.1 Ablation Study: Selecting one rubric stabilizes gains by focusing edits on crucial localized issues, whereas using all rubrics or a random rubric produces unstable later-round improvements.
  • 5.1 Ablation Study: Removing rubric history preserves consistent but smaller improvements, as past rubrics reduce repeated local repairs and encourage exploration across visual dimensions.
  • 5.2 How RubSE Mitigates Trajectory Collapse?: For frontier VLMs, RubSE reduces collapse from 18.9% to 12.8% on average and improves recovery from 20.7% to 32.8% on average.For Qwen executors, average collapse increases from 14.8% to 18.1%, while average recovery improves from 23.5% to 30.2%.
  • 5.3 Do Stronger Rubrics Transfer to Weaker Code Improvers?: GPT-5.4-generated rubrics outperform self-generated rubrics across all three Qwen models, producing stronger and more stable iterative gains, especially early.
  • 5.3 Do Stronger Rubrics Transfer to Weaker Code Improvers?: Qwen-generated rubrics are more low-level and narrowly scoped, while GPT-generated rubrics provide higher-level, actionable repair targets beyond local CSS-level loops.On Design2Code, CSS/HTML cues occur in 70% versus 53% of rubrics, and numeric or color constants in 35–46% versus 7%.

6 Conclusion

RubSE makes UI-to-code self-evolution more controlled by representing visual-repair context as structured rubrics and iteratively exploring, selecting, and preserving repair targets. Experiments show improved final-round stability and best-round performance over naïve self-evolution, alongside better recovery from regressions and transfer across model strengths.

  • RubSE defines each rubric as a focused repair target, its visual mismatch, and a possible correction direction.The framework uses rubrics to instantiate visual-repair context for UI-to-code generation.
  • The EVOLVE–SELECT–HISTORY loop explores candidate repair directions, selects one prioritized target, and carries previous rubrics forward.History discourages repeated or over-broad revisions while preserving open-ended code revision.
  • RubSE improves both final-round stability and best-round performance over naïve self-evolution across frontier and open-source vision-language models.The conclusion reports this outcome across the evaluated model groups.
  • Rubric-guided repair helps recover from severe visual regressions, while high-quality rubrics transfer from stronger rubric generators to weaker code improvers.These analyses identify recovery and cross-model transfer as additional benefits of the framework.

Limitations

The evaluation focuses on six model variants generating HTML/CSS webpages, while code correctness and evaluation stability remain imperfectly addressed.

  • Experiments evaluate three frontier models and three open-source variants on HTML/CSS webpage generation, leaving broader model families and UI languages underexplored.
  • RubSE does not explicitly guarantee code correctness, so refinement may introduce syntax or runtime errors that cause rendering failures.The authors identify an explicit debugging loop as a promising direction.
  • VLM-based evaluation can diverge from human judgments in a minority of cases, especially for subtle visual differences in later refinement rounds.The authors support the main comparison with a human study and call for more robust visual graders.

A Evaluation Protocol

The evaluation compares rendered outputs with target images using holistic and aspect-level VLM judges, while testing whether the metrics capture localized refinement changes. CLIP is excluded because it is less informative for fine-grained iterative behavior.

  • Generated code quality is evaluated by comparing its rendered image with the target image under a VLM-as-a-Judge paradigm.The protocol uses VLM judges in complementary overall and aspect-level modes.
  • The overall-rating judge assigns a single 0–100 score representing holistic visual fidelity.The overall assessment covers layout, spacing, typography, styling, and completeness.
  • The aspect-level judge scores five visual-fidelity aspects independently on a 1–7 Likert scale and averages them into the final aspect-level metric.This measure is intended to capture localized visual changes during iterative refinement.
  • Manual inspection found matching directions between aspect-level score changes and human judgments in 47 of 60 consecutive-round rendered-image pairs.This provides a sanity check for using the aspect-level metric to track localized refinement trends.
  • CLIP was excluded because its similarity scores can miss localized iterative changes: in 17 inspected pairs, scores were nearly unchanged or opposed human-observed improvement or degradation.

B.1 Human Evaluation

The human evaluation compares round-10 RubSE and naïve self-evolution outputs on 60 UI2Code-Real samples for GPT-5.2 and GPT-5.4. RubSE is preferred for both models, and additional judge runs reproduce the main ordering in nearly all settings.

  • Human pairwise evaluation: The study compares round-10 RubSE and naïve self-evolution outputs on 60 UI2Code-Real samples generated by GPT-5.2 and GPT-5.4.Two authors independently evaluate each pair before resolving disagreements into win, tie, or loss labels.
  • Human pairwise evaluation: RubSE is clearly preferred over naïve self-evolution for both GPT-5.2 and GPT-5.4, with statistically significant exact sign tests on nontied comparisons.
  • Evaluation variability: Additional judge runs match the main RubSE-versus-naïve ordering in every frontier setting and seven of nine open-source settings.RubSE has larger mean improvement in all nine frontier-model settings and seven of nine open-source-model settings.
  • Evaluation variability: Additional-run standard deviations range from 0.09 to 0.82 points and are generally smaller for frontier models.

B.3 Computational Cost

RubSE adds computation through extra rubric-generation and selection calls, increasing API cost more than open-source inference latency. The added expense accompanies more stable and effective refinement.

  • Open-source inference latency: 2.5% additional latency makes RubSE’s open-source inference time nearly comparable to naïve self-evolution.RubSE takes 105.3 seconds per round versus 102.7 seconds, with long HTML decoding dominating runtime.
  • API-based cost: 1.60× higher GPT-5.4 monetary cost accompanies RubSE’s structured repair context compared with naïve self-evolution.RubSE uses 3.16× as many input tokens, 1.21× as many output tokens, and 2.38× as many total tokens.
  • Effectiveness and efficiency: 1.4× the average round-10 performance gain is achieved with Qwen3.5-9B on the selected 50-sample subset.The comparison uses 50 randomly selected UI2Code-Real samples over 10 refinement rounds.
  • Effectiveness and efficiency: RubSE’s moderate additional cost is spent on structuring repair context rather than generating more code.The efficiency trade-off is presented alongside more stable and effective self-evolution.
  • Separation of EVOLVE and SELECT: 2.5 cents per round is the cost of merging EVOLVE and SELECT, versus 2.4 and 1.8 cents for separate calls.The merged request averages 6,168.8 input tokens and 641.7 output tokens per round.

B.4 Understanding the Exception of Qwen3-VL-32B-Instruct

Qwen3-VL-32B-Instruct benefits less consistently because its rubric distribution overemphasizes local spacing adjustments and underemphasizes completeness. Stronger externally generated rubrics provide broader structural guidance and support more controlled trajectories.

  • Rubric-type distribution: Qwen3-VL-32B-Instruct generates more spacing-density and fewer completeness rubrics than Qwen3.5-9B and Qwen3.6-35B-A3B.The imbalance is only partially corrected by SELECT.
  • Rubric-type distribution: Its selected targets remain focused on margins and line height while underemphasizing missing or extraneous components.This pattern can trap refinement in repeated local edits.
  • Rubric-type distribution: GPT-5.4-generated rubrics substantially reduce the rubric imbalance and clearly outperform naïve self-evolution.The passage attributes the improvement to stronger rubric guidance for the Qwen3-VL-32B-Instruct setting.
  • Trajectory-level behavior: RubSE produces more controlled and consistent improvements, whereas naïve self-evolution oscillates across rounds and repeatedly alters faithful regions.The comparison is reported as a trajectory-level analysis on the same target screenshot.
  • Rubric granularity: GPT-generated rubrics provide broader and more structural guidance than self-generated rubrics for the same previous-round Qwen outputs.The examples explain the granularity difference behind the quantitative trends.

E Additional Discussion

RubSE is positioned as a model-agnostic repair policy that complements coding agents’ execution environments. The discussion also documents benchmark licensing, artifact use, qualitative trajectories, rubric examples, and implementation prompts.

  • Relationship to General-Purpose Coding Agents: RubSE complements general-purpose coding agents by specifying how rendered feedback is structured, rather than defining their execution actions.Coding agents provide capabilities such as file access, command execution, screenshot inspection, and iterative code editing.
  • Artifacts and evaluation setting: The benchmarks use publicly available UI screenshots or rendered webpages and do not involve newly collected user data.UI2Code is used under MIT licensing and Design2Code under ODC-By licensing.
  • Qualitative comparisons: The qualitative trajectory comparison shows naïve self-evolution distorting the cookie modal after fixing other regions, while RubSE progressively addresses major interface elements.The comparison is reported in Table 11 for iterative rounds generated by GPT-5.2.
  • Qualitative comparisons: GPT-5.4 rubrics identify higher-level targets such as page framing, missing visual regions, and global layout structure, unlike more local self-generated rubrics.These examples compare rubric guidance for the same previous-round Qwen outputs.
  • Implementation prompts: The appendix provides separate prompts for naïve self-evolution, rubric generation, rubric selection, and single-step rubric-guided code improvement.The RubSE prompts are documented in Tables 14–16, while naïve self-evolution appears in Table 13.
Loading 2608.24138v1…