PaperlayerView exact revision on arXiv

Source-linked AI summary

UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization

Zhen Yang, Wenyi Hong, Mingde Xu, Xinyue Fan, Weihan Wang, Jiale Cheng, Xiaotao Gu, Jie Tang

arXiv:2511.08195v4cs.CV

TL;DR

UI-to-code methods often generate executable interfaces in one pass, mismatching the iterative, feedback-driven nature of real UI development. The paper reframes the task as interactive visual optimization and introduces RVPO, which learns from relative preferences among rendered candidates. UI2CodeN, a 9B open-source model, achieves state-of-the-art performance across UI generation, polishing, and editing while iterative refinement improves visual fidelity.

  • Problem

    Most UI-to-code approaches use single-pass generation even though rendering effects, coupled visual discrepancies, and executable feedback make UI development iterative.

  • Method

    The paper uses an interactive optimization loop with rendered feedback and trains UI2CodeN using RVPO, which optimizes relative visual preferences rather than unstable absolute rewards.

  • Results

    UI2CodeN achieves state-of-the-art performance on UI-to-code generation, UI polishing, and UI editing benchmarks, outperforming substantially larger open- and closed-source models.

  • Takeaways & Limitations

    Iterative visual optimization provides a model-agnostic strategy for executable generation problems with black-box feedback.

  • Takeaways & Limitations

    UI editing is limited to localized, well-specified changes rather than general instruction-driven UI design.

Abstract

from arXiv · show

UI-to-code aims to translate UI screenshots into executable front-end code. Despite progress with vision-language models (VLMs), most existing methods formulate UI-to-code as a single-pass generation, which mismatches real-world UI development that is inherently iterative and feedback-driven. We reformulate UI-to-code as an interactive visual optimization problem, where code generation is embedded in a closed-loop process of execution, visual inspection, and iterative refinement driven by rendered visual feedback. To address the non-differentiability of visual objectives and the noise of absolute visual evaluators, we propose Relative Visual Policy Optimization (RVPO), a preference-based reinforcement learning method that optimizes relative visual rankings among rendered candidates under execution feedback. We instantiate this paradigm in UI2Code^N, an open-source 9B model trained via continual pre-training, supervised fine-tuning, and reinforcement learning. Experiments demonstrate state-of-the-art performance on UI drafting, UI polishing, and UI editing benchmarks, even outperforming larger models, with performance consistently improving through iterative visual optimization. Our code and models are available at https://github.com/zai-org/UI2Code_N.

1. Introduction

UI2CodeN reframes UI-to-code from single-pass generation as interactive visual optimization, using rendered feedback to iteratively improve executable interfaces. It introduces RVPO and instantiates the paradigm in a compact 9B model that achieves state-of-the-art results across multiple UI benchmarks.

  • Motivation: UI-to-code is inherently iterative and feedback-driven, but most existing methods treat it as single-turn generation despite rendering effects and coupled visual discrepancies.Rendered feedback enables self-verification and correction during development.
  • Interactive Visual Optimization: The paper formulates UI drafting, polishing, and editing as one interactive optimization loop over executable artifacts and rendered visual outcomes.The formulation targets visual fidelity under non-differentiable, execution-based feedback.
  • UI2CodeN: UI2CodeN is an open-source 9B vision-language model trained with continual pretraining, supervised fine-tuning, and RVPO-based reinforcement learning.The same paradigm supports both UI drafting and UI polishing.
  • RVPO: RVPO uses group-wise relative visual preferences instead of unstable absolute rewards to provide a more stable reinforcement-learning signal.The method aligns optimization with comparative visual evaluation despite non-differentiable objectives.
  • Results: UI2CodeN achieves state-of-the-art performance on UI-to-code generation, UI polishing, and UI editing benchmarks, outperforming substantially larger open- and closed-source models.The interactive visual optimization paradigm and relative optimization strategy are described as model-agnostic and applicable to executable generation with black-box feedback.

2. Method

UI-to-code is formulated as a closed-loop optimization process in which executable code is rendered, visually evaluated, and iteratively refined rather than generated once.

  • Interactive formulation: Interactive visual optimization repeatedly transforms target-image and current-code inputs using rendered feedback, while conventional methods apply the transformation only once.The formulation covers drafting, refinement, and editing as related optimization processes.
  • Interactive formulation: Visual optimization improves executable code toward a target UI through black-box rendering and implicit visual comparison without assuming differentiability or explicit gradients.The objective is accessed through execution and visual comparison rather than token-level gradients.
  • UI Drafting: UI drafting provides a first-pass approximation that captures global layout and major structures but remains a cold start because rendering-dependent discrepancies persist.Drafting does not resolve all discrepancies arising from runtime rendering behavior.
  • UI Polishing (Visual Refinement): UI polishing compares rendered output with the target and progressively reduces misalignment, spacing, and style errors through feedback-driven policy improvement.Larger iteration budgets can increase visual fidelity through test-time scaling.
  • UI Editing: UI editing is conditional refinement guided by explicit modification instructions, with this work limited to localized, well-specified UI changes rather than general instruction-driven design.Examples include alignment, typography, color, and component changes.
  • Relative Visual Policy Optimization: RVPO replaces unstable absolute visual rewards with relative preferences among rendered candidates, using pairwise judgments and tournament aggregation for policy optimization.Failed renders receive zero reward, while successfully rendered candidates receive rewards based on aggregate comparisons; pairwise comparison accounts for 2.2% of iteration wall-clock time.

3. Training Pipeline

The training pipeline progressively aligns UI perception, code reasoning, and visual optimization through continual pre-training, supervised fine-tuning, and reinforcement learning.

  • Pipeline overview: The three-stage pipeline combines continual pre-training for vision–code grounding, supervised fine-tuning for refinement behavior, and reinforcement learning for direct visual optimization.The stages progressively align perception, reasoning, and optimization.
  • Continual Pre-training: Continual pre-training grounds UI images in DOM structures through code prediction, combining localized bounding-box grounding with global image–code likelihood.The localized objective predicts code associated with a DOM node, while the global objective preserves image–code coherence.
  • Supervised Fine-Tuning: Supervised fine-tuning teaches drafting, polishing, and instruction-conditioned editing using outputs that separate intermediate visual diagnosis from executable code.The structured output uses <think> T </think><answer> C′ </answer>, where T captures diagnosis and C′ contains code.
  • Reinforcement Learning: Reinforcement learning applies RVPO after supervised fine-tuning to optimize visual alignment, stabilize refinement behavior, and mitigate reward hacking with evaluation-disjoint data.The reinforcement-learning data and reward signals are disjoint from evaluation benchmarks.

4. Experiments

UI2CodeN is evaluated across UI drafting, polishing, editing, reward design, test-time refinement, agent comparisons, and evaluator reliability. Results show strong benchmark performance, gains from iterative refinement and RVPO, and better alignment of VLM-based evaluation with human preferences.

  • Results with VLM Scoring: UI2CodeN consistently outperforms open-source models across UI drafting tasks and reaches 94.0% and 80.0% polishing accuracy on synthetic and real-world benchmarks, respectively.The model remains competitive with leading closed-source systems, with particularly strong gains on structurally complex UI2Code-Real pages.
  • Results with CLIP Scoring: CLIP similarity can remain comparable despite differences in block and position accuracy, indicating that CLIP alone is insufficient for structural and functional UI evaluation.The study therefore motivates VLM-based evaluators, which provide judgments of visual fidelity, layout correctness, and overall UI quality under executable rendering.
  • Interactive Visual Optimization with UI Polishing: Performance consistently improves with more refinement rounds, saturating at N = 3 on synthetic data but continuing to improve through N = 5 on UI2Code-Real.The differing saturation points reflect the higher structural complexity of real-world webpages and support executable feedback as a source of test-time scaling.
  • Comparison with Agent-based Systems: UI2CodeN-RL achieves higher VLM-judge accuracy than agent-based baselines while requiring substantially lower latency and token cost.The comparison is against systems that use multiple detection, planning, and generation stages, whereas UI2CodeN performs end-to-end generation and refinement.
  • Reward Design for UI Polishing: RVPO consistently outperforms SFT and vanilla-verifier reinforcement learning for UI polishing, while CLIP rewards degrade drafting performance relative to SFT.The reward-design ablation isolates reward formulation using the same starting checkpoint and RL configuration; VLM-based absolute rewards perform differently from CLIP-based rewards.
  • Reward Hacking Analysis: Absolute-positioned elements decrease from 0.7% for UI2CodeN-SFT to 0.5% for UI2CodeN-RL, suggesting gains do not rely on trivial positioning shortcuts.The reward-hacking analysis links the result to improved rendered UI quality while preserving structurally reasonable code.
  • Impact of Training Stages: The training stages progressively improve accuracy from image–code grounding through SFT to 88.6% with RVPO.SFT reaches 79.3%, while reinforcement learning adds executable visual feedback beyond supervised learning.
  • Human–VLM Alignment Analysis: VLM pairwise comparisons achieve Pearson 0.93 and Spearman 1.0 correlation with human judgments and reproduce the same baseline ranking.The alignment study uses 80 Design2Code-HARD samples and evaluates both model-level decision consistency and sample-level score calibration.

5. Related Work

Related work advances UI-to-code through benchmarks, datasets, and specialized models. Existing benchmarks increasingly incorporate realistic or functional evaluation, while prior models remain limited by synthetic data, generalization, or incomplete code generation.

  • UI-to-Code Benchmarks: Design2Code established a large-scale real-world UI-to-code benchmark, while Web2Code and Flame-React refined data pipelines but relied heavily on LLM-synthesized HTML.WebGen-Bench further evaluates functional website generation through automated agents that test interactivity and execution behavior.
  • UI-to-Code Datasets: UI-to-code datasets progressed from synthetic screenshot–code pairs toward larger collections sourced from real-world webpages and mixed curated resources.Examples include WebSight, Web2Code, WebCode2M, and Vision2UI.
  • UI-to-Code Models and Systems: Prior UI-to-code models often struggle with incomplete or non-compilable code, while early synthetic-data systems show limited generalization.Representative systems include Pix2Code, ScreenAI, SightSeer, Flame, and WebCode2M.

6. Conclusion

UI2CodeN formulates UI-to-code as interactive visual optimization under executable feedback and uses RVPO to optimize relative visual preferences among rendered candidates. Its open-source 9B model achieves state-of-the-art results across UI generation, polishing, and editing while outperforming substantially larger models.

  • UI2CodeN achieves state-of-the-art performance on UI-to-code generation, UI polishing, and UI editing, outperforming substantially larger open- and closed-source models.

Impact Statement

UI2CodeN targets automated UI code generation through interactive visual optimization, with potential to reduce manual front-end effort and lower barriers to application creation. Its release is intended for research use because automated UI generation could also enable website cloning or reproduction of protected designs.

  • UI2CodeN uses executable visual feedback to improve the reliability and visual fidelity of generated UI code.
  • The approach may reduce manual front-end development effort and lower barriers to creating visually driven applications.
  • Automated UI generation may be misused to clone existing websites or reproduce protected visual designs.
  • The model is released for research purposes, with users encouraged to respect website ownership, licensing terms, and applicable copyright restrictions.

A. Reward Verifier Implementation

The reward verifier uses a vision-language model to compare rendered UIs against targets, with RVPO relying on relative preferences rather than only absolute scores. Fine-tuning substantially improves agreement with human judgments, while polishing rewards directly test whether refinement improves visual similarity.

  • Reward Verifier: GLM-4.5V provides 0–100 visual-fidelity scores for target and rendered UI images as the reward verifier.
  • Relative Comparison: RVPO compares two candidate renderings relative to the same target because zero-shot VLMs are less reliable for fine-grained relative judgments.
  • Data Curation: The verifier is fine-tuned on 8,500 retained preference triplets from 10,000 samples after filtering ambiguous, low-confidence examples.
  • Evaluation Criteria: Verifier prompts assess rendered UI similarity using layout, color, typography, spacing and alignment, and fine details.
  • Validation of Verifier Reliability: 94% agreement with human judgments exceeds the zero-shot verifier’s 62% and GPT-5’s 85% under the same protocol.
  • UI Polishing: For polishing, the binary reward is 1 when score(A, C) > score(A, B), directly targeting improvement over the initial rendering rather than absolute reconstruction quality.

B.1. Continual Pre-training Details

The implementation uses staged training, visual comparison prompts, and data-governance controls for UI-to-code modeling. Continual pre-training and supervised fine-tuning establish coding capabilities before RVPO, while evaluation spans multiple rendered-UI benchmarks and the model is restricted to research use.

  • B.1. Continual Pre-training Details: Continual pre-training uses about 10M webpage image–HTML pairs plus curated UI–code datasets, with roughly 20M vision–code samples trained overall.
  • Visual Verification: The polishing prompt scores two generated renderings against a reference on style dimensions including layout structure, color fidelity, typography, spacing ratios, and element details.
  • B.1. Continual Pre-training Details: Training mixes localized DOM grounding, global image–code likelihood, and general vision–language tasks to retain broad VLM capabilities.
  • Training Pipeline: The SFT dataset contains approximately 80K samples covering UI drafting, polishing, and instruction-conditioned editing, generated through reverse engineering of complex ground-truth HTML.
  • Training Pipeline: RVPO follows supervised fine-tuning and optimizes executable visual alignment using real-world and synthetic reinforcement-learning examples.
  • Data Governance: Data collection respects crawling policies, excludes login-protected and user-specific content, filters personally identifiable information, and avoids releasing raw webpage content.
  • Intended Use: The model is released exclusively for research and is not intended for sensitive personal data, high-stakes decision making, or commercial deployment without further licensing review.
  • Benchmarks: Evaluation includes Web2Code, Flame-React-Eval, and Design2Code, which assess rendered visual fidelity through image comparison, embedding similarity, or element-level matching.

D.2. Our Proposed Benchmarks

The paper introduces benchmarks targeting realistic UI-to-code generation and iterative UI polishing, addressing limitations of synthetic or heavily pruned webpage datasets.

  • Existing benchmarks mainly use synthetic or heavily pruned HTMLs and do not evaluate UI polishing, motivating these more realistic benchmarks.
  • UI2Code-Real contains 115 real-world webpage screenshots designed to capture the complexity, visual diversity, and noise of in-the-wild webpages.
  • UIPolish-bench evaluates iterative refinement using a reference screenshot, an initial rendering, and HTML/CSS code that must be modified to improve alignment.
  • The evaluation section reports CLIP-based semantic similarity with component metrics for block structure, text, position, and color consistency.

E.2. VLM-based Evaluation

VLM-based evaluation uses automated visual judges to score rendered webpages against references, with separate pairwise protocols for drafting and polishing.

  • VLM judges assign scalar visual-quality scores to rendered webpages, reflecting overall visual fidelity and layout correctness.
  • For UI drafting, o4-mini scores reference–rendering similarity from 0 to 100, where higher values indicate greater visual resemblance.
  • Final drafting accuracy is the proportion of examples with similarity scores at least 80, retaining only sufficiently faithful renderings.
  • For UI polishing, Gemini-2.5-Pro scores both initial and polished renderings against the reference and determines which is closer.
  • Standardized prompts define score ranges, reasoning requirements, and final comparison outputs for drafting and polishing evaluation.

F.1. Human Evaluation

Human and VLM evaluations on Design2Code-HARD show strong agreement in pairwise rankings, sample-level quality judgments, and repeated-run stability.

  • Human Evaluation: Human evaluation averages two annotators’ controlled pairwise comparisons across visual structure, aesthetics, and textual consistency, reporting win, tie, loss, and Win+Tie rates.
  • Human–VLM Agreement: VLM-based judge results closely match human relative rankings and win-or-tie rates on challenging UI-to-code cases.
  • Model-Level Decision Alignment: VLM-based rankings closely match human rankings and preserve the same baseline ordering on Design2Code-HARD, with Pearson correlation 0.93 and Spearman correlation 1.0.The VLM judge produces more conservative preference margins while retaining the preference direction across baselines.
  • Sample-Level Score Calibration: The evaluator achieves 82.3% binary agreement, 98% recall, approximately 0.82 precision, and Cohen’s κ = 0.41 when identifying human-rated high-quality outputs.Samples with VLM scores at least 80 have human scores of 3.80–3.81 versus 2.50–2.80 below 80, with Cohen’s d = 1.32–1.82.
  • Evaluation Stability: Across five independent runs, model rankings remain identical and variance stays low, while UI2CodeN-RL matches GPT-5 comparably with the lowest standard deviation.

G. Additional Experiments

Additional experiments examine reasoning format, metric cross-validation, specialized-model comparisons, polishing dynamics, evaluator robustness, and interactive UI capabilities.

  • Think/Answer Format: Explicit Think-stage generation improves Design2Code performance by +3.1% and supports structural planning before HTML generation.The reported planning includes identifying layout regions and corresponding DOM hierarchy, reducing layout hallucination and improving structural coherence.
  • Specialized Model Comparison: UI2CodeN-RL outperforms WebSight VLM-7B, Design2Code-18B, and UICopilot on WebCode2M-Long using CLIP similarity.
  • UI Polishing Dynamics: Polishing succeeds on 74.5% of lower-quality cases with initial scores below 80 but only 53.1% of cases at least 80, indicating diminishing returns near the capability ceiling.Near convergence, individual oscillations often reflect stylistic variation rather than meaningful functional improvement, although aggregate refinement remains stable.
  • Evaluator Robustness: Reinforcement-learning gains remain consistent across diverse VLM evaluators, supporting evaluator-invariant improvements rather than judge-specific bias.
  • RVPO Efficiency: Pairwise comparison accounts for only 2.2% of RVPO training time because rollouts are reused, comparisons are parallelizable, and tournament aggregation limits comparisons.
  • UI Capabilities: Demonstrations show UI2CodeN reconstructing simple and nested layouts, performing localized edits, and supporting iterative UI design workflows.
Loading 2511.08195v4…