ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents

• Text-only code generation requires verbose prompts for structure and styling, struggles with fine-grained visual details, and diverges from workflows that begin with visual sketches.Source: §1 · p. 2
• Current MLLMs struggle with UI-to-code because they must jointly perform visual understanding, structural layout planning, and domain-specific code synthesis.These difficulties produce recurring perception and planning errors.Source: §1 · p. 2
• ScreenCoder decomposes UI-to-code generation into grounding, planning, and generation, achieving state-of-the-art layout fidelity, structural coherence, and generation quality.The framework assigns visual localization, hierarchical layout construction, and HTML/CSS synthesis to separate agents.Source: §1 · p. 2
• ScreenCoder also serves as a scalable data engine for creating high-quality image-code pairs and improving open-source MLLMs through supervised fine-tuning and reinforcement learning.The framework generates Screen-10K from filtered webpages for model training.Source: §1 · p. 2
• Screen-10K contains 10,000 high-quality image-code pairs filtered from an initial crawl of 50,000 webpages.Source: §1 · p. 2
• ScreenBench contains 1,000 contemporary web designs and is intended to provide a more accurate measure of model performance on real-world tasks.Source: §1 · pp. 2–3

2.1 MULTIMODAL LARGE LANGUAGE MODELS

• Modern MLLMs support complex multimodal tasks but struggle with UI-to-code synthesis because they lack spatial-layout and hierarchical-planning inductive biases.Source: §2.1 · p. 3

2.2 VISUAL-TO-CODE GENERATION

• Prior visual-to-code methods progressed from DSL generation toward HTML/CSS, component recognition, layout understanding, interaction awareness, and multi-page generation.Source: §2.2 · p. 3
• Existing approaches often remain brittle, depend on synthetic data or heuristic segmentation, and lack interpretability for jointly modeling visual semantics, layouts, and coding patterns.Source: §2.2 · p. 3
• ScreenCoder differs by decomposing UI-to-code generation into grounding, planning, and generation while integrating domain-specific priors and scalable realistic image-code data.Source: §2.2 · p. 3

• Perception failures include omitted elements, distorted attributes or text, incorrect colors, and misidentified component types.Source: §3 · p. 3, Figure 2 · p. 4
• Planning failures include misplaced components and flat, hierarchically incoherent layouts that fail to capture nested DOM-like structure.Source: §3 · p. 3, Figure 2 · p. 4
• The task overloads one model with granular perception, spatial planning, structured code synthesis, and front-end domain knowledge.Source: §3 · pp. 3–4
• ScreenCoder separates grounding, planning, and generation so agents can specialize and use domain knowledge at the stage addressing each failure mode.Source: §3 · p. 4

4 METHOD

• ScreenCoder sequentially assigns grounding, planning, and generation to specialized agents that coordinate visual understanding, structural reasoning, and high-fidelity code synthesis.The overall framework is illustrated in Figure 3.Source: §4 · p. 4, Figure 3 · p. 5

4.1 GROUNDING AGENT: OVERCOMING PERCEPTION ERRORS

• The Grounding Agent uses an MLLM to locate and label UI regions, returning bounding boxes that can be queried and manipulated through natural language.Text-guided labels make the grounding stage extensible to new UI concepts.Source: §4.1 · p. 5
• Grounding robustness is improved with class-specific NMS, spatial-prior fallback recovery, and inference of the largest non-overlapping primary content region.The resulting layout dictionary provides the semantic and spatial foundation for planning.Source: §4.1 · p. 5

4.2 PLANNING AGENT: CORRECTING PLANNING ERRORS

• The Planning Agent deterministically maps bounding boxes into a DOM-like layout tree, converting pixel coordinates to responsive percentages and using CSS Grid for nested regions.This trades unconstrained generative flexibility for structural integrity and predictability.Source: §4.2 · pp. 5–6

4.3 GENERATION AGENT: HIGH-FIDELITY CODE SYNTHESIS

• The Generation Agent traverses the layout tree and uses adaptive prompts containing semantic labels, structural context, and optional user instructions to assemble HTML and CSS.Source: §4.3 · p. 6

4.4 PLACEHOLDER MAPPING

• Placeholder mapping extracts visual elements, partitions them by planned regions, solves CIoU-based Hungarian matching, and inserts matched image patches into the generated code.A localized affine transformation corrects minor rendering discrepancies before assignment.Source: §4.4 · p. 6

• 10,000 curated image-code pairs from 50,000 crawled webpages form Screen-10K, reducing the noise and instability of raw web-data training.The dataset supports a two-stage pipeline: supervised fine-tuning followed by reinforcement learning.Source: §5 · pp. 6–7
• SFT uses 9,000 pairs to align visual understanding with code syntax, while the remaining 1,000 pairs support reinforcement learning for higher visual fidelity.Source: §5 · p. 7
• GRPO optimizes the policy to maximize expected reward over the reinforcement-learning dataset, with x as the input image and y as generated code.Source: §5 · p. 7
• The reward is the negative MSE between the source screenshot and the rendered output, so maximizing reward minimizes pixel-level differences.This provides a holistic signal for visual replication.Source: §5 · p. 7

6.1 EXPERIMENTAL SETUP

• ScreenBench contains 1,000 contemporary image-code pairs with complex nested layouts, complementing Design2Code’s 484 older samples and structural limitations.Source: §6.1 · pp. 7–8
• Evaluation combines CLIP similarity with OCR-based measures of block reproduction, textual consistency, spatial alignment, and color similarity.Source: §6.1 · p. 8
• The study compares proprietary and open-source MLLMs alongside specialized UI-to-code methods, using Qwen2.5-VL-32B as ScreenCoder’s base model.Source: §6.1 · p. 8

6.2 MAIN RESULTS

• ScreenCoder’s agentic model achieves state-of-the-art ScreenBench performance with a Block score of 0.755 and surpasses all open-source baselines.Its fine-tuned variant achieves second-best results on many metrics and approaches closed-source systems.Source: §6.2 · p. 8

6.3 QUALITATIVE ANALYSIS

• Figure 4 shows end-to-end MLLMs producing distorted layouts and incorrect component arrangements, whereas ScreenCoder closely matches the target website’s appearance and organization.Source: Figure 4 · p. 8, §6.3 · p. 9

6.4 ABLATION STUDY

• SFT substantially improves all metrics, especially Position, and RL adds incremental gains that produce the final top-performing results.Source: §6.4 · p. 9, Table 2 · p. 9

• Users can adjust the layout tree or re-prompt components without restarting the full pipeline because grounding, planning, and generation are disentangled.Source: §7 · p. 9
• Test-time scaling lets users choose smaller models for faster, lower-fidelity drafts or larger models for high-fidelity production-ready code.High-quality generation takes tens of seconds, which the authors frame as acceptable for workflow acceleration.Source: §7 · p. 9

• ScreenCoder addresses end-to-end model limitations through modular generation and improves MLLMs using supervised fine-tuning and reinforcement learning.Source: §8 · p. 9
• The reported outcome is state-of-the-art visual fidelity and code correctness, with a foundation for future multimodal program synthesis research.Source: §8 · p. 9

• The research followed ethical standards for privacy, consent, and responsible AI throughout data collection and model development.Source: ETHICAL STATEMENTS · p. 10
• The authors report that the study introduces no new ethical risks to the best of their knowledge.Source: ETHICAL STATEMENTS · p. 10

REPRODUCIBILITY STATEMENT

• The paper makes its implementation, visual-mapping algorithm, agent prompts, and training procedure available for reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PAIRWISE COMPARISON OF VISUAL FIDELITY

• In visual-fidelity comparisons, human annotators judged ScreenCoder’s outputs superior to GPT-4o’s in 65% of cases and inferior in 11%.Source: §A.1 · p. 13, Table 3 · p. 13

A.2 WORKFLOW USEFULNESS STUDY

• Developers using ScreenCoder completed tasks 2.2 times faster than the control group, averaging 8.5 versus 18.7 minutes.All ScreenCoder participants finished within the time limit, while two control-group tasks timed out.Source: §A.2 · p. 13, Table 4 · p. 14
• ScreenCoder users achieved an average UI-similarity score of 4.6/5.0 versus 3.4/5.0 for controls, with inter-judge Pearson correlation of 0.78.Source: §A.2 · p. 13, Table 4 · p. 14

B ADDITIONAL QUALITATIVE COMPARISONS

• Additional comparisons span diverse website styles and layout complexities, highlighting fewer spatial, element, and styling failures than end-to-end baselines.The examples are presented as side-by-side comparisons with leading baseline models.Source: B ADDITIONAL QUALITATIVE COMPARISONS · pp. 13–14, Figure 5 · p. 14

C.1 PROMPT TEMPLATE

• Prompt templates used by the agents are presented in Figures 13 and 14.Source: §C.1 · p. 14

• The model is fine-tuned on 9,000 Screen-10K samples during SFT, then trained on the remaining 1,000 samples with GRPO reinforcement learning.Training uses the Llama Factory codebase on 16 NVIDIA A100 GPUs.Source: §C.2 · p. 14
• The SFT stage uses AdamW with a cosine learning-rate schedule, peak learning rate 2 × 10^-5, weight decay 0.01, and 10% warmup.Source: §C.2 · p. 14
• The appendix supplies qualitative comparisons across additional baseline figures, extending the visual evaluation of the method.Source: Figure 6 · p. 15, Figure 7 · p. 15, Figure 8 · p. 16
• The reinforcement-learning reward is based on negative mean squared error between rendered output and the ground-truth image, directly optimizing visual fidelity.Source: §C.2 · pp. 14, 18