PaperlayerView exact revision on arXiv

Source-linked AI summary

TaoD2C-Bench: Benchmarking MLLMs for Industrial UI Code Generation Beyond Visual Fidelity

Chengwei Shi, Yunnong Chen, Tingting Zhou, Qiang Lu, Shiyu Yue, Xinyuan Hu, Jianfang Ru, Liuqing Chen

arXiv:2610.10374v1cs.SEcs.AI

TL;DR

Industrial UI code may look visually similar to a design while failing its implementation requirements, and existing evaluation approaches do not adequately assess those requirements in realistic settings. TaoD2C-Bench pairs production designs with expert annotations and evaluates code generation, requirement inference, and requirement realization. Across eight MLLMs, the authors report limitations in requirement-satisfying code generation and show that visual reconstruction ability does not necessarily imply requirement satisfaction.

  • Problem

    Industrial UI code can resemble a design yet miss its component and layout requirements, while evaluation datasets and approaches provide limited evidence about requirement satisfaction.

    Source: §1 · pp. 1–2

  • Method

    TaoD2C-Bench pairs production UI designs with expert layer-level requirements and evaluates code generation, requirement inference, and requirement realization.

    Source: §1 · pp. 2–3

  • Results

    Eight MLLMs show limitations in generating code that satisfies implementation requirements, and visual reconstruction ability does not necessarily imply requirement-satisfying code.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support design-specific evaluation that accounts for requirements varying by design and for multiple valid implementations.

    Source: §1 · p. 3

    A single-reference structural comparison may penalize valid code differences even when multiple implementations satisfy the requirements.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

A key challenge for multimodal large language models (MLLMs) is moving beyond visual recognition to constraint-aware cross-modal reasoning. This involves combining visual cues with information from other modalities to understand elements' relationships under domain-specific rules. This challenge is acutely evident in industrial design-to-code (D2C), which converts user interface (UI) designs into code and requires MLLMs to connect design images with disorganized layer metadata, infer component and layout implementation requirements, and realize them in code under target-library constraints. However, these capabilities remain insufficiently evaluated in realistic industrial settings. To fill this gap, we present TaoD2C-Bench, a benchmark for evaluating MLLMs' ability to generate UI code that satisfies implementation requirements in industrial applications. The TaoD2C dataset consists of 2,861 production designs from 17 commercial platforms with 97,652 expert annotations across four categories: Component, Group, Alignment, and Position. These annotations distinguish required constraints from permitted implementation choices. TaoD2C-Bench defines three tasks: end-to-end UI code generation, requirement inference, and requirement realization. Evaluating eight MLLMs reveals substantial gaps in generating UI code that satisfies implementation requirements, alongside distinct performance profiles in inference and realization. We further show that MLLMs' visual reconstruction ability does not necessarily imply an ability to generate code that meets these requirements. We release TaoD2C to support research on industrial UI code generation.

1 Introduction

Industrial UI code can look visually similar to a design yet violate its implementation requirements, and existing evaluation approaches do not reliably capture those requirements. TaoD2C-Bench pairs industrial designs with expert annotations and evaluates code generation, requirement inference, and requirement realization; experiments reveal limitations across these capabilities.

1 Introduction

  • Visual similarity can coexist with failures to use intended components, properties, grouping, alignment, positioning, or frontend coding standards.Such code can degrade user experience and require manual rework before deployment.Source: §1 · p. 1
  • Existing evaluations compare appearance, favor coding heuristics that may conflict with design requirements, or rely on one reference that can penalize valid alternatives.For example, a lower absolute-positioning ratio may be inappropriate for overlapping elements.Source: §1 · p. 2
  • TaoD2C-Bench evaluates end-to-end code generation, requirement inference, and requirement realization using expert-annotated implementation requirements.Separating inference from realization helps distinguish limitations in those capabilities.Source: §1 · pp. 2–3
  • Eight MLLMs show limitations in generating code that satisfies implementation requirements, with distinct inference and realization profiles; visual reconstruction does not necessarily imply requirement-satisfying code.The findings motivate design-specific evaluation because requirements vary across designs and one design may admit multiple valid implementations.Source: §1 · p. 3
  • TaoD2C pairs 2,861 production designs from 17 commercial platforms with expert layer-level annotations across Component, Group, Alignment, and Position.Required annotations constrain implementations, while optional annotations describe permitted choices.Source: §1 · pp. 2–3

2 Related Work

Prior UI-to-code work has advanced interface reconstruction, while evaluation has broadened beyond visual appearance to code quality and structure. However, assessing whether code satisfies design-specific requirements while allowing multiple valid implementations remains an open challenge.

2 Related Work

  • UI-to-code methods progressed from screenshot reconstruction to synthetic-data and feedback-based MLLMs, decomposition, hierarchical generation, and iterative visual refinement.Recent methods also use multimodal design inputs, including images, metadata, and assets, while production-ready industrial code remains challenging.Source: §2 · p. 3
  • Evaluation spans visual similarity, element attributes and layouts, coding-practice heuristics, and comparison with a reference implementation.Visual metrics include SSIM, CLIP, and LPIPS; WebUIBench adds attribute and spatial-layout evaluation.Source: §2 · p. 3
  • Visual similarity, aggregate coding statistics, and single-reference comparison cannot establish requirement satisfaction while accommodating design-specific choices and valid alternatives.Assessing implementation requirements under multiple valid implementations remains an open challenge.Source: §2 · p. 3

3 TaoD2C-Bench

TaoD2C was constructed from industrial UI designs through collection, filtering, expert annotation, and evaluation-subset selection. It combines diverse production designs with dense, verified requirements and uses shared expert annotations to characterize dataset complexity and evaluate requirement inference and code implementation.

3.1 Dataset Construction

  • Design collection began with approximately 30,000 pages from 17 commercial platforms, spanning consumer-facing and business-facing workflows, before filtering, normalization, and quality checks.The retained designs include rendered images, layer metadata, and references to target component libraries.Source: §3.1 · pp. 3–4
  • Two senior frontend engineers annotated layer requirements using design images, layer trees, and target-library documentation; a third expert iteratively reviewed the annotations.The process took 18 months and distinguished required constraints from optional permitted choices across Component, Group, Alignment, and Position.Source: §3.1 · p. 4

3.2 Dataset Characteristics

  • TaoD2C contains 2,861 production designs from 17 platforms across eight scenarios, averaging 64.2 visible layers per design.The dataset spans complex business forms and tables as well as image-rich consumer interfaces.Source: §3.2 · p. 4
  • Its 97,652 expert annotations combine design images, layer metadata, and target component libraries to specify layer-level implementation requirements and relations.The annotations provide a shared reference for evaluating requirement inference and code that satisfies design-specific requirements.Source: §3.2 · pp. 4–5
  • The three tasks evaluate end-to-end UI code generation, requirement inference, and requirement realization against shared expert annotations.Visual fidelity is assessed separately by comparing rendered pages with design images.Source: Figure 5 · p. 5, §3.2 · p. 5

4 Tasks and Metrics

The benchmark separates end-to-end code generation, requirement inference, and requirement realization using shared designs and expert annotations. It evaluates requirement prediction and code implementation separately from visual fidelity.

T1: End-to-End UI Code Generation.

  • T1 generates executable UI code from a design image, layer metadata, and target-library documentation, assessing requirement satisfaction and visual fidelity separately.Requirement satisfaction is checked through source-code and runtime analysis.Source: T1: End-to-End UI Code Generation · p. 5

T2: Requirement Inference.

  • T2 predicts structured Component, Group, Alignment, and Position requirements from the same inputs as T1, without generating code.Predictions are compared with expert annotations to isolate requirement inference from implementation.Source: T2: Requirement Inference · p. 6

T3: Requirement Realization.

  • T3 supplies expert annotations alongside T1 inputs and evaluates whether generated executable code realizes those requirements.Visual fidelity is evaluated using the same metrics as T1.Source: T3: Requirement Realization · p. 6

4.2 Metrics

  • Requirement prediction uses layer-level correspondences and micro-F1 for Component, Group, Alignment, and Position; code implementation uses source, structural, and geometric evidence without reference code.Code implementation reports micro-F1 for Type, Props, Group, Alignment, and Position; Props is scored only on type-matched components.Source: §4.2 · p. 6
  • Visual fidelity compares rendered T1/T3 pages with design images using CV, DINOv2-based VES, and normalized RGB MAE.Higher CV/VES and lower MAE indicate better visual fidelity.Source: §4.2 · p. 6

5 Experiments

Across eight MLLMs, end-to-end code generation remains weak on implementation requirements, especially alignment and positioning. Inference and realization have distinct performance profiles, and visual reconstruction does not necessarily imply requirement-compliant code.

5.1 Setup

  • The evaluation covered eight MLLMs on 421 reviewed, deduplicated designs: 152 B-end and 269 C-end.All models received the same design context and task prompts.Source: §5.1 · p. 6

5.2 Results

  • 23.30 mean F1 for T1 alignment was the lowest across all eight MLLMs; best T1 scores were 63.25 for component type, 44.25 for properties, and 59.46 for grouping.Positioning averaged 32.22 F1, with a best score of 45.35.Source: §5.2 · p. 7, Table 2 · p. 7
  • In T2, mean F1 was 29.94 for Alignment and 45.98 for Position, below Component (58.01) and Group (56.91); T3 alignment remained lowest at 41.13 F1.The findings point to difficulty reasoning about element relationships and translating spatial constraints into layout code.Source: §5.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7
  • Inference and realization leaders differ: GPT-5.6 Terra led alignment inference at 37.11 F1, while Gemini 3.1 Pro led realization at 57.82.For grouping, Qwen3.8-Max led inference at 71.40 F1, while Gemini 3.1 Pro led realization at 71.34.Source: §5.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7
  • From T1 to T3, code-dimension mean gains ranged from 6.82 to 24.24 F1 points, while mean visual scores changed little; Figure 6 shows errors despite similar renders.Mean visual-score changes were CV +0.18, VES −0.0039, and MAE −0.0033.Source: Table 3 · p. 7, §5.2 · p. 8, Figure 6 · p. 8
  • Implementation requirements depend on design context, and one design can admit multiple valid implementations that single-reference comparison may penalize.The evaluation includes optional positioning, alignment, and grouping annotations to record permitted choices.Source: §5.2 · p. 8

6 Conclusion

TaoD2C-Bench evaluates generated UI code against industrial designs’ layer-level expert requirements. Its three tasks expose limitations in requirement inference and code implementation, with distinct strengths across models and tasks.

6 Conclusion

  • TaoD2C-Bench pairs industrial UI designs with layer-level expert requirements to evaluate generated code.Source: §6 · p. 8
  • Across its three tasks, the benchmark reveals limited capabilities in both inferring requirements and implementing them in code.Source: §6 · p. 8
  • The findings show distinct strengths across MLLMs and tasks and provide insights for future UI code generation research.Source: §6 · p. 8

A Dataset and Annotation

TaoD2C combines production UI designs from commercial platforms with expert annotations of their implementation requirements. Its construction covers varied application settings, structural complexity, layer-linked annotation, normalized metadata, and expert review.

A.1 Industrial Realism and Application Diversity

  • The dataset contains 2,861 production designs from 17 commercial platforms, spanning varied consumer- and business-facing scenarios.Figure 7 summarizes eight primary application scenarios across the designs.Source: §A.1 · p. 14, Figure 7 · p. 14

A.2 Structural Complexity and Annotation Coverage

  • Across all 2,861 designs, the median is 43 visible layers and 23 annotations per design, with 90th percentiles of 146 and 81, respectively.These statistics describe the full dataset, not only the evaluation subset.Source: §A.2 · p. 14, Table 5 · p. 14

A.3 Annotation Platform

  • Experts use a platform linking the original design, rendered image, and parsed layer tree to annotate Component, Group, Alignment, and Position requirements as required or optional.Acceptable alternative component types can also be recorded.Source: §A.3 · p. 15, Figure 8 · p. 15

A.4 Layer Parsing and Normalization

  • Layer parsing retains node identifiers, geometry, names, component references, text, styles, and assets, while removing non-public links and redundant artboard images.Nodes with non-positive area remain in metadata but are excluded from member derivation and scoring; URL deduplication reduces 2,882 records to 2,861 designs.Source: §A.4 · p. 15

A.5 Quality Control

  • Two senior frontend engineers annotated the designs, and a third expert reviewed every sample over multiple rounds before errors were revised and rechecked.The annotation and verification process spanned 18 months.Source: §A.5 · p. 15

A.6 The Evaluation Subset

The evaluation uses a frozen 421-design subset and a shared required/optional scoring reference. Three tasks vary whether models generate code, infer requirements, or implement supplied requirements, with shared inputs and defined scoring rules.

A.6 The Evaluation Subset

  • The evaluation subset contains 421 designs from five collections, including 152 B-end and 269 C-end designs, and was frozen before inference without content- or complexity-based filtering.Six duplicates and seven withdrawn designs were removed.Source: §A.6 · p. 15

A.7 Scoring Reference Construction

  • The shared scoring reference links each required or optional annotation to its design region and layers across Component, Group, Alignment, and Position.T1 and T3 code are evaluated against the reference, while T2 predictions are evaluated against it; T3 also receives the expert annotations.Source: §A.7 · pp. 15–16, Table 6 · p. 16

B.1 Shared Inputs and Prompt Design

  • All tasks receive design images, layer metadata, and target-library documentation, while T3 additionally receives the design’s expert annotations.Each task uses the same prompt template across MLLMs.Source: §B.1 · p. 17

B.2 T1: End-to-End UI Code Generation

  • T1 generates a React page in TypeScript and CSS Modules from the shared design inputs.Source: §B.2 · p. 17

B.3 T2: Requirement Inference

  • T2 predicts structured implementation requirements from the shared design inputs without generating code.Source: §B.3 · p. 18

B.4 T3: Requirement Realization

  • T3 implements supplied expert requirements while using original design inputs for text, images, geometry, and visual styles; its code output and evaluation match T1.Source: §B.4 · p. 19

B.5 Output Formats

  • T1 and T3 return three marked source files, whereas T2 returns one JSON object containing four arrays.The figure’s examples are schematic; T2 responses must contain valid JSON, and T3’s supplied requirements use the same public schema as T2 output.Source: §B.5 · p. 20, Figure 12 · p. 20

C Evaluation Details

  • Requirement Prediction scores T2’s structured requirements, while Code Implementation scores their realization in code and runtime DOM/CSS, reporting five F1 scores rather than four.Both use micro-averaged precision, recall, and F1, with F1 reported in the tables.Source: C Evaluation Details · p. 21

C.2 Code Implementation Scorer

Code implementation is scored by matching expert annotations to generated-code and rendered-DOM evidence. Visual fidelity is reported separately, and results use final outputs with task-specific failure handling and aggregation.

C.2 Code Implementation Scorer

  • Implementation scoring matches expert annotations to generated-code and rendered-DOM evidence from pages bundled with esbuild and rendered in Playwright.Runtime evidence includes bounding boxes, parent-child relations, computed styles, and component types and properties associated with visual roots.Source: §C.2 · p. 21
  • Component Type matching requires target-library-compatible names and both IoU and area similarity of at least 0.7; Props are checked only after type matches.Matching is one-to-one, and plain containers and image types are excluded from component evaluation.Source: §C.2 · p. 21
  • Group requires IoU and area similarity of at least 0.7, Alignment also requires the annotated type, and Position checks positioning mode at IoU of at least 0.7.Runtime boxes are calibrated by the median anchor offset before geometric tests, removing global page shift without changing relative geometry.Source: §C.2 · p. 21

C.3 Visual Fidelity

  • CV, VES, and MAE compare rendered outputs with designs and are reported separately rather than combined into an overall visual score.CV weights perceptual image similarity, element completeness, and layout similarity by 0.5, 0.3, and 0.2, respectively.Source: §C.3 · p. 22
  • VES compares DINOv2-base CLS-token embeddings, while CV and VES increase with fidelity and MAE indicates better fidelity when lower.MAE preprocessing crops near-white borders, resizes the render proportionally, and center-pads it with white.Source: §C.3 · p. 22

C.4 Aggregation and Failure Handling

  • All eight models use the same 421 designs for three tasks, producing 10,104 outcomes; results use the final output after retries and repairs.There are 10,101 successful outcomes and three failures, all for Qwen3-VL-235B on one B-end design that exceeded the provider’s token limit; failures receive zero F1, CV, and VES and MAE of one where applicable.Source: §C.4 · p. 22
  • Requirement scores pool counts across designs, whereas CV, VES, and MAE average across designs; reported model means and T3–T1 differences use metric-specific precision.F1 and CV use two decimals, while VES and MAE use four.Source: §C.4 · p. 22

D Experimental Settings

The experiments use shared design inputs and task templates across models, with model-specific inference settings. Output validation, retry, rendering, and evaluation procedures are specified for each task.

D.1 Models and Generation Settings

  • All models use the same 421 designs and task templates, while GPT-5.6 Terra and GPT-5.4 use Codex CLI and the others use an OpenAI-compatible gateway.Qwen3.8-Max is reported under its canonical name despite using two gateway entries.Source: §D.1 · p. 22

D.2 Execution and Output Validation

  • Requests share design inputs; T3 inserts expert requirements after the layer tree, and experiment manifests record inputs, prompts, component documentation, and references.Supported models use prompt caching without changing task content.Source: §D.2 · p. 22
  • Transient request failures allow up to five retries, timeouts may trigger an identical streaming request, and invalid outputs trigger one repair request; predictions are not manually edited or selected by evaluation score.T2 repair checks JSON parsing and schema compliance; T1/T3 repair checks for required file blocks or files.Source: §D.2 · p. 23
  • Generated pages are rendered at each design’s root width; T2 is evaluated against compiled references, while T3 uses T1’s rendering and evaluation procedure.Failed renders retain an error record.Source: §D.2 · p. 23

E Domain-wise Results

The domain-wise results cover three tasks across B-end and C-end designs, with shared cases across models and task-specific metrics. The public evaluation subset has separate terms for benchmark-created materials and platform-owned design assets.

E Domain-wise Results

  • Tables 8–13 report all three tasks separately for 152 B-end and 269 C-end designs, using the same cases across eight models and retaining terminal failures.F1 is micro-aggregated within each domain; pooled B+C F1 comes from counts rather than averaging domain F1 values, while CV, VES, and MAE are case means.Source: E Domain-wise Results · p. 24, Table 8 · p. 24, Table 9 · p. 24

F License

  • The public evaluation subset’s expert annotations and compiled implementation requirements are licensed CC BY-NC 4.0 for sharing and adaptation for non-commercial purposes with credit and indication of changes.Original platform-owned design files, images, extracted assets, and layer metadata may not be separately redistributed or sublicensed; commercial use requires platform permission.Source: F License · p. 26
Loading 2610.10374v1…