PaperlayerView exact revision on arXiv

Source-linked AI summary

Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping

Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, Michael R. Lyu

arXiv:2411.03292v3cs.SEcs.AIcs.HC

TL;DR

Existing UI-code benchmarks focus on static webpage appearance and omit dynamic interaction. This paper formulates Interaction-to-Code, builds a benchmark, and studies MLLM performance; it identifies four limitations and proposes four improvement strategies.

  • Problem

    Existing research focuses on static webpage appearance rather than dynamic interactive properties and functionality.

    Source: §I · p. 1

  • Method

    The paper formulates Interaction-to-Code and builds Interaction2Code, a benchmark with 127 webpages and 374 interactions across 15 topics and 31 interaction types.

    Source: §I · p. 2

  • Results

    The study identifies four MLLM limitations: weaker interaction than full-page generation, ten failure types, poor performance on visually subtle interactions, and insufficient understanding from visual-only descriptions.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors propose interactive-element highlighting, failure-aware prompting, visual-saliency enhancement, and combining visual and textual descriptions to address the identified limitations.

    Source: §VII · p. 11

    The benchmark omits back-end code, so the authors cannot verify generated effects for server-side interactions such as login or search.

    Source: §VI · p. 11

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages. To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failureaware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs' performance on the Interaction-toCode task. The Interaction2Code benchmark and code are available in https://github. com/WebPAI/Interaction2Code.

I. INTRODUCTION

Existing design-to-code research evaluates static webpage appearance, while dynamic interaction remains outside its scope. The paper formulates Interaction-to-Code, builds a benchmark, identifies four MLLM limitations, and proposes four improvement strategies.

I. INTRODUCTION

  • Translating GUI designs into functional code is complex and time-consuming, and implementations can differ from their designs.The paper frames GUI design and development as important to website aesthetics and functionality.Source: §I · p. 1
  • Existing research focuses on static webpage appearance, although interactive elements account for more than 50% of cases in a sample of 10 real-world webpages.The omitted behaviors include controls such as size selection and quantity adjustment.Source: §I · p. 1
  • Interaction2Code formulates the Interaction-to-Code task and provides a benchmark of 127 webpages, 374 interactions, 15 webpage topics, and 31 interaction types.The benchmark also includes failure annotations for MLLM-generated webpages.Source: §I · p. 2
  • The study identifies four MLLM limitations: weaker interaction than full-page generation, ten failure types, poor performance on visually subtle interactions, and insufficient understanding from visual-only descriptions.These findings concern interaction generation rather than only static webpage appearance.Source: §I · p. 2
  • The proposed improvements are interactive-element highlighting, failure-aware prompting, visual-saliency enhancement, and combining visual with textual descriptions.The strategies respectively target interaction focus, potential failures, perception of the interaction area, and interaction understanding.Source: §I · p. 2

A. Related Work

Prior UI-code-generation work addresses static interfaces, while Interaction-to-Code targets interactive behavior. Interaction2Code defines that task using interactive prototypes and collects real-world webpages with annotated interactions.

A. Related Work

  • Prior UI-code-generation approaches include deep-learning, computer-vision, and MLLM methods, but the cited work does not focus on generating interactive webpages.Examples include CNN-based GUI generation, sketch or OCR-based methods, and MLLM-based UI generation.Source: A. Related Work · p. 2

B. Problem Definition

  • UI2Code maps a static UI-Mockup to a static webpage, whereas Interaction-to-Code takes an interactive prototype and generates an interactive webpage.An interactive prototype represents an interaction with the original and post-interaction UI-Mockups.Source: B. Problem Definition · p. 2

A. Dataset Collection

  • The benchmark collects webpages from CommonCrawl and GitHub, then retains 1–10 meaningful, functional interactions per webpage and annotates topics, frameworks, and interaction types.The four PhD student annotators selected complex interactions and captured screenshots before and after each interaction; the collection contains 374 interactions.Source: A. Dataset Collection · pp. 2–3

B. Data Statistics and Diversity

Interaction2Code spans diverse webpage topics, frameworks, and interaction categories. Its evaluation automates interactions and assesses both full-page quality and interaction implementation, position, and usability.

B. Data Statistics and Diversity

  • Interaction2Code covers more than 15 webpage topic types and mainstream front-end frameworks including React, Next.js, Vue, and Angular.The distribution is shown separately for topics and frameworks.Source: B. Data Statistics and Diversity · p. 3
  • The benchmark includes 23 tag categories and 8 visual categories, and one interaction may belong to multiple categories.Tag categories are based on HTML elements, while visual categories include changes such as color, size, position, and text.Source: B. Data Statistics and Diversity · p. 3

C. Evaluation

  • WebDriver locates generated interactive elements by ID and performs interactions; affected regions are extracted from before-and-after screenshots.Pixel-wise subtraction is used when dimensions stay fixed; a diff tool is used when dimensions change.Source: C. Evaluation · pp. 3–4
  • Evaluation measures full-page visual, structural, text, and widget similarity, alongside interaction position similarity, implementation rate, and human-judged usability rate.An interaction counts as implemented when WebDriver can detect it.Source: C. Evaluation · p. 4

IV. STUDY SETUP

The study evaluates three commercial MLLMs and three Qwen2.5-VL model sizes. Its prompt designs include direct prompting, interaction-focused CoT and marking, and failure-aware prompting.

A. Evaluation Models

  • The evaluation uses Gemini-1.5-flash, GPT-4o-20240806, Claude-3.5-Sonnet-20240620, and Qwen2.5-VL-Instruct at 3B, 7B, and 72B.The authors selected Qwen2.5-VL because many open-source MLLMs do not support multiple image inputs.Source: A. Evaluation Models · p. 4

B. Prompt Design

  • The direct prompt supplies the original webpage screenshot and screenshots after specified interactions, with interaction IDs supporting automated testing.The numbered interaction IDs allow WebDriver to locate elements in generated code.Source: B. Prompt Design · p. 4
  • For interaction-focused prompting, CoT asks models to analyze effects, locate elements, and implement interactions; Mark uses red boxes around interaction areas.The study also prompts models with failure types to help them avoid corresponding failures.Source: B. Prompt Design · p. 4

C. Data Contamination Check

The contamination check suggests models generate from interactive prototypes rather than copying original webpage code. Automatic evaluation finds stronger overall performance from GPT-4o and Claude-3.5-Sonnet, but weaker results on interactive content; interaction-focused prompts improve several metrics.

C. Data Contamination Check

  • Low BLEU between generated and original HTML suggests models generate from interactive prototypes rather than copying original webpage code.The authors argue test pages were unlikely to be seen during training because C4 contains only text and current MLLM training datasets lack interactive prototyping data.Source: C. Data Contamination Check · pp. 4–5, Table III · p. 5

1) Automatic Evaluation:

  • GPT-4o and Claude-3.5-Sonnet outperform other models on average, while interaction-part performance trails full-page performance.Among open-source models, Qwen2.5-vl-72B performs best, comparable to Gemini-1.5-flash; performance improves with model size.Source: 1) Automatic Evaluation · p. 5, Table IV · p. 6
  • For Gemini-1.5-flash, Mark prompting raises interaction CLIP, SSIM, text, position, and IR scores to 0.5194, 0.3898, 0.3454, 0.4612, and 0.7326 from direct prompting’s 0.4737, 0.3616, 0.2809, 0.4302, and 0.6738.CoT prompting also improves all five metrics, but Mark performs better; widget similarity remains consistently poor.Source: 1) Automatic Evaluation · p. 5, Table IV · p. 6
  • Webpage scale and complexity challenge widget-detection algorithms, making webpage-suitable widget-level consistency detection necessary.The analysis reports significant webpage-element omissions in GUIPilot’s output.Source: 1) Automatic Evaluation · pp. 5–6

2) Human Evaluation:

Human evaluation finds that GPT-4o and Claude-3.5-Sonnet generally outperform baselines, and Mark prompting yields the largest reported usability gains. Failure analysis identifies ten types of MLLM errors in generated interactions.

2) Human Evaluation:

  • Human pairwise comparisons use Gemini-1.5-flash with Direct prompting as the baseline, counting a win or loss only with at least three of five votes.All other vote outcomes count as ties.Source: 2) Human Evaluation · p. 6
  • Claude-3.5-Sonnet’s Mark prompt raises usability by 11% over Direct and 8% over CoT, while GPT-4o and Claude-3.5-Sonnet consistently outperform other baseline models.CoT and Mark generally improve win and usability rates over direct prompting, except Qwen-vl-7B-instruct’s CoT prompt.Source: 2) Human Evaluation · p. 6

B. Failure Type Analysis

  • Annotators identified 10 types of MLLM failure across a random 25% sample of interactions, refining the categories by consensus.When a new failure type arose during annotation, annotators updated the failure types to guide subsequent reviews.Source: B. Failure Type Analysis · p. 7, Table V · p. 7

1) Failure on Interactive Elements:

The analysis groups interaction-generation failures into recurring spatial, widget-type, and behavioral problems, and evaluates failure-aware prompting as a response. Failures can make generated interactions unusable.

1) Failure on Interactive Elements:

  • Interactive-element failures include omitted controls, wrong functions or widget types, and incorrect control placement.Examples include a missing chat button, a price field rendered as a progress bar, and a button placed below the image instead of upper-right.Source: 1) Failure on Interactive Elements · p. 7

2) Failure on Interactive Effects:

  • Interaction-effect failures also include misplaced pop-ups, effects applied to the wrong element, and incomplete or incorrect behavior.Examples include changing only a block rather than the page background, or allowing a button to select but not unselect.Source: 2) Failure on Interactive Effects · p. 8
  • The ten failure types cluster into spatial errors, widget-type errors, and behavioral errors in state-change logic.Spatial failures involve locating the linked element; type failures misclassify widgets, while behavioral failures miss correct state changes.Source: 2) Failure on Interactive Effects · p. 8
  • The most frequent failure modes are no interaction, partial implementation, missing interactive elements, and wrong function.The distribution analysis identifies these as the main failure modes.Source: 2) Failure on Interactive Effects · p. 8
  • Failures (a), (b), and (j) each have usability rates below 10%, making their generated interactions completely ineffective.The paper uses usability rate (UR) to indicate failure severity; lower UR means greater severity.Source: 2) Failure on Interactive Effects · p. 8
  • Failure-aware prompting improves Interaction-to-Code performance across all tested models.The prompt incorporates failure examples and tells models to avoid those failure types.Source: 2) Failure on Interactive Effects · p. 8

C. The Impact of Interaction Visual Saliency

The section relates interaction-area visual saliency to generation outcomes and tests magnifying interaction regions. Higher saliency aligns with stronger structural and positional results, while full-page similarity remains stable under magnification.

C. The Impact of Interaction Visual Saliency

D. The Impact of Different Modalities

The modality study compares visual-only, text-only, and combined inputs for interaction generation. Combined visual-textual input performs better than either single modality across prompt types.

D. The Impact of Different Modalities

E. Ablation Study

The ablation evaluates combinations of four enhancement strategies, and a small developer study compares use of the Interaction2Code tool with LLMs alone. The tool group shows reported gains in efficiency and implementation quality.

E. Ablation Study

  • Combining Interactive Element Highlighting, Failure-aware Prompt, Visual Saliency Enhancement, and Textual Description achieves the best ablation performance; removing any one lowers performance.Source: E. Ablation Study · pp. 10–11, Table IX · p. 11

F. Human Evaluation of Interaction2Code Tool

  • Two additional front-end experts rated implementations on a 1-to-5 scale, while the study also recorded task-completion time.Source: F. Human Evaluation of… · p. 11
  • Development efficiency improves by over 32.4% and implementation quality by over 16.1% when participants use the Interaction2Code tool.The user study involved four PhD students completing four interaction tasks, with two using the tool and two using LLMs without it.Source: F. Human Evaluation of… · p. 11

VI. THREATS TO VALIDITY

The study identifies context length, model selection, and backend-dependent interactions as threats to validity. It concludes with a first systematic study of MLLMs for interactive webpage generation, including the Interaction2Code task and benchmark.

VI. THREATS TO VALIDITY

  • As webpages grow more complex, interaction-heavy inputs may exceed MLLM context windows, though iterative generation over multiple rounds can mitigate this constraint.The passage gives GPT-4o's context window as 128K tokens.Source: §VI · p. 11
  • The experiments cover three prominent MLLMs but omit models such as Pixtral, whose performance the authors leave for future work.Source: §VI · p. 11
  • The benchmark lacks backend code, so the study cannot verify generated interactions that require server-side scripting, such as login or search.The authors describe this as a limitation for complex functional interactions.Source: §VI · p. 11

VII. CONCLUSION

  • The conclusion presents the first systematic study of MLLMs generating interactive webpages, formulates Interaction-to-Code, and establishes the Interaction2Code benchmark.It also reports four limitations and proposes interactive element highlighting, failure-aware prompting, visual saliency enhancement, and combined visual-textual descriptions.Source: §VII · p. 11
Loading 2411.03292v3…