PaperlayerView exact revision on arXiv

Source-linked AI summary

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

Grace Man Chen, Litao Guo, Yifan Wu, Yiyu Chen, Yenchi Tseng, Sicheng Liu, Yuyu Luo, Ying-Cong Chen

arXiv:2607.06306v1cs.SEcs.AIcs.CV

TL;DR

UI2App addresses the lack of systematic evaluation for interaction inference from screenshots by introducing a benchmark and end-to-end protocol for runnable multi-route applications. Across six frontier vision-language models, visual reconstruction and interaction realization diverge sharply, while cross-route state persistence remains a major bottleneck.

  • Problem

    Existing text-driven web-generation approaches struggle to specify detailed layouts and maintain visual coherence across multiple pages.

  • Method

    UI2App evaluates runnable multi-route artifacts from screenshot sets across executability, navigation reachability, visual fidelity, and taxonomy-based interaction inference.

  • Results

    The visual-fidelity leader ranks fourth on IIS, trailing the IIS leader by 5.2×, while three of six models score zero on cross-route persistence.

  • Takeaways & Limitations

    Inferring complete interaction behavior from static screenshots remains a frontier challenge, especially for cross-route state persistence.

  • Takeaways & Limitations

    The evaluation zero-imputes artifacts that fail EXEC@3 for navigation, visual-fidelity, and interaction-inference scores.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.

1 Introduction

UI2App addresses the gap between screenshot-based visual reconstruction and recovering application behavior from screenshots alone. It benchmarks runnable multi-route artifacts across executability, navigation reachability, visual fidelity, and interaction inference, revealing that strong visual reconstruction does not guarantee functional interaction behavior.

  • Motivation: Existing image-to-webpage benchmarks primarily measure visual fidelity, allowing behaviorally inert façades to appear successful.They assess rendered-output similarity rather than whether the artifact works as an application.
  • Interaction inference: IIS measures interaction inference rather than specification-following, accommodating multiple valid implementations because static screenshots underdetermine behavior.The benchmark organizes common web interactions into a taxonomy of seven categories and evaluates them with a rubric-based protocol.
  • UI2App benchmark: 327 screenshots are organized into 45 state-coherent sets for runnable, multi-route web application generation without captions, instructions, or interaction descriptions.Models must reconcile structure, palette, and inferred behavior across multiple routes simultaneously.
  • Results: 39.3 is the highest overall IIS achieved by the six evaluated frontier VLMs, showing that complete interaction inference from static screenshots remains difficult.The benchmark’s four-metric protocol spans EXEC@1 / EXEC@3, NRS, VFS, and IIS.
  • Results: 5.2× is the IIS gap between the visual-fidelity leader and the IIS leader, with the visual-fidelity leader ranking fourth on IIS.Cross-route persistence is especially difficult: three of six models score exactly zero on S3-scope interactions.

2 Related Work

Prior work has progressed from static screenshot-to-code generation toward interaction specified through text or demonstrated visually. UI2App differs by requiring multi-state interaction inference from a single canonical screenshot per route, without a predefined target transition.

  • Static Visual-to-Code: Vision-language models have advanced screenshot-to-code generation, with benchmarks evolving from real-world webpages and visual similarity scoring to synthetic and large-scale corpora.This line of work focuses on translating UI visuals into frontend code.
  • Interaction Specified by Text: WebGen-Bench and ArtifactsBench specify interactive behavior through natural-language instructions and evaluate generated artifacts with automated functional tests or an LLM judge.ArtifactsBench covers web interfaces, visualisations, and mini-games, while WebGen-Bench targets multi-file website codebases.
  • Interaction Demonstrated Visually: Interaction2Code and IWR-Bench provide visual interaction demonstrations through state-pair screenshots or user-interaction videos recording state transitions over time.These benchmarks evaluate interaction using Selenium-driven probes or functional and visual fidelity scoring.
  • Interaction Demonstrated Visually: Unlike prior interaction benchmarks, UI2App provides a single canonical screenshot per route and no target transition, making multi-state inference intrinsic to the task.Earlier benchmarks provide the target interaction trajectory beforehand as a behavioral oracle for evaluation.

3 UI2App

UI2App evaluates whether models can generate runnable React + TypeScript applications from screenshot sets alone, emphasizing both visual reconstruction and inferred interaction behavior. Its benchmark combines curated multi-route applications with automated execution, navigation, and visual checks, plus rubric-based human evaluation of interactions.

  • Task and dataset: Each task provides 4–14 screenshots (mean 7.3) spanning distinct routes or visual states, without action traces or interaction descriptions.Models must generate source code for a runnable application using a fixed React + TypeScript scaffold.
  • Task and dataset: 45 runnable multi-route reference applications were selected from 164 filtered repositories using page discriminability, application complexity, and corpus diversity.The initial pool contained 2,013 repositories from 24 GitHub queries across 12 application categories, then underwent licensing, structural, buildability, and authentication checks.
  • Evaluation protocol: Evaluation proceeds through generation, production build and rendering, and self-debugging, with EXEC@1 measuring one-shot execution and EXEC@3 measuring pass rates after up to three retry rounds.An app passes EXEC when its production build succeeds and its home route renders meaningful content.
  • Evaluation protocol: NRS measures the fraction of screenshot routes reachable through visible navigation, while VFS uses optimal block matching and averages Size, Text, Position, and Color sub-metrics.Human verification supports navigation assessment, and VFS and IIS use direct URL loading; apps failing EXEC@3 receive zero for NRS, VFS, and IIS.
  • Interaction inference: IIS evaluates screenshot-implied interactions by coverage, functional result, and state-logic complexity rather than matching a single reference implementation.The taxonomy includes toggle, expand/collapse, list operations, data CRUD, form validation, notification, and cross-route state; scope weights are wS1 = 1, wS2 = 2, and wS3 = 3.
  • Interaction inference: 315 IIS items per model and 1,890 items across six models are manually annotated, with runtime-based judgments and substantial inter-annotator agreement (Krippendorff α in 0.72–0.84).Three annotators evaluate generated applications, while two front-end engineering experts construct the reference interaction inventory.

4 Experiments

Experiments across six frontier VLMs show that executability and visual fidelity do not reliably translate into interaction inference, with cross-route state and administrative workflows remaining major bottlenecks. Within the Qwen2.5-VL family, usable application generation appears only after a sharp 32B-to-72B phase transition, while build failures become more semantically complex.

  • Experimental setup: Six frontier VLMs were evaluated through official APIs, alongside a Qwen2.5-VL 3B–72B scaling ladder under the same protocol.The ladder reports EXEC@1, EXEC@3, and VFS only; NRS and IIS were omitted because of annotation cost.
  • Overall results: 95.6%: Claude Sonnet 4.6 leads EXEC@1, while Gemini 3.1 Pro Preview leads VFS at 78.1 and Claude leads IIS at 39.3.Gemini and Claude tie at 100% on EXEC@3; GLM-4.6V is lowest on every metric, and IIS shows the greatest divergence.
  • Interaction inference: 78.1 versus 7.5: Gemini leads VFS but ranks fourth on IIS, whereas Claude leads IIS at 39.3, 5.2× ahead of Gemini.Gemini scores exactly 0.0 on CRUD, notification, and cross-route-state row means, demonstrating that pixel-faithful reconstruction can miss interaction logic.
  • Interaction inference: 20.7 and 13.2: Kimi and Qwen3.5 outperform Gemini at 7.5 and GPT-5.4 at 6.7 on IIS, reversing the VFS quality hierarchy.The closed-versus-open ordering observed on VFS therefore does not transfer to interaction inference.
  • Interaction inference: Three of six models score exactly zero at S3 cross-route state, while only Claude meaningfully clears the floor at 21.6.S-restricted IIS generally decreases from S1 to S2 to S3, and between-model spread roughly halves as scope hardens.
  • Interaction inference: 9.4: Admin apps have the lowest six-model mean IIS, roughly half that of Content and Transaction despite Transaction having comparable item counts and more cross-route state.The narrow Admin model spread suggests a shared limitation across current models.
  • Failure analysis: 50%: EXEC@3 failures violate one of four scaffold-respect constraints, while NRS losses arise from mismatched navigation and hallucinated routes.Instruction compliance ranges from 100% for Claude, Gemini, and GPT-5.4 to approximately 53% for GLM-4.6V; IIS zeros concentrate on S2 CRUD and S3 cross-route state.
  • Within-family scaling: 62.2% and 35.2: Qwen2.5-VL EXEC@3 and VFS jump at 72B from at most 2.2% and 0.2 below 32B, yet 37.8% still fail to build.Usable React-app generation emerges between 32B and 72B, while failure categories shift from syntax and inconsistency toward hallucinated dependencies and deeper build-stage errors.

5 Conclusion

UI2App introduces a benchmark for recovering application behavior from image-only multi-page screenshots without textual or demonstration specifications. It evaluates generated artifacts across executability, navigation reachability, visual fidelity, and interaction inference, showing that visual fidelity is a poor proxy for interaction capability.

  • Benchmark scope: UI2App measures interaction inference by recovering application behavior from image-only multi-page screenshots without textual or demonstration specifications.The benchmark targets behavior recovery from screenshots alone.
  • Evaluation dimensions: Each artifact is evaluated on executability, navigation reachability, visual fidelity, and IIS.IIS is grounded in an interaction taxonomy that tiers interactions by state-logic complexity.
  • Main finding: Across six frontier vision-language models, visual fidelity proves a poor proxy for interaction inference.The conclusion links visual quality and interaction capability as distinct evaluation targets.

A Dataset construction details · B IIS taxonomy, and inter-annotator agreement · B.1 Category catalog

UI2App filters candidate repositories for licensed, structurally valid, buildable, authentication-free applications, then captures and deduplicates route screenshots through a controlled browser pipeline. IIS organizes inferred interactions into seven families spanning local UI changes, list and data operations, validation, feedback, and cross-route state.

  • A Dataset construction details: 2,013 candidate repositories undergo four filters: permissive licensing, structural validity, buildability, and authentication-wall detection.The license filter accepts MIT, Apache-2.0, BSD, or ISC; builds must complete within 180 seconds.
  • A Dataset construction details: 164/2,013 ≈8% of repositories survive all four filtering stages and enter expert review.
  • A Dataset construction details: Playwright captures screenshots in headless Chromium at viewport 1440×900 with device_scale_factor=1.
  • A Dataset construction details: The capture pipeline waits for hydration, suppresses development overlays, checks asset readiness, completes animations, and rechecks loading indicators.Hydration settles for 3 seconds; asset readiness has a 6-second timeout, animation completion a 3-second timeout, and loading detection a 2-second recheck.
  • A Dataset construction details: Per-route login-redirect detection complements repository-level authentication filtering, while surviving screenshots are deduplicated using perceptual hashing.Deduplication uses a 32×32 average-hash on the content area with Hamming distance < 6%.
  • B.1 Category catalog: IIS defines seven interaction families: toggle, expand/collapse, list operations, data CRUD, form validation, notification, and cross-route state.Their typical scopes range from S1 for local interactions to S2 for list/data operations and S3 for cross-route state.
  • B.1 Category catalog: Cross-route state is defined as state persistence across navigation and assigned typical scope S3.

B.2 State management and scope design … C Provider-rejected projects

The merged sections define IIS state-scope levels, outcome scoring, annotation procedures, and treatment of a provider-rejected GPT-5.4 project. Together, they specify how interaction complexity, implementation outcomes, agreement, and exceptional runs are operationalized.

  • B.2 State management and scope design: S1 UI-state remains within an immediate rendered subtree, whereas S2 data-state requires collection-level transformations observed by other components.Examples include toggles for S1 and filtering, sorting, grouping, or CRUD for S2.
  • B.2 State management and scope design: S3 cross-route persistence applies when an interaction’s effect must survive navigation, such as a cart remaining non-empty after navigating to /checkout.The scope hierarchy is operationalized as S1, S2, and S3 for IIS classification.
  • B.2 State management and scope design: 1:2:3 weights are assigned to S1, S2, and S3 because implementation cost broadly rises across S1<S2<S3.The scope-weight choice is described as a linear ordinal weighting.
  • B.2 State management and scope design: Kendall τ=1.0 between every weighting pair, and the six-model IIS ranking remains unchanged under 1:1:1, 1:2:3, and 1:2:4 weightings.The 1:2:3 weighting reproduces the headline IIS of Table 2.
  • B.3 Outcome rubric and paired-merge interpolation: W means all implied behavior works, P means some behavior works or one behavior is partial, and F means none of the implied behavior works.These labels map to per-item scores ri ∈{1, 0.5, 0}.
  • B.4 Inter-annotator agreement: 15×{A+B}, 15×{A+C}, and 15×{B+C} paired annotations were collected from three annotators, with arbitration triggered by coverage mismatch, result gap > 1, or scope gap > 1.Bootstrap confidence intervals resampled 45 paired-labeled app-model runs with B=1,000 and seed 42.
  • C Provider-rejected projects: GPT-5.4 had one project rejected by the OpenAI content-safety filter before receiving screenshots, and Table 2 counts it as 0 in EXEC@1, EXEC@3, and VFS.The main-table convention uses zero-imputation with denominator N=45; excluding the rejected project gives EXEC@3 =84.1% (37/44) and VFS=66.4.

D Conditional metrics: VFS⋆and IIS⋆

UI2App separates end-to-end usability from intrinsic capability by reporting zero-imputed headline VFS/IIS scores and build-conditioned VFS⋆/IIS⋆ diagnostics. The headline-to-conditional gap quantifies build-failure contribution, motivating headline scores as the main benchmark measure.

  • Denominator conventions: Headline VFS and IIS average over all N=45 projects with zero-imputation, assigning 0 to runs that fail to build after three self-debug rounds.This jointly penalizes executability and downstream visual or interaction capability, reflecting end-to-end usability.
  • Denominator conventions: Conditional VFS⋆ and IIS⋆ average only over the N⋆≤N projects that pass EXEC@3, isolating visual fidelity and interaction inference from build success.N⋆ is the pass-conditioned project count.
  • Gap interpretation: The headline→conditional gap equals the contribution of build failures to the headline score; models with EXEC@3=100% have ∆=0 by construction.Sonnet 4.6 and Gemini 3.1 Pro Preview are cited as examples with no build failures.
  • Gap interpretation: +37.2 points is GLM-4.6V’s largest reported VFS gap, rising from 22.6 headline to 59.8 conditional across N⋆=16 projects.The example shows substantial conditional visual-language ability alongside strong build-failure effects on the headline score.
  • Choice for the main table: UI2App headlines zero-imputed VFS and IIS in Table 2 because non-building generations are unusable, while VFS⋆ and IIS⋆ serve as diagnostics.The conditional metrics measure faithfulness only on projects the model builds.

E Conditional sub-metric breakdown (VFS⋆decomposition) · F VFS algorithm: DOM-alignment matching and cascade fallback

The conditional VFS⋆ breakdown shows that build success explains most cross-model visual-score variation, while the VFS pipeline uses judge-free rendering, DOM/OCR extraction, global route matching, and cascade fallbacks to evaluate generated applications.

  • E Conditional sub-metric breakdown (VFS⋆decomposition): Gemini 3.1 Pro Preview leads all four zero-imputation sub-metrics, whereas GLM-4.6V is lowest on all four because of 35.6% buildability.The four columns—Size, Text, Position, and Color—average to the headline VFS exactly.
  • E Conditional sub-metric breakdown (VFS⋆decomposition): 58.6 to 9.4 points: controlling for buildability narrows the Position spread, while Color shrinks from 47.9 to 11.9 points.These reductions indicate that visual sub-skill differences are much smaller than overall VFS differences, with most variation arising from build success.
  • E Conditional sub-metric breakdown (VFS⋆decomposition): GLM-4.6V’s conditional results suggest preferential representational capacity for color over geometry across its 76 pages from 16 build-pass apps.This conclusion is supported by cross-validation across the four sub-metrics within those apps.
  • F VFS algorithm: DOM-alignment matching and cascade fallback: VFS is fully judge-free: it invokes no VLM, LLM, or pixel-classifier and evaluates a reference application and generated application on the same input set.The pipeline operates on reference screenshots and routes paired as (s_i, r_i).
  • F VFS algorithm: DOM-alignment matching and cascade fallback: The pipeline extracts generated routes, boots a headless Chromium server, renders candidate routes, extracts reference DOM blocks or OCR, and globally matches screenshots to routes.Reference extraction falls back to OCR when routes are unreachable because of authentication, missing dependencies, or related failures.
  • F VFS algorithm: DOM-alignment matching and cascade fallback: The scoring pipeline deduplicates repeated route signatures, applies render-validity probes and gate handling, and penalizes incomplete coverage through nmatched/ntot.For example, 1/12 matched at Lmatched^3=31.7 becomes VFS=2.6, while EXEC@3=F applications receive zero-imputation and VFS⋆ averages only EXEC@3=P applications.
  • F VFS algorithm: DOM-alignment matching and cascade fallback: β=0.08 provides a path-token tie-breaker, and Hungarian assignment solves the global matching problem without overriding strong DOM evidence.The token bonus applies when DOM-alignment scores are within approximately 0.05; the default matcher is retained unless reference startup fails.
  • F VFS algorithm: DOM-alignment matching and cascade fallback: Rendered routes use a 1440×900 viewport, 4,000 ms reference idle time, and 3,000 ms generated idle time before TreeWalker-based DOM block extraction.Extracted blocks include normalized text, bounding boxes, and visual attributes; aligned assignments are retained only when ρ≥θtext=0.3.

F.1 Cascade fallback (T1–T b 5, used only when Sref fails to start) · G Failure taxonomy details

When the reference server cannot start, evaluation uses a one-to-one path-name and visual cascade with progressively weaker matching tiers. The appendix also decomposes 52 unrecovered EXEC@3 failures into a nine-mode taxonomy and examines their causes across models.

  • F.1 Cascade fallback (T1–T b 5, used only when Sref fails to start): Reference-server failures trigger a path-name and visual cascade because DOM alignment is undefined, with each input matched to at most one generated route.This fallback applies to missing node_modules, installation timeouts, or port collisions under heavy parallelism; Kmax=1 and successful tiers skip later tiers.
  • F.1 Cascade fallback (T1–T b 5, used only when Sref fails to start): T1–T4 match routes by exact names, dynamic-route regexes, shared suffixes, or synonym-structure overlap.Examples include /product/1 versus /product/:id, /dashboard versus /app/dashboard, and fixed synonym groups such as home/landing/index/overview.
  • F.1 Cascade fallback (T1–T b 5, used only when Sref fails to start): T5 visual-first matching builds a 16×16 perceptual-hash similarity matrix, solves Hungarian assignment on 1 − sim, and accepts pairs with sim≥θphash=0.45.The method operates on input PNGs and generated screenshots.
  • F.1 Cascade fallback (T1–T b 5, used only when Sref fails to start): Legacy visual fallback combines 0.7 SSIM and 0.3 HistCorr, requiring both a relative margin best≥1.2¯µ and an absolute floor θabs vis=0.55.The comparison uses 256×160 resized RGB images and considers only inputs unmatched by earlier tiers.
  • F.1 Cascade fallback (T1–T b 5, used only when Sref fails to start): DOM evidence remains robust to route renaming and rejects path-name false positives, while the token-Jaccard bonus β=0.08 primarily resolves noisy ties.DOM alignment can pair a renamed route serving Blog content with /blog and prevent a music landing from matching an unrelated path; DOM evidence dominates when decisive.
  • G Failure taxonomy details: 52 unrecovered EXEC@3 build/runtime failures are decomposed into a nine-mode taxonomy spanning the benchmark’s four scoring axes.The breakdown includes per-model incidence, prompt-clause attribution, and representative code anchors.
  • G Failure taxonomy details: The failure analysis traces how the dominant failure mode changes along the Qwen2.5-VL size ladder and specifies the fixed self-debug feedback protocol used for EXEC@3.It also examines recurring patterns that reduce performance, although the supplied passage truncates before listing them.

G.1 Generation-level failures · G.2 Per-size failure-mode shifts on the Qwen2.5-VL ladder · G.3 Self-debug feedback protocol

The paper attributes 52 unrecovered EXEC@3 failures to five high-level causes, while failure categories shift across Qwen2.5-VL sizes from syntax to inconsistency and mixed syntax-plus-hallucination regimes. EXEC@3 measures repair ability under a fixed three-round Self-Debug protocol that returns toolchain diagnostics to the model.

  • G.1 Generation-level failures: 52 unrecovered EXEC@3 failures decompose into nine fine-grained modes and five high-level causes: hallucinated imports, scaffold overwrites, undeclared dependencies, broken paths, and parse/runtime errors.The causes group C1/C6, C2/C8, C3, C4/C5, and C7/C9, respectively.
  • G.1 Generation-level failures: 37% of failures involve hallucinated lucide-react exports, including brand icons, renamed symbols, and identifiers from foreign libraries.The scaffold pins lucide-react@ˆ0.460.0, making unsupported imports a factuality failure.
  • G.1 Generation-level failures: 35% of failures involve overwriting scaffold package.json with unpublished versions or invented package names, causing pnpm installation errors.One example is using mantine instead of the registry package @mantine/core.
  • G.1 Generation-level failures: 10% of failures use packages absent from extra_dependencies, and Self-Debug does not repair these omissions by scanning generated source.The plan-step contract assigns dependency declaration to the model.
  • G.1 Generation-level failures: 4% of failures use unsupported @/* aliases, while 6% contain cross-file export-name mismatches that can trigger home-route ReferenceErrors.The scaffold requires relative imports and does not declare the @/* alias.
  • G.1 Generation-level failures: 10% of failures come from long-tail modes C5, C7, C8, and C9, while half of all failures are instruction-following violations.Among instruction-following violations, scaffold-file constraints account for 73%, dependency declarations for ∼20%, and relative-import constraints for 8%; source-location constraints have zero violations.
  • G.2 Per-size failure-mode shifts on the Qwen2.5-VL ladder: At 3B, 7B, and 32B, the dominant failure category shifts from syntax to inconsistency to mixed syntax plus hallucination; at 72B, residual failures are dominated by deeper-stage syntax and runtime errors.The categories map onto the C1–C9 taxonomy, with syntax including C7 and inconsistency including C2.
  • G.3 Self-debug feedback protocol: EXEC@3 retries each failing app for up to three rounds with the same model, forwarding toolchain diagnostics after every round.The metric therefore captures iterate-and-fix ability, including root-cause localization and repair from standard build output.

G.4 Navigation and visual-fidelity failures · G.5 Interaction-inference failures

Navigation failures primarily arise from mismatched or hallucinated routes, while visual-fidelity variation among executable apps is relatively narrow across sub-skills. Interaction inference exposes failures that static visual evaluation misses, especially when shared state must persist across routes.

  • G.4 Navigation and visual-fidelity failures: Mismatched navigation dominates navigation failures, with links resolving to no registered route in the application router.Hallucinated routes are another recurring pattern: applications may register and render routes absent from the input screenshot set.
  • G.4 Navigation and visual-fidelity failures: 68.9–80.8 is the conditional Color spread among EXEC@3-passing apps, compared with 73.9–83.3 for Position.The reported conditional spreads are 11.9 for Color and 9.4 for Position.
  • G.4 Navigation and visual-fidelity failures: 70.6–75.6 and 87.0–91.7 are the conditional ranges for Size and Text, whose spreads are 5.0 and 4.7, respectively.Position drops are attributed primarily to generated DOMs containing strict subsets of reference blocks rather than horizontal grid collapse.
  • G.5 Interaction-inference failures: Claude Sonnet 4.6 threads Add-to-Cart through a CartContext, and the cart route renders the added product after navigation to /cart.The example uses five e-commerce screenshots from 48_shopco-ecommerce and the same annotated action sequence.
  • G.5 Interaction-inference failures: Gemini 3.1 Pro Preview produces a visually faithful product page, but its Add to Cart handler fails to propagate the interaction across the cart route.The comparison uses the same five screenshots and the same Add-to-Cart action as the Claude Sonnet 4.6 example.
  • G.5 Interaction-inference failures: VFS cannot distinguish the two outputs from static cart-route captures, whereas IIS flags the missing S3 affordance.Both outputs render an empty cart at protocol time; the divergence is the Broken shared state pattern.

G.6 Case studies: latent-affordance inference … Stage 3 (Self-Debug).

The case studies show that models often reproduce screenshots while failing to infer latent behavioral affordances, including audio, state persistence, editing, file upload, and quiz logic. The pipeline addresses these failures through staged planning, code generation, and self-debugging prompts with explicit output and repair constraints.

  • G.6 Case studies: latent-affordance inference: Only Kimi K2.5 turns the visually specified synthesizer into a playable application; four baselines produce static replicas and GLM-4.6V fails to build.Kimi constructs an AudioContext → OscillatorNode → GainNode chain with a 500 ms decay and connects keyboard and pointer inputs.
  • G.6 Case studies: latent-affordance inference: Claude Sonnet 4.6 best recovers e-commerce state behavior, while GPT-5.4 provides only partially wired cart logic across the two tasks.Sonnet implements cart deduplication, quantity-zero removal, checkout form control, and submit-driven search filtering.
  • G.6 Case studies: latent-affordance inference: Only Kimi K2.5 persists todo state across reloads, and Kimi plus Claude infer native HTML5 drag-and-drop import behavior from visual cues.Kimi stores mutations in localStorage through AppContext and rehydrates on mount; both models bind onDragOver and onDrop to JSON import.
  • G.6 Case studies: latent-affordance inference: Live interaction tests expose editor and quiz failures that static rendering misses: Sonnet and Qwen enable typing, while only Sonnet validates answers and awards XP.GPT-5.4 renders inert editor panes, and Gemini lacks the editor route; in the quiz, Kimi navigates without validation or scoring.
  • G.6 Case studies: latent-affordance inference: Across the cases, latent-affordance inference is neither monotonic with model scale nor visual fidelity, motivating IIS, NRS, and failure-taxonomy analysis beyond pixel scores.The cases show baseline disagreement over behavioral types and structural primitives that rendered visual comparisons cannot expose.
  • Stage 1 (Plan).: The Plan stage gives models screenshots and requires a JSON source-file plan, while enforcing source-location, import, dependency, and scaffold-preservation clauses.All generated source files must be under src/, imports must be relative, extra packages must be declared, and scaffold files must not be regenerated.
  • Stage 2 (Generate).: The Generate stage requests complete source code for every planned file in exact file-block formatting, with no truncation, explanation, or markdown fences.If generation is truncated, the pipeline sends up to three continuation turns without repeating prior content.
  • Stage 3 (Self-Debug).: The Self-Debug stage retries execution failures for up to three rounds using file localization, build-error repair, and holistic runtime-error repair prompts.Build repair supplies the error, file listing, and current contents; runtime repair supplies candidate files when the app builds but the home route is blank.
Loading 2607.06306v1…