PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

Xuanle Zhao, Qiushi Sun, Jingyu Xiao, Xuexin Liu, Haoyue Yang, Qiaosheng Chen, Xianzhen Luo, Jing Huang, Yufeng Zhong, Lei Chen, Shuai Fu, Zhenlin Wei, Jinhe Bi, Lei Jiang, Haibo Qiu, Siqi Yang, Peng Shi, Jian Hu, Zhixiong Zeng

arXiv:2606.15932v2cs.CL

TL;DR

Multimodal code tasks use visual artifacts and grounded states, but existing evaluations often capture only part of correctness. This survey organizes the field across four domains and finds that visual fidelity and interactive correctness can diverge, motivating validation beyond single-output similarity.

  • Problem

    Multimodal code intelligence lacks unified coverage of visual inputs, executable representations, and correctness dimensions spanning appearance, structure, semantics, and interaction.

  • Method

    The survey classifies tasks by code’s role, organizes methods and benchmarks into four domains, and records their task formulations and evaluation signals.

  • Results

    Visual fidelity reached 64.25% while interactive function reached 24.39% in IWR-Bench, showing that rendered similarity and interaction correctness measure different capabilities.

  • Takeaways & Limitations

    Correctness assessment should combine signals aligned with each visual-code artifact rather than treat a single image, program, or preference score as universal evidence.

  • Takeaways & Limitations

    Cross-method comparison remains limited because benchmarks observe different slices of correctness, while cross-task transfer evaluations rarely isolate causal transfer.

Abstract

from arXiv · show

While Large Language Models (LLMs) have substantially advanced text-to-code synthesis, many real programming tasks specify intent through visual artifacts such as screenshots, charts, vector drawings, videos, and interactive states. These tasks require models to connect visual perception to executable programs, because correctness depends not only on syntax but also on layout, data semantics, interaction behavior, and domain-specific constraints that apply after execution. This survey examines Multimodal Code Intelligence, covering systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable symbolic structure, a scientific representation, an intermediate reasoning trace, or an executable policy or tool interface. We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects mature artifact-generation problems to emerging agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Looking ahead, we argue that future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move this field from single-output imitation toward evidence-grounded executable systems. An ongoing project and resources are available on \href{https://github.com/xjywhu/Awesome-Multimodal-LLM-for-Code}{GitHub}.

1 Introduction

Multimodal Code Intelligence addresses the gap between text-centric code generation and programming tasks grounded in visual artifacts or states. This survey formalizes the field by code role, organizes it into four domains, and defines a review scope covering visually grounded code generation, editing, verification, execution, and reasoning.

  • Motivation: Visual inputs encode dense spatial and structural information that text-centric NL2Code approaches often miss, especially in visual-centric domains.The survey motivates multimodal code intelligence by contrasting sequential text with high-bandwidth visual communication.
  • Taxonomy: The survey organizes Multimodal Code Intelligence into Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks.These domains cover frontend code generation, plotting scripts, structured vector or diagram code, and emerging agentic or unified settings.
  • Problem formulation: The survey distinguishes tasks by whether code functions as a rendered artifact, editable structure, reasoning trace, scientific representation, executable policy, or tool interface.This formulation connects heterogeneous multimodal code-generation tasks to the dominant role played by code in each domain.
  • Survey methodology: The review follows source collection, candidate screening, taxonomy assignment, and manual consistency checks, using a literature snapshot updated through January 2026.Sources are collected from arXiv and major venues, with emphasis on recent work from 2022–2026 and earlier papers defining important tasks or evaluation protocols.
  • Scope: Its scope includes visually grounded generation, editing, verification, execution, and reasoning, while excluding purely language-driven code generation and unrelated software-engineering issue resolution.Included works may use visual inputs, outputs, or grounded states, and may treat code as a renderable representation, intermediate trace, executable artifact, or action interface.

2 Task Formulation

Section 2 formalizes Multimodal Code Intelligence through visual-to-code synthesis and code-centric reasoning, distinguishing how code functions as an artifact, symbolic intermediary, or executable policy. It defines three synthesis subtasks and two reasoning pathways that connect visual evidence, execution, and interaction.

  • 2 Task Formulation: The taxonomy organizes Multimodal Code Intelligence into visual-to-code synthesis and code-centric reasoning paradigms.It provides the section’s formal task framework.
  • 2.1 NL2Code Preliminaries: Conventional NL2Code synthesizes an executable program from natural language but cannot perceive spatial requirements essential to visually grounded intent.This limitation motivates multimodal extensions beyond logic-centric tasks.
  • 2.2 Multimodal Code Synthesis: Multimodal Code Synthesis generates or modifies code when visual context, rendered feedback, or visually specified intent is central.The framework distinguishes direct generation, instruction-driven editing, and reference-based refinement.
  • 2.2 Multimodal Code Synthesis: Direct generation synthesizes code from visual context and text to produce a requested artifact, with visual fidelity requiring correct layout, geometry, style, and visible content.The task covers reconstruction and multimodal specifications, while execution determines the resulting artifact.
  • 2.2 Multimodal Code Synthesis: Instruction-driven editing changes visual content from textual or visual prompts, evaluating visual reasoning, spatial grounding, and counterfactual code synthesis.Editing may operate with or without structural guidance.
  • 2.2 Multimodal Code Synthesis: Reference-based refinement improves a potentially flawed draft against a visual reference or constraints, using the draft as a structural prior for alignment and debugging.Unlike source-code-agnostic editing, refinement explicitly leverages an initial code state.
  • 2.3 Code-Centric Reasoning and Acting: Programmatic tool-use either solves queries entirely through deterministic execution or processes visual inputs before subsequent inference, supporting symbolic derivation and perception-aware reasoning.Execution may produce the final answer or an augmented view such as a crop or edge-detected image.
  • 2.3 Code-Centric Reasoning and Acting: Code-centric reasoning treats executable code as a symbolic intermediary or policy, covering programmatic tool-use and executable policies for visual reasoning and environmental control.Programmatic tool-use decomposes visual queries into modular programs, while policies map observations and goals to structured actions across sequential environments.

3 Graphical User Interface

GUI code intelligence spans web and mobile generation, editing, and refinement, but evaluation must connect visual reconstruction with executable interaction and runtime behavior. Browsers provide the clearest verification substrate for web tasks, whereas fragmented mobile environments make native correctness harder to assess.

  • Scope: GUI code generation covers visual-to-executable implementation across web and mobile, including direct generation, editing, refinement, action replay, and environment-transition checks.
  • Web-to-Code Evaluation: Web benchmarks expose three dominant correctness signals: static rendering, executable interaction, and specialized preference or agent-task evaluation.The browser acts as both renderer and evaluator, enabling scalable inspection and interaction through HTML, CSS, JavaScript, and WebDriver.
  • Method Trajectory: Web-to-code methods progress from imitation-based reconstruction and direct supervised fine-tuning to decomposed agents, iterative visual feedback, critic repair, replay, and executable verification.These approaches increasingly use grounding, planning, implementation, review, rendering feedback, and interaction feedback as inspectable or corrective signals.
  • Verification Direction: GUI evaluation should pair visual targets with structured requirements, replayable actions, state assertions, and code-level checks rather than treating browser rendering as sufficient verification.Visual metrics capture perceptual alignment, interaction tests capture behavioral correctness, and code analysis captures maintainability and architectural validity.
  • Web-to-Code Evaluation: 64.25% visual fidelity versus 24.39% interactive function on IWR-Bench shows that rendered similarity and interaction correctness measure different capabilities (Chen et al., 2025m).Static screenshots can conceal event logic, state changes, component boundaries, and data flow.
  • Mobile-to-Code: Mobile-to-code evaluation is structurally fragmented because native applications are compiled binaries without a shared rendering or interaction environment across platforms.Current proxy signals do not by themselves verify native compilation, platform widgets, gestures, state changes, accessibility, or maintainable implementation.

4 Scientific Visualization · 4.1 Statistical Charts · 4.2 Structured Document

Scientific visualization code intelligence must preserve the semantics underlying charts and structured documents, not merely their appearance. The survey organizes these tasks around intent satisfaction, reconstruction fidelity, multi-grammar recovery, and verification through executable structured outputs.

  • 4 Scientific Visualization: Scientific visual-code artifacts require preserving the scientific semantics behind visual outputs, with code serving mainly for generation or refinement and sometimes as an explanatory or validation trace.
  • 4.1 Statistical Charts: Statistical chart generation splits into NL-to-Chart intent-to-code synthesis and Chart-to-Code visual reverse engineering, which impose different constraints and correctness targets.NL-to-Chart is underspecified because multiple visualizations may satisfy one query, whereas Chart-to-Code reconstructs data, encodings, and plotting logic from a rendered chart.
  • 4.1 Statistical Charts: Chart benchmarks evaluate executable, semantically appropriate outputs for textual intent, but reconstruction benchmarks assess recovered data, visual encodings, and editable or executable plotting logic.Chart-to-Code benchmarks emphasize reconstruction fidelity, while NL-to-Chart benchmarks approximate intent satisfaction through execution, semantic judgment, and task compliance.
  • 4.1 Statistical Charts: Chart methods mirror this split: NL-to-Chart uses planning, visual feedback, and instruction tuning, while Chart-to-Code uses chart-code data, rendering-aware feedback, and editing-oriented optimization.Preference-driven approaches optimize programs against executed visual and data outcomes rather than textual code similarity alone.
  • 4.1 Statistical Charts: Chart correctness is distributed across data, visual encoding, and intent fidelity, so plausible appearance or code similarity alone cannot establish correct values, aggregations, comparisons, or analytic claims.Future benchmarks should pair visual reconstruction with data recovery and executable rerendering, while evaluating NL-to-Chart intent satisfaction and analytic appropriateness through multi-signal verification.
  • 4.2 Structured Document: Structured document generation is a multi-grammar recovery problem requiring preservation of reading order, layout, text, tables, formulas, and cross-region references in one serialized output.Its canonical formulations include Document-to-Markdown, Table-to-Code, and Formula-to-LaTeX benchmarks spanning heterogeneous document structures and output languages.
  • 4.2 Structured Document: Document parsing has progressed beyond character accuracy toward evaluating reading order, correctness, and structural coherence across heterogeneous elements, including full-page layouts, tables, and formulas.Pipeline, end-to-end VLM, and reinforcement-learning methods address different structural rules for prose, tables, formulas, and layout relations.
  • 4.2 Structured Document: Structured document correctness spans reading order, layout, table grids, formula grammar, and output executability, motivating realistic benchmarks with dense layouts, domain conventions, and cross-page dependencies.The long-term goal is converting diverse documents into renderable structured representations using complementary Markdown, HTML, and LaTeX targets.

4.3 Academic Presentations · 4.4 Scientific Demonstration

Academic presentations transform research into coherent, editable visual narratives, while scientific demonstrations generate executable artifacts whose explanations must remain faithful to domain knowledge and pedagogical intent. Across both settings, visual plausibility is insufficient: correctness requires inspectable links among claims, code, rendered states, interaction, and validation.

  • 4.3 Academic Presentations: Academic presentations require models to select, reorder, and emphasize evidence while preserving visual coherence, editability, and presentation flow across slides, posters, and narrated videos.Their correctness spans argument selection, editable object structure, audience attention, sequential narrative, single-canvas density, and temporal alignment.
  • 4.3.1 Academic Presentations Generation Benchmarks: Presentation benchmarks cover slide generation and editing, slide-to-code reconstruction, presentation videos, and poster compression, but only partially measure rhetorical sequencing, presenter intent, and communicative effectiveness.Zenodo10K contains 10,448 curated presentations but reports generation results on 500 sampled tasks per configuration; poster evaluation includes reader-simulation and checklist criteria.
  • 4.3.2 Academic Presentations Generation Methods: Presentation methods use programmatic rendering APIs, editable object manipulation, and visual feedback, including binary-tree poster layouts and painter-commenter repair for text overflow.These routes support high-level content planning, visual-to-code reconstruction, spatial planning for variable-length content, and dynamic layout repair.
  • 4.3.2 Academic Presentations Generation Methods: Future presentation systems should use intermediate representations linking claims, evidence, layout roles, visual salience, and revision history, with benchmarks checking overflow, overlap, edit success, and argument recovery.Evaluation should verify claim-evidence alignment and reader recovery rather than merely whether an artifact looks plausible.
  • 4.4 Scientific Demonstration: Scientific demonstrations generate executable visual explanations, such as molecular scripts, interactive STEM webpages, and theorem animations, whose code must preserve equations, domain constraints, mechanisms, and pedagogical intent.Unlike ordinary plotting, the rendered result must function as evidence-backed explanation across source content, executable program, artifact, and validation signal.
  • 4.4.1 Scientific Demonstration Generation Benchmarks: Scientific demonstration benchmarks assess code-rendered explanations and interactive pedagogy, including multi-domain visual reasoning, interactive front-end artifacts, and workflow or tool-use requirements.EduVisBench uses a fine-grained rubric informed by pedagogical theory, while adjacent ScienceAgentBench and ScienceBoard cover broader scientific-agent workflows.
  • 4.4.2 Scientific Demonstration Generation Methods: Scientific demonstration methods combine renderer-driven supervision with agentic explanation workflows, using executable tools, retrieval, runtime repair, staged planning, coding, and animation generation.CoSyn synthesizes 400K multimodal images and 2.7M instructions, while agentic systems coordinate learning objectives, reasoning, retrieved knowledge, code, and rendered explanations.
  • 4.4.2 Scientific Demonstration Generation Methods: Because plausible visuals can hide invalid mechanisms, future demonstrations should keep equations, parameters, package calls, code, tool outputs, and rendered states inspectable through domain-specific validators.Visual similarity alone cannot detect wrong values, broken structure, omitted claims, invalid mechanisms, or unverifiable tool outputs.

5 Structured Graphics

Structured graphics treat code as symbolic, editable, and executable representations whose correctness depends on object structure, logical relations, and construction procedures, not rendering alone. SVG, diagrams, and CAD therefore require verification of structural equivalence alongside visual fidelity.

  • Overview: Structured graphics shift generation from pixel reproduction to symbolic, editable, executable representations exposing objects, relations, constraints, and construction procedures.Their central failure mode is structural non-equivalence: plausible rendering can conceal lost object hierarchies, relation graphs, or parametric construction history.
  • SVG: SVG generation must balance rendered fidelity with compact, meaningful vector structure across text-conditioned, image-conditioned, and editing tasks.SVG code combines discrete commands for objects and paths with continuous coordinates for geometry, making its low-level, lengthy representation difficult to learn, write, and revise.
  • SVG: SVG evaluation combines semantic alignment, perceptual reconstruction, and structural or code-quality proxies because visual fidelity, geometric precision, and editability jointly determine correctness.Representative resources span text-guided synthesis, image-to-SVG reconstruction, and rendering-feedback settings, including IconShop (Wu et al., 2023), StarVector, and RLRF (Rodriguez et al., 2025b).
  • Diagrams: Diagram code generation must preserve logical relations such as control flow, dependencies, workflow state, and typed connectivity rather than merely reconstructing shapes and coordinates.Future evaluation should test graph topology, edge direction, branch conditions, path reachability, branch equivalence, execution behavior, and targeted visual questions.
  • CAD: CAD research should move from shape reconstruction toward verifiable parametric design synthesis using validators for geometry, constraints, feature dependencies, edit propagation, and manufacturability.Future benchmarks should include complex parts, assemblies, multi-view drawings, revision instructions, and domain-specific manufacturing requirements.

6 Frontier Tasks and Frameworks

This section examines frontier settings where code mediates perception, reasoning, and action as intermediate traces, tool calls, environment policies, or repair interfaces. It organizes these settings around five process-reliability challenges spanning visual manipulation, video, embodied control, grounded programming, and unified multimodal generation.

  • 6 Frontier Tasks and Frameworks: Frontier tasks treat generated code as an intermediate trace, tool call, environment policy, or repair interface rather than only a rendered artifact.These settings shift emphasis from final visual objects toward process reliability.
  • 6 Frontier Tasks and Frameworks: The section connects these settings to programmatic tool-use and executable-policy formulations while considering unified models across synthesis and acting.
  • 6 Frontier Tasks and Frameworks: Five settings organize the discussion: programmatic visual manipulation, video code generation, embodied control, visually grounded programming, and unified multimodal code generation.

6.1 Programmatic Visual Manipulation

Programmatic visual manipulation uses executable code to inspect, transform, and construct visual evidence rather than merely generate visual artifacts. Its central challenge is verifying that intermediate operations are executable, grounded, replayable, and causally relevant to the answer.

  • 6.1 Programmatic Visual Manipulation: The field treats code as an executable interface for cropping, detecting, measuring, drawing, masking, plotting, or querying visual evidence.This shifts visual reasoning into an intermediate action space whose correctness depends on the process, not only the final answer.
  • 6.1 Programmatic Visual Manipulation: Final-answer accuracy, trace executability, and process rewards remain insufficient because none alone proves that an operation caused the answer.Stronger evaluation should combine answer accuracy with operation replay and region grounding, checking whether interventions target the visual evidence claimed by the reasoning trace.
  • 6.1 Programmatic Visual Manipulation: Methods use either predefined expert tools with bounded, inspectable traces or generative programs that synthesize task-specific visual operations.Predefined tools support known operations, whereas generative code expands coverage by creating crops, masks, sketches, plots, measurements, and formal constructions.
  • 6.1 Programmatic Visual Manipulation: Training and reward methods increasingly supervise planning, execution, pixel-space operations, and evidence consistency instead of optimizing only final-answer accuracy.This includes instruction tuning, chain-of-manipulation reasoning, process rewards, and rewards for tool use that is executable and evidence-consistent.
  • 6.1 Programmatic Visual Manipulation: Generative systems increasingly construct auxiliary visual views, such as sketches, spatial indicators, plots, and highlighted regions, when the original image lacks sufficient evidence.Mathematical and geometric settings further improve verifiability when generated plots or formalized geometric elements provide external structure for checking intermediate artifacts.
  • 6.1 Programmatic Visual Manipulation: Future code-agent workflows should extend beyond fixed APIs to terminals, files, libraries, logs, and scaffolds while validating answer-relevant visual evidence.The intended abstractions are executable, inspectable, replayable, grounded in visual regions, and tied to the answer.

6.2 Video Code Generation

Video Code Generation comprises Code-to-Video authoring and Video-to-Code procedure recovery, with code providing temporal structure but current benchmarks often failing to verify faithful timing and state evolution. The section therefore frames trajectory-aware, multi-signal evaluation as the key direction beyond final-outcome scoring.

  • Task Formulation: Video Code Generation divides into Code-to-Video, which programs visual artifacts, and Video-to-Code, which recovers executable procedures from demonstrations.Code-to-Video uses code as an authoring scaffold for layouts, keyframes, narration, and transitions, whereas Video-to-Code extracts compressed procedures.
  • Evaluation Bottleneck: Current benchmarks assess communicative usefulness or task completion, but neither reliably verifies temporal consistency, motion fidelity, or faithful state changes.A video can preserve content while using awkward pacing or discontinuous transitions, and a policy can succeed while discarding velocity, contact timing, or recovery behavior.
  • Code-to-Video: Code-to-Video systems use executable scripts as planning layers that constrain high-level structure for scenes, narration, transitions, and explanatory animations.Code2Video (Chen et al., 2025n), PresentAgent (Shi et al., 2025), and Theorem ExplainAgent (Ku et al., 2025b) exemplify this authoring role.
  • Video-to-Code: Video-to-Code systems compress visual demonstrations into executable strategies, exemplified by RoboPro synthesizing robotic manipulation policies from large-scale video data.This abstraction can reduce the need for expensive robot trajectory collection, but the subsection treats RoboPro as video-to-program abstraction rather than a full deployment account.
  • Scope and Trajectory: Future benchmarks should jointly evaluate end outcomes, trajectory consistency, event timing, and narration- or motion-aligned behavior rather than relying on final outcomes alone.The proposed trajectory is an evaluation expansion, not an established empirical trend, moving from sequencing visual states toward time-aware state evolution.

6.3 Embodied Control

Embodied Control studies code-centric policies that map visual observations and goals to actions, using programs, specifications, rewards, or policy scripts to mediate execution. It emphasizes inspectable symbolic intent while recognizing that continuous control, safety, recovery, and embodiment remain essential to verification.

  • Embodied Control: Embodied Control focuses on code-centric settings where generated programs, reward functions, specifications, or policy scripts mediate execution from visual observations and goals.This scope excludes a general survey of embodied vision-language-action policies.
  • Embodied Control: VirtualHome [Puig et al., 2018] models household activities as executable programs, while later benchmarks test whether language models can generate embodied control logic.These environments make visual grounding observable through executable environments, skill programs, and task outcomes.
  • Embodied Control: Code as Policies [Liang et al., 2022] and ProgPrompt expose API calls, assertions, and hierarchical logic, making behavior more inspectable than end-to-end policies.The core code-generation route emits action scripts, skill programs, or geometric procedures that directly structure execution.
  • Embodied Control: PACT [Wei et al., 2023], RoboCodeX [Mu et al., 2024], and Octopus [Yang et al., 2024b] use sensorimotor data, multimodal post-training, iterative SFT, or environmental rewards to improve generated control code.Environmental feedback connects code to evaluated outcomes, but binary success signals can conceal unsafe intermediate actions, weak recovery, and simulator- or robot-specific behavior.
  • Scope and Trajectory: Future benchmarks should verify the boundary between symbolic intent and continuous control across specification, execution, contact timing, safety, recovery, camera, object, tool, and robot-morphology changes.Code should express goals, object references, spatial constraints, skill composition, reward terms, and recovery conditions, while continuous adaptation remains with robot controllers.

6.4 Visually Grounded Programming

Visually grounded programming generates or repairs executable code using visual artifacts as program-relevant constraints or evidence. The field spans algorithmic and software-engineering benchmarks, with methods increasingly combining structured visual representations, agentic feedback, and verification against visual states.

  • Visually Grounded Programming: Visually grounded programming treats diagrams, screenshots, visual examples, or rendered failures as evidence that constrains executable programs or repository patches.Correctness depends on whether visual information survives into code generation or patch refinement, not merely on syntactic validity.
  • Visually Grounded Programming: Benchmarks cover visually grounded algorithmic programming and software engineering, but final pass rates may not prove image grounding when text or repository context enables shortcuts.Table 7 summarizes task types, output languages, visual-evidence roles, and correctness signals; software-engineering benchmarks increasingly include screenshots and videos as repair evidence.
  • Visually Grounded Programming: Methods follow two routes: translating visuals into textual or symbolic surrogates before synthesis, or using visual feedback within software-engineering agents.CodeVision (Wang et al., 2025d) converts flowcharts into Mermaid code, while SWE-agent M (Yang et al., 2024d;e) adds browser interaction, screenshots, image viewing, and terminal operations.
  • Visually Grounded Programming: Textual surrogates ease code generation but can lose spatial and state information, whereas agentic feedback depends on reproducible environments, stable screenshots, and meaningful post-patch tests.The central bottleneck is preserving visual evidence through code generation or patch refinement rather than perception alone.
  • Visually Grounded Programming: Future systems should preserve graph structure, DOM and browser state, rendered failures, and interaction traces, while benchmarks test whether visual evidence changes the expected code location.Screenshots, browser traces, terminal logs, reproduction scripts, localized edits, and post-patch executions should remain linked to generated programs or patches.

6.5 Unified Multimodal Code Generation

Unified multimodal code generation seeks shared models and visual-code representations spanning synthesis, editing, refinement, tool use, interactive artifacts, OCR, visualization, and software tasks. Its central unresolved question is whether broader data, model integration, and feedback produce reusable cross-task abstractions rather than merely broader task acceptance.

  • Unified Benchmarks: Unified benchmarks test distinct forms of visual-code grounding, including structured-code reconstruction, rendering-code generation, interactive artifacts, visually grounded programming, and iterative refinement.Their diversity supports unified evaluation but makes direct score comparisons difficult.
  • Unified Benchmarks: Image2Struct (Roberts et al., 2024) evaluates extracting structured code from images, while ArtifactsBench (Zhang et al., 2025a) verifies executable interactive artifacts across nine domains.Image2Struct covers webpages, mathematical formulas, and musical scores using CIS and EMS; ArtifactsBench uses 1.8k queries and checklist-guided MLLM judging.
  • Method Development: Unified methods combine cross-domain data, integrated vision-language and code models, and feedback-aware optimization to reduce interface fragmentation and move beyond supervised fine-tuning alone.Examples include VisCoder2 (Ni et al., 2025a), VisCodex (Jiang et al., 2025a), JanusCoder (Sun et al., 2025b), VinciCoder (Zhao et al., 2025b), and OCRVerse (Zhong et al., 2026).
  • Scope and Trajectory: Unification remains unproven because accepting many task formats does not establish shared visual-code abstractions or measurable transfer across tasks.Current reports more often show in-distribution task acceptance, leaving unresolved whether mixtures induce reusable notions of layout, events, and state change.
  • Takeaway: Frontier settings extend multimodal code generation from artifact production to inspecting evidence, modeling time, acting in environments, repairing programs, and attempting transfer across tasks.Correctness consequently depends on interaction behavior, domain meaning, editable symbolic structure, and other executable constraints beyond artifact appearance.

7 Future Directions

Future work should replace single correctness signals with role-aligned validator profiles and evaluate visual-code systems through execution episodes, cross-task transfer, and evidence-grounded agent traces. These directions aim to establish whether generated code preserves visual, structural, semantic, editable, and interactive intent across states and environments.

  • Verification signals: Role-aligned validator profiles should separate visual similarity, execution success, textual correctness, semantic fidelity, structural validity, editability, and interaction correctness rather than rely on one score.A single image, program, or VLM preference cannot certify all relevant properties; reliable evaluation generally requires complementary checks.
  • Multi-state verification: Stateful visual-code tasks should be evaluated as execution episodes containing initial states, generated actions, intermediate observations, expected transitions, validator outputs, and recovery cases.Static rendering can hide failures under clicks, resizing, state updates, timing, invalid mechanisms, occlusion, contact, or controller limits.
  • Cross-task transfer: Unified models should be tested for cross-task transfer on held-out skills, primitives, and compositions, reporting both positive and negative transfer against matched mixture controls.This distinguishes reusable layout, symbolic-relation, and interaction skills from merely accepting more task formats or improving in-distribution performance.
  • Verifiable agent traces: Agentic visual-code systems need evidence logs linking observations and visual regions to changed code or actions, expected validator improvements, replay results, and fallback decisions.Such logs enable replay, visual ablation, counterfactual testing, permission controls, simulator or emulator guards, human review, and failure attribution.

8 Limitations

The survey is limited by incomplete coverage of available work and by the lack of universally comparable evaluations across domains. It therefore presents an organizing taxonomy rather than a definitive field boundary or universal ranking.

  • 8 Limitations: Coverage is bounded by publicly available papers, benchmarks, and repositories, potentially omitting recent systems, closed-source deployments, and domain-specific tools.The survey may overrepresent benchmark-proposing works and underrepresent deployed systems without public artifacts.
  • 8 Limitations: Different benchmarks observe different slices of correctness, limiting cross-method comparison and preventing a universal ranking.The survey instead emphasizes within-domain comparisons, common failure modes, and reliability risks such as leakage, benchmark saturation, and judge sensitivity.
  • 8 Limitations: Cross-task transfer discussion is agenda-setting because current evaluations rarely isolate causal transfer and leave deployment-facing concerns underexplored.This limitation constrains conclusions about transferable visual-code capabilities in deployed settings.

9 Broader Impact

Multimodal code intelligence can make visual programming and executable artifact revision more accessible, but visually plausible outputs may conceal serious errors. Agentic deployment also raises privacy, security, and safety risks, motivating provenance, permissions, logging, validation, and human oversight.

  • 9 Broader Impact: Multimodal code intelligence lowers barriers to visual programming by converting screenshots, diagrams, sketches, videos, or natural language into code for interfaces, charts, documents, SVG, CAD, or robot policies.It can also help experts turn visual feedback into executable revisions, making artifacts easier to inspect, edit, and reuse.
  • 9 Broader Impact: Visually plausible outputs can conceal serious errors, including incorrect chart data, so appearance alone is insufficient evidence of correctness.
  • 9 Broader Impact: Agentic systems create privacy and safety risks when operating browsers, files, APIs, design tools, proprietary repositories, or robots.Screenshots and design files may expose private information, generated code may leak or be misused in proprietary contexts, and embodied policies may behave differently outside simulation.
  • 9 Broader Impact: Deployment should pair generation with provenance tracking, permission scopes, execution logs, domain validators, and human review for high-stakes use cases.

10 Conclusion

The paper presents a structured survey of Multimodal Code Intelligence across four domains, reviewing benchmarks and methodologies for translating visual perception into executable representations.

  • The survey organizes Multimodal Code Intelligence into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks.
  • It extensively reviews existing benchmarks and methodologies for multimodal code generation tasks.
  • The surveyed methods translate visual perception into diverse executable representations across a wide range of tasks.
Loading 2606.15932v2…