Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

Motivation

• Text-only specifications can lose spatial and structural information, motivating visual perception as a core prerequisite for visually driven programming.The introduction identifies frontend development, data visualization, and computer-aided design as domains where textual descriptions can misalign human intent with generated code.Source: §1 · p. 3

Survey Framework

• The survey organizes research into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks.These cover web and mobile applications; charts, documents, and presentations; SVG, diagrams, and CAD; and emerging visual manipulation, video generation, and unified multimodal models.Source: §1 · p. 3, Figure 1 · p. 4
• A formal problem formulation connects tasks to code’s dominant role, including rendered artifact, editable structure, reasoning trace, or executable policy.Domain reviews cover benchmarks and methodologies; subsection summaries identify task objectives, dominant evidence signals, and validation gaps, while domain takeaways summarize bottlenecks.Source: §1 · p. 3

Verification Agenda

• The research agenda emphasizes multi-signal validation, multi-state verification, cross-task transfer testing, and verifiable agent traces.These directions address artifacts, tool-use traces, and executable policies after rendering, execution, interaction, or replay.Source: §1 · p. 3

Survey Methodology

• The review uses staged collection, screening, taxonomy assignment, and manual consistency checks, with a literature snapshot updated through January 2026.Coverage emphasizes 2022–2026 research and earlier task-defining benchmarks or datasets, with ongoing survey and repository updates. LLMs assist drafting, consistency checking, and metadata organization; authors manually check inclusion decisions, taxonomy assignments, and final descriptions.Source: §1 · pp. 3–4
• Included work uses visual inputs, outputs, or grounded states to generate, edit, verify, execute, or reason with code; purely language-driven work is excluded unless visually grounded.Works are coded by domain, task formulation, code role, and evaluation signal; ambiguous cases are checked against original task definitions, benchmark protocols, or method objectives.Source: §1 · p. 4

2 Task Formulation

• Multimodal Code Intelligence is organized into two core paradigms: visual-to-code synthesis and code-centric reasoning.This distinction separates tasks that produce visual artifacts through code from tasks that use code for reasoning or environmental control.Source: §2 · p. 4, §2.3 · p. 6

2.1 NL2Code Preliminaries

• NL2Code synthesizes executable programs from natural-language descriptions, but its unimodal formulation cannot perceive spatial requirements essential to visually specified intent.It provides the conventional baseline for formulating multimodal extensions.Source: §2.1 · pp. 4–5

2.2 Multimodal Code Synthesis

• Direct generation synthesizes code from visual context and textual specifications, but visual fidelity alone cannot establish structural, semantic, or functional correctness.It reconstructs a reference or realizes a multimodal specification, targeting layout, geometry, style, and visible content. Validation must also check underlying data, editability, constraints, handlers, and domain semantics.Source: §2.2 · p. 5
• Instruction-driven editing tests visual reasoning and spatial grounding by modifying code using textual instructions, optional visual localization cues, and optional structural guidance.Text-guided editing uses language alone, while visual-prompt editing adds cues such as bounding boxes. Source-agnostic variants infer programs from pixels; code-aware or tool-based variants modify existing source or structured representations.Source: §2.2 · p. 5
• Reference-based refinement corrects a potentially flawed code draft to match a visual reference or satisfy specified constraints.Unlike instruction-driven content manipulation, refinement targets error correction and quality improvement, using the draft as a structural prior for precise alignment and functional debugging.Source: §2.2 · p. 5

2.3 Code-Centric Reasoning and Acting

• Programmatic tool-use treats code as an intermediate reasoning trace, either executing to a final answer or augmenting visual inputs for subsequent inference.Generated programs invoke perceptual tools or symbolic calculations; visual augmentation can crop images or detect edges. Included work must use code to create, inspect, transform, or verify visual evidence, or evaluate the code trace itself.Source: §2.3 · pp. 6–8
• Executable policies map visual observations and goals to structured actions through control loops and API calls, prioritizing functional correctness and goal attainment.Generated action scripts drive successive environment states in settings including embodied robotics and digital GUI navigation.Source: §2.3 · p. 8

3.1 Website Application

• Web generation benefits from a shared browser loop connecting code, rendering, inspection, and interaction, although evaluations can overemphasize appearance when functionality remains untested.VLMs extract structural and stylistic attributes from pixels to reconstruct interfaces using HTML, CSS, and JavaScript.Source: §3.1 · p. 8

3.1.1 Website Code Generation Benchmarks

• Web benchmarks expose three distinct correctness signals: static rendering, executable interaction, and specialized preference or agent-task evaluation.Static benchmarks span synthetic and real-site reconstruction, DOM recovery, layout, fidelity, editing, and aesthetics; dynamic benchmarks test reactive behavior, navigation, sequential development, and video-grounded reconstruction. Preference arenas and agentic development evaluations should remain separate because they provide different evidence.Source: §3.1.1 · pp. 8–9

3.1.2 Website Code Generation Methods

• Web methods span supervised imitation, inspectable agent decomposition, and feedback-driven refinement, with browser execution supplying training, repair, and verification signals.Supervised approaches scale synthetic and real-world screenshot-code data and reduce inference costs through token compression; agent workflows separate grounding, planning, implementation, review, and test-driven refinement while supporting editing and reusable components. Feedback methods use visual critics, execution loops, agent replay, and some RL-style optimization, but visual rewards, task completion, and code checks require separate reporting because critics and agents retain coverage and reliability limits.Source: §3.1.2 · pp. 9–10
• 64.25% visual fidelity versus 24.39% interactive function on IWR-Bench demonstrates that appearance and interaction correctness measure different capabilities (Chen et al., 2025m).Static screenshots hide event logic, latent state changes, component boundaries, data flow, and error handling, allowing visually convincing implementations to fail under user actions or code inspection.Source: §3.1.2 · p. 10

3.2 Mobile Application

• Mobile generation faces fragmented execution environments, with platform-specific compiled applications and no shared rendering or interaction environment comparable to the browser.Native applications also lack publicly crawlable source, making this fragmentation a structural constraint that shapes benchmark and method proxy signals.Source: §3.2 · p. 11

3.2.1 Mobile Code Generation Benchmarks

• Mobile benchmarks substitute mockup reconstruction, design-tool editability, localized critiques, and screenshot preferences for direct native runtime verification.Platform-specific build systems, device or emulator execution, and interaction instrumentation make standardized end-to-end evaluation difficult; scalable artifact and evaluator scores therefore provide only partial correctness evidence.Source: §3.2.1 · pp. 11–12

3.2.2 Mobile Code Generation Methods

• Mobile methods expose partial correctness through structure, retrieval, compilation checks, learned rewards, and editable prototypes without closing the native execution loop.Component segmentation, transition graphs, hierarchy modeling, reference retrieval, and self-critique support structural recovery, while rewards, editable prototypes, and query-generated interactive UIs provide operational proxies. These signals leave native runtime, platform widgets, gestures, state changes, accessibility, and maintainability incompletely verified, while learned rewards risk favoring measurable quality over user intent.Source: §3.2.2 · p. 12

3 Graphical User Interface

• GUI evaluation needs multi-state verification connecting screenshots, generated code, executed states, user actions, and implementation structure rather than treating visual similarity as complete validation.Stronger protocols combine structured requirements, replayable actions, state assertions, and code-level checks, reporting perceptual alignment, behavioral correctness, and architectural validity separately.Source: §3.1.2 · p. 11, §3.2.2 · p. 13

4 Scientific Visualization

• Scientific visualization treats code as a renderable, inspectable representation that must preserve scientific meaning, not just appearance.The scope includes charts, structured documents, academic presentations, and scientific demonstrations. Correctness encompasses data, structure, argument flow, equations, and domain constraints; demonstration code can also provide explanatory or validation traces.Source: §4 · p. 13

4.1 Statistical Charts

• NL-to-Chart synthesizes visualizations from underspecified intent, whereas Chart-to-Code reconstructs the data, visual encodings, and plotting logic implied by a rendered chart.Multiple visualizations can satisfy one textual query, while multiple reconstruction programs must preserve the input chart's data semantics and encodings. This asymmetry produces different capability requirements and methodological trajectories.Source: §4.1 · p. 13

4.1.1 Chart Code Generation Benchmarks

• Chart benchmarks evaluate intent satisfaction for NL-to-Chart and reconstruction fidelity for Chart-to-Code, increasingly extending to multi-turn refinement, editing, and infographic charts.Evaluation spans executability, semantic judgment, and reference-code matching. Editing additionally requires preserving unchanged data while following textual or visual instructions.Source: §4.1.1 · pp. 13–14, Table 2 · p. 15

4.1.2 Chart Code Generation Methods

• Chart methods target intent alignment or reconstruction fidelity, but reliable verification must separately assess visual reconstruction, data accuracy, executable correctness, and design quality.NL-to-Chart uses planning, visual feedback, and instruction tuning; Chart-to-Code scales chart-code pairs and adds rendering-aware, preference-driven optimization. Synthetic pairs can miss professional conventions, while visual-similarity rewards can conceal incorrect data, groupings, axes, or non-executable code.Source: §4.1.2 · pp. 14–16

4.2.1 Structured Document Code Generation Benchmarks

• Structured-document benchmarks test full-page reading order, table grids, and formula equivalence rather than character accuracy alone.Table-to-Code adds span preservation and compile-time validity, requiring rendered comparisons alongside structural metrics; Formula-to-LaTeX increasingly tests scientific context through structural and render-based verification. Provenance and de-duplication also matter because source documents may overlap pretraining corpora.Source: §4.2.1 · pp. 16–17

4.2.2 Structured Document Code Generation Methods

• In some formula-recognition settings, broad visual-language pretraining can complement or exceed specialized designs; Docfusion (Chai et al., 2025b) unifies layout detection and content recognition.DocTron-Formula (Zhong et al., 2025b) handles diverse granularities without task-specific designs, while formula recognition requires spatial nesting and context-sensitive, grammar-aware decoding. Docfusion uses Gaussian-Kernel Cross-Entropy Loss to bridge coordinate-based detection and token-based recognition.Source: §4.2.2 · pp. 18–19

4.2 Structured Document

• Structured-document synthesis recovers multiple structural grammars within one serialized representation, preserving reading order, layout, table grids, formula nesting, and executable syntax beyond ordinary OCR.Visual inputs become Markdown, HTML, or LaTeX while preserving text semantics and cross-region references. Local readability alone does not establish structural correctness.Source: §4.2 · p. 16, §4.2.2 · p. 19

4.2.2 Structured Document Code Generation Methods

• Structured-document methods need adaptive routing among page context, grid-aware parsing, and grammar-aware decoding, with rerendering or execution validating structured outputs.Full-page systems trade staged layout-and-recognition pipelines against end-to-end VLM extraction, with reinforcement learning adding structural rewards or binary unit tests. Table systems retain grid-aware components; Table2LaTeX-RL (Ling et al., 2025) jointly rewards TEDS structural accuracy and CW-SSIM visual fidelity.Source: §4.2.2 · pp. 18–19

4.3 Academic Presentations

• Academic presentations require selecting, reordering, and emphasizing research evidence into visually coherent, human-editable narratives rather than merely reconstructing sources.Slides require sequential narrative, posters require single-canvas density, and narrated presentations require temporal alignment. Visual polish can still conceal overflow, overlapping objects, inconsistent style, weak claim-figure alignment, or poor audience guidance.Source: §4.3 · p. 19, §4.3.2 · p. 20

4.3.1 Academic Presentations Generation Benchmarks

• 10,448 curated presentations comprise Zenodo10K (Zheng et al., 2025), but its generation benchmark uses 500 sampled tasks per experimental configuration.Broader benchmarks cover slide generation, editing, slide-to-code reconstruction, audiovisual alignment, and poster compression; evaluation includes MLLM-based PPTEval, reader simulation, and fine-grained checklists. These benchmarks still provide only partial evidence of rhetorical sequencing, presenter intent, and communicative effectiveness.Source: §4.3.1 · p. 19

4.3.2 Academic Presentations Generation Methods

• Slide-generation methods follow three routes: programmatic rendering interfaces, editable object manipulation, and visual or aesthetic feedback for layout repair.AutoPresent (Ge et al., 2025) delegates rendering to SlidesLib APIs, SlideCoder (Tang et al., 2025b) retrieves rendering code hierarchically, and template-based systems enable targeted object edits. PreGenie (Xu et al., 2025d) uses code and page reviewers, while EvoPresent (Liu et al., 2025a) uses PresAesth for scoring, defect adjustment, and layout comparison.Source: §4.3.2 · p. 20
• Presentation methods should connect claims, evidence, layout roles, visual salience, and revision history while addressing poster compression, overflow, and style consistency.PosterAgent (Pang et al., 2025) combines binary-tree layouts with painter-commenter overflow repair; P2P (Sun et al., 2025d) and PosterGen (Zhang et al., 2025i) separate extraction from design, while EfficientPosterGen (Tang et al., 2026) uses key-information identification and visual token compression. Proposed verification checks claim-evidence alignment, overflow, overlap, salience, edit success, and reader recovery of the intended argument.Source: §4.3.2 · pp. 20–21

4.4 Scientific Demonstration

• Scientific demonstration generation produces executable visual explanations of mechanisms, requiring fidelity to equations, domain constraints, and pedagogical intent rather than merely displaying data.Outputs include molecular scripts, interactive STEM webpages, and theorem animations; their rendered results must function as evidence-backed explanations.Source: §4.4 · p. 21

4.4.1 Scientific Demonstration Generation Benchmarks

• Scientific demonstration benchmarks evaluate domain-valid diagram translation and pedagogical explanation through molecular scripts, interactive artifacts, and theorem animations.The ChemDraw benchmark (Zhao et al., 2025b) tests molecular-diagram translation into Python using SMILES and chemistry libraries; EduVisBench (Ji et al., 2025) uses a pedagogy-informed rubric. InteractScience (Chen et al., 2025g) combines Programmatic Functional Testing with Visually-Grounded Qualitative Testing, while TheoremExplainBench (Ku et al., 2025a) assesses Python-generated theorem-explanation videos.Source: §4.4.1 · p. 21

4.4.2 Scientific Demonstration Generation Methods

• 400K multimodal images and 2.7M instruction-tuning samples are synthesized by CoSyn (Yang et al., 2025f), illustrating renderer-driven supervision alongside staged agentic explanation workflows.Renderer-driven approaches use executable rendering, domain libraries, model-in-the-loop synthesis, or runtime repair, but rendered appearance remains a proxy for scientific reasoning. EduVisAgent (Ji et al., 2025) structures objectives and interactive lessons; TheoremExplainAgent (Ku et al., 2025a) combines planning, coding, and Agentic RAG for Manim animations, with every stage requiring checks against the same scientific claim.Source: §4.4.2 · pp. 21–22
• Scientific demonstrations require inspectable scientific claims, because visually plausible outputs can conceal invalid mechanisms, incorrect equations, or unsupported relationships.Correctness spans numerical values, equations, domain constraints, executable code, interaction behavior, and pedagogical interpretation. Proposed validation combines domain validators, execution tests, simulation logs, provenance checks, and expert inspection, keeping parameters, package calls, code, tool outputs, and rendered states inspectable.Source: §4.4.2 · p. 22

5.1.1 SVG Code Generation Benchmarks

• SVG benchmarks distinguish text-conditioned semantic alignment from image-conditioned reconstruction, combining semantic, perceptual, and structural proxies that remain vulnerable to memorized primitives.Evaluation also considers edit success, code economy, and human preference; high scores on templated icon suites need not demonstrate robust vector reasoning.Source: §5.1.1 · pp. 22–23

5.1.2 SVG Code Generation Methods

• SVG methods must reconcile discrete commands with continuous geometry, motivating higher-level, dynamic programming frameworks rather than static path fitting.Differentiable rendering struggles with complex optimization, token sequences risk geometric precision loss, and LLM/RL systems add semantic representations, reasoning, and rendering feedback. Proposed frameworks expose parametric geometry, animation, interaction, groups, styles, and constraints, with separate tests of rendering, structure, behavior, edits, and reuse.Source: §5.1.2 · pp. 24–25

5.2.1 Diagram Code Generation Benchmarks

• Diagram benchmarks differ by input direction and target formalism, requiring graph-level checks because layout similarity can hide logical errors and synthetic leakage.Targets include diagram syntax, workflow JSON, pseudocode, programming-language code, UML, and Verilog, each exposing different logical failures. Text-to-diagram permits multiple valid layouts, while visual translation must recover relations; synthetic renderer templates can obscure whether models preserve the intended graph.Source: §5.2.1 · pp. 25–26

5.2.2 Diagram Code Generation Methods

• Diagram generation should prioritize logical correctness over visual polish, verifying topology, edge direction, branch conditions, typed relations, and domain constraints.Methods combine supervised fine-tuning, synthetic data, reasoning traces, and rendered feedback. Evaluation should test path reachability, branch equivalence, execution behavior, and targeted visual questions.Source: §5.2.2 · p. 26

5.3.1 CAD Code Generation Benchmarks

• CAD benchmarks extend command reconstruction toward executable, language-conditioned, understanding, and repair tasks, requiring separate checks of compilation, solids, constraints, edit propagation, and manufacturability.Settings include code commenting, symbolic-program semantic consistency, reference-image-guided repair, and CadQuery generation from prompts or three-view drawings. Because many resources accompany particular methods rather than independent suites, comparisons emphasize observable signals and validation gaps.Source: §5.3.1 · pp. 26–27

5.3.2 CAD Code Generation Methods

• CAD methods span procedural sequences, executable multimodal scripts, and execution-feedback pipelines, but valid syntax and plausible geometry do not establish editable design intent.Sequence vocabularies constrain complex boolean operations, assemblies, and long-range dependencies, while scripts broaden input modalities. Feedback pipelines use preference learning, reinforcement learning, compiler-based self-debugging, or program-image repair to address geometric execution and constraint failures.Source: §5.3.2 · pp. 27–28
• CAD generation should target verifiable parametric design, using compiler- and geometry-aware validators rather than judging rendered surface similarity alone.Validation should test boolean operations, solid closure, constraints, feature-tree dependencies, edit propagation, and manufacturability. Benchmarks should include complex parts, assemblies, multi-view drawings, revision instructions, and manufacturing constraints.Source: §5.3.2 · p. 28

5 Structured Graphics

• Structured graphics require structure-aware validation because plausible renderings can conceal lost object hierarchies, relation graphs, or parametric construction histories.Evaluation must test editability, graph reasoning, constraint propagation, compilation or execution, and downstream reuse.Source: §5 · p. 22, §5.3.2 · p. 28

• Generated code often serves as an intermediate trace, tool call, environment policy, or repair interface rather than solely a final rendered artifact.Source: §6 · p. 28
• Five settings stress process reliability: programmatic visual manipulation, video code generation, embodied control, visually grounded programming, and unified multimodal code generation.Source: §6 · p. 29
• These settings align most directly with programmatic tool-use and executable-policy formulations, while unified models span the full synthesis and acting space.Source: §6 · p. 29

6.1 Programmatic Visual Manipulation

• Programmatic visual manipulation treats code as an executable interface for inspecting images, with process faithfulness requiring intermediate visual states to support the final answer.In Thinking with Image (Su et al., 2025), operations include cropping, detecting, measuring, drawing, masking, plotting, and querying. Most methods augment visual reasoning with tools; direct programmatic solvers return execution results as answers.Source: §6.1 · p. 29

6.1.1 Programmatic Visual Manipulation Evaluation Signals

• Answer accuracy, executability, and process rewards cannot individually establish causal evidence use; stronger evaluation should inspect and replay grounded visual interventions.Proposed checks combine answer accuracy with operation replay, region grounding, evidence ablation, and counterfactual-image tests. Future methods should learn executable, inspectable, replayable visual actions tied to answer-relevant regions and evidence.Source: §6.1.1 · p. 29, §6.1.2 · p. 31

6.1.2 Programmatic Visual Manipulation Methods

• Predefined-tool systems offer bounded, inspectable reasoning traces, but their API vocabulary limits which visual evidence they can access.MM-React (Yang et al., 2023) routes decisions to vision experts, while Hydra (Ke et al., 2024) uses an RL controller to select reasoning paths. Unanticipated crops, spatial constructions, or operation compositions can leave decisive regions unexamined.Source: §6.1.2 · p. 29
• Video tool-use systems select temporal evidence through frame traversal, retrieval, memory, and progression displays, but answer accuracy cannot establish the selected evidence’s causal contribution.DoraemonGPT (Yang et al., 2024g) schedules spatial-temporal tools with MCTS, and MoReVQA (Min et al., 2024) separates video parsing from reasoning through external memory. VTimeCoT (Zhang et al., 2025d) exposes temporal progression using progress bars and highlighted moments.Source: §6.1.2 · p. 29
• Instruction tuning, manipulation reasoning, and process rewards supervise intermediate operations, yet executable or format-correct traces can still lack answer-relevant evidence.MLLM-Tool (Wang et al., 2025b) improves API selection through instruction tuning, while VISTA-R1 (Lu et al., 2025b) scales interleaved reasoning and tool execution in standardized environments. Skywork-R1V4 (Zhang et al., 2025g) learns planning-execution trajectories, and CodeV (Hou et al., 2025) rewards executable, evidence-consistent tool use.Source: §6.1.2 · p. 30
• Generative code reasoning synthesizes task-specific programs and new visual views, expanding operation coverage while making evidence relevance and faithfulness harder to verify.ViperGPT (Surís et al., 2023) composes vision-and-language APIs into Python subroutines, while ReFocus (Fu et al., 2025c) highlights or masks regions for multi-hop reasoning. Sketches, spatial indicators, and multi-turn analysis can create intermediate evidence when the original image is difficult to inspect.Source: §6.1.2 · pp. 29–30
• Mathematical plotting and formal geometric constructions provide external structure for checking intermediate artifacts, making traces more verifiable than plausible masks or sketches.CodePlot-CoT (Duan et al., 2025) generates executable plotting code for mathematical reasoning, while Geoint-R1 (Wei et al., 2025b) constructs auxiliary geometric elements with Lean4-style formalization.Source: §6.1.2 · pp. 30–31
• Native code-agent workflows could broaden visual operations beyond fixed APIs, but remain relevant only when they create, inspect, or validate answer-supporting visual evidence.Terminal commands, file operations, external libraries, runtime logs, and broader agent scaffolds could support task-specific operations. Richer access helps only when each operation changes the available evidence to support subsequent reasoning.Source: §6.1.2 · p. 31

6.2.1 Video Code Generation Benchmarks

• Code-to-Video benchmarks assess educational and presentation usefulness, but content fidelity can coexist with awkward pacing, weak synchronization, or discontinuous transitions.MMMC (Chen et al., 2025n) evaluates educational videos through TeachQuiz and model-based judgment. PresentEval (Shi et al., 2025) evaluates document-to-presentation videos through content fidelity and user studies, treating temporal quality indirectly.Source: §6.2.1 · pp. 31–32
• 115k video-code-observation triplets in Video2Code support recovering manipulation policies, but task-success evaluation does not establish motion fidelity.A successful recovered policy can discard velocity, contact timing, or recovery behavior present in the demonstration.Source: §6.2.1 · p. 32

6.2.2 Video Code Generation Methods

• Code-to-Video methods use executable scripts to constrain objects, layouts, scene order, and narration alignment, while renderers handle transition smoothness and continuous motion.Code2Video (Chen et al., 2025n) synthesizes Python scripts for educational videos, while PresentAgent (Shi et al., 2025) segments documents and synchronizes visual assets with narration. Theorem ExplainAgent (Ku et al., 2025b) retrieves and generates explanatory animations for scientific theorems.Source: §6.2.2 · p. 32
• Video-to-Code methods compress demonstrations into executable strategies, working best with ordered subgoals rather than fine-grained motion requirements.RoboPro combines VLMs and Code LLMs to synthesize manipulation policies from video, reducing expensive robot trajectory collection. JanusCoder (Sun et al., 2025b) encompasses static visual tasks and dynamic code-driven videos such as Manim animations, while highlighting that presentation videos and robotic policies encode different kinds of time.Source: §6.2.2 · p. 32
• Time-aware program abstractions should extend discrete visual states with trajectories, transition constraints, synchronization relations, and task-relevant motion dynamics.Because benchmarks remain sparse, this direction is an evaluation expansion rather than an established empirical trend. Scripts leave interpolation and pacing to rendering engines, while demonstration abstraction struggles with velocity, acceleration, contact timing, force response, and recovery.Source: §6.2.2 · p. 32
• Future video-code benchmarks should assess outcomes alongside trajectory consistency, event timing, narration-visual synchronization, motion smoothness, and preservation of task-relevant dynamics.Current measures can verify teaching quality, content fidelity, user preference, or task success without checking whether time evolves correctly.Source: §6.2.2 · p. 32

6.3.1 Embodied Control Benchmarks

• Embodied benchmarks evaluate executable activities, robot control logic, vision-conditioned function calls, and skill-code alignment, but incompletely expose failures, safety, recovery, and cross-embodiment transfer.VirtualHome (Puig et al., 2018) models household activities as programs; Code as Policies (CaP) (Liang et al., 2022) introduces RoboCodeGen with 37 tasks assessing spatial-geometric reasoning and hierarchical control-flow synthesis. OctoVerse (Yang et al., 2024b) spans photorealistic interiors, Minecraft, and GTA-V, while STEVE-21K (Zhao et al., 2024) provides 21k multimodal pairs of vision-environment data and skill-code triplets; simulator-specific assumptions and binary rewards can overstate robot transfer.Source: §6.3.1 · p. 33

6.3.2 Embodied Control Methods

• Code-centric methods either generate executable action programs directly or generate rewards and constraints that supervise or diagnose downstream policies.Code as Policies (Liang et al., 2022) and ProgPrompt expose API calls, assertions, and hierarchical logic, while EmbodiedCoder (Lin et al., 2025c) synthesizes trajectories through geometric parameterization of object point clouds. VLM-CaR (Venuto et al., 2024) instead generates executable dense rewards; reliability depends on controller APIs, camera-robot calibration, contact uncertainty, and feedback that repairs plans rather than merely reports failure.Source: §6.3.2 · p. 33
• Learning and feedback adapt code generation to physical dynamics, but Octopus (Yang et al., 2024b) illustrates how binary simulator feedback can conceal safety and recovery weaknesses.PACT (Wei et al., 2023) learns from sensorimotor sequences, RoboCodeX (Mu et al., 2024) uses multimodal post-training and iterative SFT, and Robotic Programmer converts video demonstrations into reusable procedures, reducing dependence on robot trajectory collection. Octopus (Yang et al., 2024b) uses RL with Environmental Feedback from simulator success and failure, which can also hide dependence on a particular simulator or robot body.Source: §6.3.2 · pp. 33–34

6.3 Embodied Control

• Embodied code should specify inspectable intent and constraints, while low-level controllers and closed-loop feedback handle physically grounded execution.The scope covers code-centric programs, rewards, specifications, and policy scripts rather than all embodied vision-language policies. Symbolic code must interface with sensors, controllers, and object geometry under continuous, stochastic dynamics constrained by morphology, calibration, contact, and safety.Source: §6.3 · p. 33, §6.3.2 · p. 34

6.3.2 Embodied Control Methods

• Future benchmarks should distinguish specification failures from controller execution, contact timing, safety, recovery, and changes in sensing, objects, tools, or robot morphology.Evaluation should test the boundary between symbolic intent and continuous control, preserving code as an interpretable task-and-constraint interface while leaving continuous adaptation to controllers.Source: §6.3.2 · p. 34

6.4.1 Visually Grounded Programming Benchmarks

• 253 tasks in HumanEval-V (Zhang et al., 2024a) require indispensable images and minimize text, moving algorithmic programming benchmarks from visual presence toward visual necessity.MMCode (Li et al., 2024a) contains 3.5k online-judge problems with 6.6k images, although many images are supplementary. ScratchEval (Fu et al., 2025b), TurtleBench (Rismanchian et al., 2025), and Code-Vision (Wang et al., 2025d) respectively test block-program perception, graphics reproduction, and flowchart reverse engineering; visual dependence still requires ablations or counterfactual images.Source: §6.4.1 · p. 34
• 617 JavaScript tasks from 17 repositories in SWE-bench MM (Yang et al., 2024e) cover visual scenarios including interactive mapping and web rendering.Building on SWE-bench, OmniGIRL (Guo et al., 2025b) adds multilingual and multimodal issues, while CodeV (Zhang et al., 2025e) selects 133 repository tasks explicitly requiring visual cues. These benchmarks approach real development, but final pass rates alone cannot rule out text-only shortcuts.Source: §6.4.1 · pp. 34–35

6.4.2 Visually Grounded Programming Methods

• Textual or symbolic intermediates connect visual understanding to stronger code generators, but can lose geometry, topology, grouping, and transient UI state.CodeVision (Wang et al., 2025d) converts flowcharts into Mermaid before program synthesis, reducing compilation failures. HumanEval-V (Zhang et al., 2024a) separates visual description from synthesis, while CodeV (Zhang et al., 2025e) produces fine-grained descriptions and structured summaries.Source: §6.4.2 · p. 35
• GUIRepair (Huang et al., 2025) generates reproduction scripts through Image2Code and captures post-patch screenshots through Code2Image, comparing them with issue images.SWE-agent M (Yang et al., 2024d;e) adds browser and image operations, but repair infrastructure becomes visually grounded only when observations inform localization, reproduction, or validation. Reliable feedback requires reproducible failures, correct localization, stable screenshots, and post-patch checks covering nearby states rather than only visible symptoms.Source: §6.4.2 · p. 35

6.4 Visually Grounded Programming

• The central bottleneck is preserving program-relevant visual constraints through code generation or patch refinement, rather than perception alone.Future systems should retain graph structure, DOM and browser state, rendered failures, and interaction traces connected to generated programs or patches.Source: §6.4 · p. 34, §6.4.2 · pp. 35–36

6.4.2 Visually Grounded Programming Methods

• Benchmarks should test whether visual evidence changes the expected code, because passing programs can exploit text-only descriptions or repository context.For repairs, screenshots, browser traces, terminal logs, reproduction scripts, localized edits, and post-patch executions should remain linked to show which visual failure motivated each change.Source: §6.4.1 · p. 35, §6.4.2 · p. 36

6.5.1 Unified Multimodal Code Generation Benchmarks

• Unified benchmarks test distinct forms of visual-code grounding; direct score comparisons require explicit objectives and controls for leakage, saturation, and metric agreement.Image2Struct (Roberts et al., 2024) evaluates structured-code extraction using visual-fidelity metrics including Cosine Inception Similarity (CIS) and Earth Mover Similarity (EMS). CoSyn (Yang et al., 2025f) synthesizes rendering code across 9 image categories before constructing question-answer pairs, emphasizing generated-data utility rather than extraction.Source: §6.5.1 · p. 36
• Dynamic benchmarks expand evaluation to executable interaction, image-dependent programming, and iterative visualization refinement, making the measured correctness signal essential to interpreting unified scores.ArtifactsBench (Zhang et al., 2025a) uses 1.8k queries across nine domains and checklist-guided MLLM judging of executability and interaction logic. InfiBench-V (Jiang et al., 2025a) covers 322 image-dependent questions across 13 programming languages, while VisPlotBench (Ni et al., 2025a) evaluates 888 visualization tasks across eight programming languages with multi-round self-debugging.Source: §6.5.1 · p. 36

6.5.2 Unified Multimodal Code Generation Methods

• Unified methods build cross-domain infrastructure, supervised task mixtures, and integrated models, but broader interfaces do not establish reusable visual-code primitives.GOT (Wei et al., 2024) extends OCR through three-stage training, while BigDoc (Rodriguez et al., 2024) supplies open-source multimodal document- and code-centric data. VisCoder2 (Ni et al., 2025a) introduces VisCode-Multi-679K for visualization generation and correction; VisCodex (Jiang et al., 2025a) merges a Code LLM with a VLM and introduces MCD-598k; JanusCoder (Sun et al., 2025b) integrates text- and vision-centric tasks with JanusCode-800K.Source: §6.5.2 · pp. 36–37
• VinciCoder (Zhao et al., 2025b) rewards coarse-to-fine visual similarity, while OCRVerse (Zhong et al., 2026) uses decoupled textual and visual rewards for reinforcement learning.OCRVerse unifies OCR and programmatic tasks in an end-to-end VLM. The proposed research priorities are data-efficient training, explicit primitive sharing, and execution-aware validation, with controlled transfer as the stronger evaluation criterion.Source: §6.5.2 · p. 37

6.5 Unified Multimodal Code Generation

• Unification remains unproven until controlled transfer distinguishes shared visual-code mechanisms from broad task acceptance and tests their effects on specialized constraints.Shared interfaces aim to support synthesis, editing, refinement, tool use, and interactive artifacts. Evaluation should examine reusable primitives and whether shared representations improve or harm specialized syntax, layout, interaction, and domain constraints.Source: §6.5 · p. 36, §6.5.2 · p. 37

6.5.2 Unified Multimodal Code Generation Methods

• Frontier tasks require verification of intermediate traces and state trajectories alongside final success; unified models additionally require held-out tests of cross-task transfer.Programmatic visual manipulation, video code generation, embodied control, and visually grounded programming motivate replay, temporal or physical diagnostics, visual ablations, and counterfactual inputs.Source: §6.5.2 · p. 37

7.1 Toward Multi-Signal Validation

• Validation should use role-specific diagnostic profiles and companion checks, rather than treating visual similarity, reference artifacts, or VLM preference as universal correctness signals.Profiles should distinguish visual similarity, execution success, textual correctness, semantic fidelity, structural validity, editability, and interaction correctness. Reward design should identify the optimized property, report companion validators, and separate training rewards from held-out reliability checks.Source: §7.1 · p. 38, Table 8 · p. 38

7.2 Toward Multi-State Verification

• Stateful visual-code systems should be evaluated as execution trajectories with intermediate transitions and recovery cases, rather than isolated renderings.Episodes should specify initial states, generated code or actions, intermediate observations, expected transitions, and validator outputs, exposing behavioral, scientific-semantic, or timing failures hidden by plausible endpoints. Checks should match the substrate—web DOM/state assertions, mobile operation traces or emulator gestures, video synchronization, and embodied controller diagnostics—while recognizing that mobile benchmarks currently use partial substitutes for native execution.Source: §7.2 · pp. 38–39

7.3 Toward Testing Cross-Task Transfer

• Unified models should demonstrate cross-task transfer of reusable visual-code skills, not merely accept more task formats or improve in-distribution performance.Tests should hold out skills, primitives, and compositions, comparing base, source-domain-augmented, and matched-size control mixtures on provenance-filtered target tasks while reporting positive and negative transfer. Counterfactual tests, modality ablations, and de-duplication checks should determine whether broader coverage conceals weakened specialized syntax, layout, or domain constraints.Source: §7.3 · p. 39

7.4 Toward Verifiable Agent Traces

• Visual-code agents need evidence logs connecting observations, tool use, code actions, and outcomes, because final success alone cannot establish trace faithfulness or causal responsibility.Each entry should record the observation, cited visual region or tool output, changed code region or action, expected validator improvement, replay result, and fallback or rollback decision. Logs should support replay, visual ablation, counterfactual inputs, permission controls, simulator or emulator guards, and human review, while attributing failures to perception, synthesis, execution, validator design, or unsafe action selection.Source: §7.4 · p. 39

8 Limitations

• Coverage is limited to public resources available during collection, so the taxonomy is an organizing view rather than a definitive boundary.Recent systems, closed-source deployments, and domain-specific tools may be missing.Source: §8 · p. 40
• The survey may overrepresent benchmark-proposing works and underrepresent deployed systems without public artifacts, while restricted or changing sources limit comparison reproducibility.These constraints include closed-source model reports, private evaluation sets, and rapidly changing arXiv releases.Source: §8 · p. 40
• Benchmarks observe different slices of correctness, limiting cross-method comparison and motivating within-domain comparisons rather than a universal ranking.The survey emphasizes common failure modes and reliability risks, including leakage, benchmark saturation, and judge sensitivity.Source: §8 · p. 40
• Cross-task transfer remains an agenda-setting discussion because current evaluations rarely isolate causal transfer and leave deployment-facing concerns underexplored.Source: §8 · p. 40

Broader Impact

• Visual plausibility can conceal serious errors in generated artifacts, code, and physical actions.Errors include incorrect chart data, lost document structure, invalid scientific mechanisms, broken interactions, insecure code, and unsafe physical actions.Source: §9 · p. 40
• Multimodal code intelligence can lower visual-programming barriers and help experts turn visual feedback into executable revisions, making artifacts easier to inspect, edit, and reuse.Users can express intent through screenshots, diagrams, sketches, videos, or natural language to obtain code for interfaces, charts, documents, demonstrations, SVG, CAD, or robot policies.Source: §9 · p. 40
• Deployment should combine provenance tracking, permission scopes, execution logs, domain validators, high-stakes human review, rollback mechanisms, and separation of model suggestions from user-approved actions.These safeguards address systems operating browsers, files, APIs, design tools, proprietary repositories, or robots.Source: §9 · p. 40
• Agentic systems raise privacy and safety risks through sensitive visual inputs, proprietary-code leakage or misuse, and embodied policies that behave differently outside simulation.Screenshots and design files may contain private information.Source: §9 · p. 40

Conclusion

• Multimodal code intelligence should be evaluated through rendering, execution, interaction, and replay evidence rather than visual plausibility alone.The survey identifies four verification directions: multi-signal validation, multi-state verification, cross-task transfer testing, and verifiable agent traces. Outputs should be executable, verifiable, editable, and grounded in intended visual evidence.Source: §10 · p. 40
• The survey organizes benchmarks and methods into Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks.Coverage includes interfaces, charts and documents, SVG/diagram/CAD programs, and code-mediated tool use or embodied policies.Source: §10 · p. 40
• Executable code provides an intermediate medium for grounding visual reasoning, invoking external tools, and addressing open-ended problems in dynamic environments.The survey addresses fragmented research on code as an executable interface for visual problem-solving.Source: §10 · p. 40