PaperlayerView exact revision on arXiv

Source-linked AI summary

How Far Can Coding Agents Go in Visual Generation?

Hengguang Zhou, Yuanhao Ban, Jiaqi Feng, Xiaohuan Pei, Justin Cui, Cho-Jui Hsieh

arXiv:2610.19025v1cs.CVcs.LG

TL;DR

General-purpose coding agents offer an alternative to image generators whose construction process is implicit in learned parameters. The paper evaluates their drawing, use of executable visual code for question answering, and behavioral evidence for shared scene representations; results show strong structural control alongside limits in perceptual realism and support for the proposed representation hypothesis.

  • Problem

    Specialized text-to-image models leave image construction largely implicit in learned parameters, motivating the question of how far general-purpose coding agents can go in visual generation.

    Source: §1 · p. 2

  • Method

    The paper evaluates zero-shot agents across drawing harnesses and a 200-prompt benchmark, tests reconstructed SVG code for visual question answering, and probes cross-harness consistency and spatial coherence.

    Source: §1 · p. 2

  • Results

    Agents show strong structural fidelity but limited perceptual realism, while reconstructed code retains question-answering information and behavioral probes provide complementary support for a shared scene-content hypothesis.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    Executable visual representations connect drawing and scene interpretation, while cross-harness consistency, pixel queries, and room-viewpoint coherence provide behavioral evidence for a possible shared organization of scene content.

    Source: §7 · p. 12

    The internal visual language-of-thought investigation is exploratory and behavioral, and most ablations use one primary agent with limited human verification.

    Source: §7 · p. 12

Abstract

from arXiv · show

Modern image generation largely relies on specialized generative models that leave the process of image construction implicit in learned parameters. We explore an alternative: general-purpose coding agents that construct images through executable programs. Our study of zero-shot drawing on AgentDrawBench, a 200-prompt benchmark, reveals a distinctive capability profile. Agents excel at preserving structure and satisfying precise visual constraints, achieving competitive results in domains where compositional accuracy matters more than photorealistic appearance, while remaining limited in perceptual richness. Their visual expressiveness varies with the drawing interface, and a qualitative study on reconstruction shows how procedural skills extracted from human demonstrations support different artistic media. We then examine code as an external visual language of thought: a structured, executable representation that agents can construct and inspect. We test it in visual question answering task and find reconstructed visual code retains substantial information for question answering and, in some domains, supports higher accuracy than the original image. These abilities raise a further question: do agents draw on a shared internal representation of scene content beyond a particular drawing program? We explore an internal visual language-of-thought hypothesis, in which a shared, compositionally structured representation supports visual operations across drawing languages. Cross-harness drawings and images assembled from independent single-pixel queries exhibit related layouts and tones. Agents also maintain self-generated room arrangements across viewpoints after visible drawings are removed. These results demonstrate that the coding agent would maintain a consistent and coherent world inside its imagine even before the harness gets involved and the generation starts.

1. Introduction

The paper asks whether general-purpose coding agents can construct images through executable programs rather than relying on specialized generators. It evaluates their drawing, use of code for visual reasoning, and possible shared scene representations across drawing interfaces.

Zero-shot drawing

  • Agents show strong structural fidelity and precise control but limited perceptual realism, with design and symbolic tasks yielding a 92.0% holistic win rate against gpt-image-1 for GPT-6-Astra with Canvas 2D.The paper introduces the 200-prompt AgentDrawBench to characterize this task-dependent profile.Source: §1 · p. 2

Code as an external visual language of thought

  • Reconstructed SVG code supports visual question answering at 69.0% accuracy versus 65.0% from its render and 74.0% from the original image.In academics and digital-world domains, the agent answers more accurately from code than from the original images.Source: §1 · pp. 2–3

Procedural guidance and drawing capabilities

  • Human-derived drawing procedures—including stroke operations, construction order, value grouping, and edge control—support qualitative reconstruction in Chinese ink, charcoal, watercolor, and oil painting.Fourfold test-time refinement brings modest structural improvement and a 50.5% holistic win rate against the standard run.Source: §1 · p. 2

Internal visual language of thought

  • The internal visual language-of-thought hypothesis proposes a compositionally structured representation of entities, attributes, and relations that can be expressed through different drawing interfaces.The paper frames consistency and coherence as the behavioral properties this hypothesis predicts.Source: §1 · pp. 2–3

Behavioral probes of shared scene content

  • Cross-harness drawings and independently queried pixels show related layouts and tones, while self-generated room arrangements remain largely consistent across viewpoints after drawings are removed.Fresh calls given only the furniture list and turn history show lower agreement across revisits and between views and floor plans.Source: §1 · pp. 2–3

2. Related Work

Related work covers learned text-to-image generators, executable visual code, and agentic systems that coordinate image creation through tools and iterative refinement.

Text-to-image generative models

  • Diffusion and other text-to-image models achieve substantial success while encoding image construction in learned model parameters [28].Related approaches include latent diffusion [26], cascaded diffusion [27], rectified-flow generation [9], classifier-free guidance [13], and unified image-language transformers [39].Source: §2 · p. 3

Code generation for visual content

  • Visual content is also represented by executable graphics programs, learned vector-generation systems, painting policies, and in-context stroke-by-stroke sketching [8, 25, 33, 31, 15, 19, 30].Source: §2 · p. 3

Agentic image-generation systems

  • Agentic image-generation systems select tools, gather external knowledge, and iteratively refine outputs, with recent work adding staged verification, search, and programmable media [32] [22] [6] [10] [2] [38].Source: §2 · p. 3

3. Experimental Setup

The setup compares programmable drawing interfaces and evaluates outputs on a 200-prompt benchmark using structural, perceptual, and holistic measures.

3.1. Harness Design

  • Drawing harnesses range from direct pixel assignments and geometric marks to executable Canvas 2D programs, declarative SVG documents, and GUI painting with visual feedback.The pixel and shape harnesses build images directly; Canvas 2D permits reusable procedures, while SVG describes a structured document and the GUI acts through painting tools.Source: §3.1 · p. 3
  • The setup also includes Python-built Blender scenes rendered with Cycles and reference-based stroke workflows guided by medium-specific procedures.Source: §3.1 · pp. 3–4

3.2. AgentDrawBench

  • AgentDrawBench contains 200 prompts across seven families, selected from four published benchmarks, and is evaluated for structural fidelity, perceptual realization, and holistic quality.The paper states it will publicly release the benchmark.Source: §3 · p. 3, §3.2 · p. 4
  • Structural fidelity uses prompt-specific checklist pass rates; perceptual quality compares outputs across ten dimensions, and holistic preference uses pairwise judgments with ties counted as half a win.Perceptual comparisons swap image positions to reduce position bias; holistic comparisons also use swapped positions and disclose excluded judgments.Source: §3.2 · p. 4

4. Coding Agents Are Zero-Shot Drawers

The zero-shot drawing evaluation finds that agent performance depends on the task: they can compete on structured design but trail the native image-generation baseline on naturalistic subjects and appearance.

Structural Fidelity

  • 85.8% overall structural pass rate for GPT-6-Astra exceeded gpt-image-1 by 17 percentage points; DeepSeek V4.1 Flash reached 89.5% on design and symbolic prompts, 31.4 points above gpt-image-1.In design and symbolic categories, Astra slightly exceeded the structural pass rate of gpt-image-2.Source: §4 · p. 4

Holistic Performance

  • 92.0% of design and symbolic prompts favored Astra with Canvas 2D over GPT-image-1, versus 13.0% for scenes, 6.0% for people and organic subjects, and 47.7% overall.The pattern favors agents on explicit structure and legible design, while GPT-image-1 is preferred for naturalistic scenes and people.Source: §4 · p. 5

Perceptual Realization

  • 77.6% and 74.8% typography win rates for Astra with Canvas 2D and SVG contrast with below-50% results across configurations on anatomy, perspective, light transport, material, and scene physics.The table reports perceptual win rates against GPT-image-1 across ten dimensions.Source: Table 2 · p. 5, §4 · p. 5

5. Code as an External Visual Language of Thought

The section treats executable visual code as a structured medium for constructing and interpreting images, then tests whether coding-agent abilities such as refinement and in-context learning extend to drawing. Results show code supports image construction and question answering, while procedural guidance can produce medium-specific reconstructions.

5. Code as an External Visual Language of Thought

  • Code is framed as an external visual language of thought: an executable medium for organizing, manipulating, and reasoning about visual content.The section tests code for image construction and for extracting visual information from SVG source.Source: §5 · p. 6, §5.1 · p. 6

5.1.1. Generation without Visual Feedback

  • 83.0% structural fidelity is achieved by blind Canvas 2D drawing, compared with 85.6% when the agent also receives rendered-image feedback.In the blind setting, the agent receives execution results but never sees the RGB canvas.Source: §5.1.1 · p. 6

5.2.1. Test-Time Scaling

  • 85.6% to 86.8% structural fidelity accompanies a fourfold refinement budget, but the extended run has only a 50.5% holistic win rate against the standard run.The comparison uses the MMRB2 rubric; the limited return motivates examining drawing interfaces and procedures.Source: §5.2.1 · p. 7

5.2.2. In-Context Learning for Visual Reconstruction

  • Human-demonstration procedures specify medium-specific marks and construction workflows, and qualitative reconstructions retain recognizable compositions with distinct ink, charcoal, watercolor, and oil effects.The workflows guide construction and revision from broad forms toward local detail; drawing code cannot sample, trace, blit, or filter reference pixels.Source: §5.2.2 · pp. 7–8

6. The Internal Visual Language-of-Thought Hypothesis

The paper tests whether visual structure recurs across drawing interfaces, independently queried pixels, and changing viewpoints, as behavioral evidence for a hypothesized shared internal representation. Across these probes, coarse layouts and scene arrangements show consistency, though pixel-query details remain unresolved.

6. The Internal Visual Language-of-Thought Hypothesis

  • The hypothesis treats drawings from different harnesses as realizations of shared scene content, and defines consistency as visual agreement and coherence as preserved structure through change.The probes examine cross-harness drawings, independent pixel-color queries on a 128 × 128 canvas, and room arrangements across viewpoints and revisits.Source: §6 · p. 8

6.1.1. Cross-Harness Consistency

  • Cross-harness 2D drawings reached mean layout correlation 0.60, palette cosine 0.81, and DreamSim 0.19, close to independent SVG runs at 0.66, 0.84, and 0.14.Mouse-driven GUI drawings also fell within the 2D cluster (0.56, 0.79, 0.22); Blender was less similar to 2D outputs but more similar than different-prompt drawings.Source: §6.1.1 · pp. 8, 10, Figure 7 · p. 10

6.1.2. Independent Queries, Consistent Images

  • Independently queried pixels assembled into a recognizable Mona Lisa composition, with compatible broad regions and tones despite noisy boundaries and unresolved facial features.Each pixel response came from a separate fresh context without access to other answers, supporting consistency among independently generated parts.Source: §6.1.2 · p. 10

6.2. Coherence

  • With earlier drawings deleted, room revisits retained Jaccard 0.99 and relation agreement 0.96, while wall views matched the agent’s plan on 99% of pieces.When the furniture list was also deleted, Jaccard was 0.88 and plan agreement 0.99; the table reports results across four conditions and 30 rooms.Source: §6.2 · p. 11, Table 4 · p. 11

7. Conclusion

The conclusion presents coding agents as capable of drawing and reasoning with executable visual representations, while identifying structural accuracy alongside limits in perceptual richness. It treats behavioral evidence across interfaces, pixel queries, and room views as support for—not proof of—a shared organization of scene content.

7. Conclusion

  • Coding agents draw zero-shot with strengths in structural accuracy and compositional constraints, but persistent limitations in perceptual richness.Their visual programs also let agents construct and interpret scene information, while cross-harness consistency, independent pixel queries, and self-furnished-room coherence provide behavioral evidence for an internal visual-language-of-thought hypothesis.Source: §7 · p. 12
  • The internal visual-language-of-thought investigation is exploratory and behavioral, with evidence limited by selected agents, interfaces, transformations, and probes.Most ablations use one primary agent, automated judgments receive limited human verification, and pixel queries cover five subjects.Source: §7 · p. 12

A.1. The harnesses

The appendix specifies drawing interfaces and procedural workflows, then details prompts, budget controls, and run-recording practices. The protocols vary in feedback, rendering cost, and interaction mechanics.

A.1. The harnesses

  • The harnesses range from pixel and geometric-mark tools to Canvas 2D, SVG, GUI painting, and Blender, with differing execution and feedback capabilities.Blender scripts rebuild scenes without external assets; the GUI may return screenshots, while code harnesses use executable programs or vector documents.Source: §A.1 · p. 15
  • Human demonstrations inform medium-specific stroke workflows, but these qualitative reconstructions do not establish controlled improvement on AgentDrawBench.The described procedures adapt ordering and marks for oil, watercolor, charcoal, and Chinese ink; reference pixels cannot be sampled, traced, blitted, or filtered.Source: §A.1 · pp. 15–16

A.2. System-prompt skeleton and turn mechanics

  • With explicit ordered tool-call instructions, pilot runs issued 20–64 calls per turn and none hit the cap; without them, 3 of 9 hit the cap unfinished.The prompt sentence specifies that calls within a turn execute in listed order.Source: §A.2 · p. 16

A.3. Imposed-budget mode

  • Dollar-matched arms enforce minimum and maximum provider-reported spend, while image-derived character-grid feedback is excluded from no-inspection arms.Runs below the floor are nudged to continue; the ceiling is checked before each turn.Source: §A.3 · p. 17

A.4. Run records, resumability, and cost accounting

  • Run records preserve final images, sources where applicable, intermediate frames, interaction traces, prompts, judge outputs, and usage and cost metadata.Batch generation and judging skip completed records when resumed.Source: §A.4 · p. 17

B. WorldBench Selection and Paired Answer Analysis

This analysis uses a frozen, proportionally stratified sample of 200 WorldBench image–question items to compare answers across original images, code, and rendered reconstructions. Paired transitions show reconstruction-related answer changes, while domain rankings remain descriptive.

B. WorldBench Selection and Paired Answer Analysis

  • The frozen sample contains 200 items drawn randomly and proportionally across seven WorldBench domains, with its manifest defining the exact selection.Domain allocations range from 17 items in Agents and Documents/Charts/Tables to 50 in Objects.Source: B. WorldBench Selection and Paired… · p. 17
  • 32 original-to-render losses and 14 reverse gains yield a net decrease of 18 correct answers, or 9 percentage points.The loss rate is conditional on items answered correctly from the original image.Source: B. WorldBench Selection and Paired… · p. 17
  • The correctness census reports 107 items correct in all three conditions, 30 wrong in all three, and 63 mixed.Each item label gives original, code, and render correctness in that order.Source: Figure 10 · p. 18
  • Original-to-code has 30 losses and 20 gains, while code-to-render has 19 losses and 11 gains; these paired transitions do not establish absent visual information.Small domain samples make rankings descriptive, with no claim of statistical significance.Source: B. WorldBench Selection and Paired… · p. 18

C. Independent Pixel Queries: Additional Diagnostics

Independent single-pixel queries recover spatially structured images and coarse composition, but not fine structure. Repeated queries at the same coordinate are often inconsistent, though their returned colors differ only modestly.

C. Independent Pixel Queries: Additional Diagnostics

  • Across five subjects, pixel-query images have luminance Moran’s I of 0.85–0.96, versus 0.01 for shuffled colors.The assembled pictures also correlate in luminance 0.51–0.88 with whole-image Canvas 2D drawings of the same subjects.Source: C. Independent Pixel Queries:… · p. 18
  • Assembled pixel-query images share coarse composition with whole-image Canvas 2D drawings of the same subjects.Luminance correlations range from 0.51 to 0.88.Source: C. Independent Pixel Queries:… · p. 18
  • Re-asking a coordinate returns the identical color only 2–38% of the time, with differences of about 10 of 255 per channel; facial features and branch leaves do not survive.Source: C. Independent Pixel Queries:… · p. 18

D.1. Self-Furnished Rooms

Agents choose furniture layouts for 30 rooms and revisit them across viewpoints, including conditions that remove visible drawings. The evaluation measures consistency against each agent’s own chosen arrangement rather than an external ground truth.

D.1. Self-Furnished Rooms

  • For 30 ProcTHOR-10K validation rooms, the agent chooses furniture positions from room descriptions and draws an initial wall view before six turns and a final top-down plan.The furniture list contains 7–14 pieces, but no positions.Source: §D.1 · p. 18
  • Four conditions retain prior SVGs, stub them, stub both SVGs and the furniture list, or use fresh calls with only the list and turn history.Every condition covers all 30 rooms.Source: §D.1 · p. 18
  • Scoring compares revisited views with the agent’s own chosen layout using piece-set, geometric-relation, placement, wall–plan, and within-wall order agreement.Permuted plan wall assignments provide a chance reference; the measures assess persistence and coordination, not accuracy against an external arrangement.Source: §D.1 · p. 18

Related papers

Loading 2610.19025v1…