PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning

Shaofeng Yin, Jiaxin Ge, Zora Zhiruo Wang, Chenyang Wang, Xiuyu Li, Michael J. Black, Trevor Darrell, Angjoo Kanazawa, Haiwen Feng

arXiv:2601.11109v3cs.CVcs.AIcs.GR

TL;DR

Reconstructing complex images as editable programs remains difficult for vision-language models with limited fine-grained spatial grounding. VIGA iteratively couples code generation with rendering and visual inspection; the authors report gains over existing baselines across diverse tasks.

  • Problem

    Reconstructing complex scenes as symbolic programs remains difficult, while vision-language models lack fine-grained spatial grounding in one-shot settings.

    Source: §1 · p. 2

  • Method

    VIGA is a training-free agent that iteratively synthesizes programs, renders visual states, inspects discrepancies, and edits scenes.

    Source: §1 · pp. 2–3

  • Results

    VIGA delivers substantial empirical gains across diverse tasks and significant improvements over existing baselines.

    Source: §1 · p. 4

  • Takeaways & Limitations

    VIGA is presented as a foundational step toward fully automated scene generation for complex, long-horizon inverse graphics.

    Source: §5 · p. 15

    Performance is bounded by the underlying vision-language models’ spatial perception and context-window constraints in extremely long sequences.

    Source: §5 · p. 14

Abstract

from arXiv · show

Vision-as-inverse-graphics, the concept of reconstructing images into editable programs, remains challenging for Vision-Language Models (VLMs), which inherently lack fine-grained spatial grounding in one-shot settings. To address this, we introduce VIGA (Vision-as-Inverse-Graphics Agent), an interleaved multimodal reasoning framework where symbolic logic and visual perception actively cross-verify each other. VIGA operates through a tightly coupled code-render-inspect loop: synthesizing symbolic programs, projecting them into visual states, and inspecting discrepancies to guide iterative edits. Equipped with high-level semantic skills and an evolving multimodal memory, VIGA sustains evidence-based modifications over long horizons. This training-free, task-agnostic framework seamlessly supports 2D document generation, 3D reconstruction, multi-step 3D editing, and 4D physical interaction. Finally, we introduce BlenderBench, a challenging visual-to-code benchmark. Empirically, VIGA substantially improves accuracy compared with one-shot baselines in BlenderGym (35.32%), SlideBench (117.17%) and our proposed BlenderBench (124.70%).

1 Introduction

VIGA reframes vision-as-inverse-graphics as interleaved multimodal reasoning: an agent iteratively writes code, renders scenes, and uses visual evidence to guide edits. The training-free framework spans 2D document layout, 3D reconstruction and editing, and 4D interaction, with reported gains across three benchmarks.

Introduction

  • Complex inverse graphics needs active observation and geometric refinement: gradient-based methods can be trapped by local minima, while one-shot VLM code generation lacks fine-grained visual grounding for precise verification and refinement.The paper motivates combining the complementary strengths of visual grounding and semantic reasoning.Source: §1 · p. 2
  • VIGA interleaves symbolic program generation with rendered visual perception in a code-render-inspect loop, so code logic and visual evidence cross-verify one another.The agent synthesizes a program, renders the scene, inspects it through read-only interfaces, and uses feedback to guide the next step.Source: §1 · p. 2, Figure 2 · p. 3
  • VIGA sustains long-horizon, evidence-based edits with a high-level semantic skill library and evolving multimodal context memory.Read-only interfaces gather multi-angle evidence before execution interfaces apply targeted edits; sliding-window memory retains recent plans, code diffs, and render histories while bounding context.Source: §1 · pp. 2–4
  • +35.32% on BlenderGym, +117.17% on SlideBench, and +124.70% on BlenderBench are reported as VIGA’s improvements across diverse benchmarks.BlenderBench is a 27-task suite covering spatial adjustments, progressive editing, and compositional generation.Source: §1 · pp. 3–4
  • The zero-shot, training-free framework requires neither auxiliary predictors nor differentiable renderers and generalizes across 2D layout, 3D reconstruction and editing, and 4D physical interaction.The paper introduces BlenderBench as a challenging benchmark for stress-testing agentic inverse-graphics capabilities.Source: §1 · pp. 3–4

2 Related Work

The related work frames VIGA as a unified approach to visual reasoning and inverse graphics, addressing limitations in text-centric agents and separate code-based analysis and synthesis methods.

LLM-based Agent

  • Text-centric LLM agents—including browser and computer agents—struggle with visual input and often produce low-quality visuals in 2D and 3D environments.VIGA addresses this gap with interleaved multimodal reasoning and adaptive agent memory for multi-turn verification and refinement.Source: §2 · p. 4

Visual Analysis or Synthesis with Code

  • Visual programming methods [43] generate code in one pass, while instruction-driven synthesis methods [59] [63] do not effectively ground scenes back to editable code.VIGA unifies analysis and synthesis through an iterative loop that executes programs, verifies generated visuals, and edits code.Source: §2 · p. 4

Vision-as-Inverse-Graphics through Programs

  • Inverse-graphics research seeks interpretable scene programs, but prior LLM-based methods [29] rely on synthetic data or domain-specific verification and task-specific workflows.Earlier approaches include PICTURE’s probabilistic programming for 2D and 3D scenes and Wu et al.’s structured markup inference, which struggled to generalize beyond abstract scenes.Source: §2 · pp. 4–5
  • VIGA combines code generation, active visual verification, semantic skills, and evolving multimodal memory in a training-free, task-agnostic framework for long-horizon inverse-graphics tasks.It adapts zero-shot across tasks from compositional 3D scene editing to dynamic 4D physical simulations, rather than relying on rigid, task-specific pipelines.Source: §2 · p. 5

3 Method

VIGA couples symbolic program generation with visual inspection and refinement, supported by evolving multimodal memory and high-level semantic skills.

3.1 Interleaved Multimodal Reasoning Framework

  • VIGA iteratively executes generated programs, explores rendered scenes, and turns visual discrepancies into feedback for targeted code edits.The visual and symbolic spaces cross-verify one another in a self-correcting loop rather than operating as isolated steps.Source: §3.1 · pp. 6–7

3.2 Evolving Multimodal Memory for Long-Horizon Reasoning

  • A fixed-size sliding window retains recent multimodal history to support long-horizon refinement while limiting context bloat.The latest program encapsulates prior successful geometric edits, making older intermediate traces redundant; without the window, generation quality collapses over long horizons.Source: §3.2 · pp. 8–9, Figure 5 · p. 8

3.3 Skill Library as a Semantic Interface for Graphics

  • High-level semantic skills compress a brittle low-level graphics action space and separate read-only observation from persistent scene modification.The agent can make multi-turn spatial queries before translating its grounded understanding into deliberate parametric changes.Source: §3.3 · pp. 9–10, Table 1 · p. 9

4 Experiments

Experiments show VIGA generalizes across 2D, 3D, and 4D visual tasks, and improves results on BlenderBench, BlenderGym, and SlideBench. Its interleaved reasoning also supports multi-step agentic behavior and single-step generation and editing.

4.1 Qualitative Results across Diverse Tasks

  • VIGA reconstructs 3D scenes from images or sketches, simulates 4D physical dynamics from an image and prompt, and generates 2D presentation slides.It retrieves and positions external meshes for scenes, configures physical parameters for simulations, and uses a programmatic execution engine to iteratively plan and refine slide layouts, text, and images.Source: §4.1 · pp. 10–11, Figure 6 · p. 11, Figure 7 · p. 11

4.2 BlenderBench: Evaluating Agentic Behaviors

  • BlenderBench tests active spatial exploration, iterative attribute editing, and exploratory editing that combines both challenges.Its three progressive tracks are Camera Adjustment, Fixed-View Editing, and Exploratory Editing; evaluation uses PL, N-CLIP, and VLM Score.Source: §4.2 · pp. 11–12, Figure 9 · p. 13
  • In Task 1, GPT-4o best-of-4 raises VLM Score from 1.44 to 3.25; in Task 3, VIGA improves VLM Score by +187.50% over BlenderAlchemy.BlenderAlchemy often overwrites earlier edits, whereas VIGA tracks states for sequential consistency; with Qwen, VIGA gains +312.20% over BlenderAlchemy under best-of-1.Source: §4.2 · pp. 12–13, Table 2 · p. 13

4.3 Evaluation on Existing Benchmarks

  • VIGA achieves average improvements of +124.70% on BlenderBench, +35.32% on BlenderGym, and +117.17% Overall on SlideBench.The gains are reported across BlenderBench evaluation tracks and model settings, BlenderGym settings, and foundation models on SlideBench.Source: §4.2 · p. 12, Table 2 · p. 13, Table 3 · p. 14
  • On SlideBench, VIGA’s interleaved loop recovers from syntax or rendering errors by parsing execution logs and correcting faulty code.On BlenderGym, it localizes and modifies fine-grained attributes while preserving unedited scene elements.Source: §4.3 · p. 14, Table 4 · p. 14, Table 3 · p. 14

5 Conclusion

VIGA couples program synthesis with active visual verification to perform inverse graphics across 2D, 3D, and 4D domains. The conclusion presents BlenderBench as a benchmark for multi-step spatial reasoning and progressive scene editing, while noting limits from VLM spatial perception and long-context constraints.

5 Conclusion

  • VIGA maps abstract spatial concepts to executable code across 2D, 3D, and 4D by interleaving program synthesis with active visual verification.The framework combines an evolving memory and semantic skill library in a unified, training-free agent for complex, long-horizon inverse graphics.Source: §5 · pp. 14–15
  • BlenderBench is introduced to quantify multi-step spatial reasoning and progressive scene editing in dynamic agentic behaviors.Source: §5 · p. 14
  • VIGA’s performance is bounded by its underlying VLMs’ spatial perception and context-window constraints in extremely long sequences.The framework is described as extensible, with future foundation models and external generative tools such as Meshy, Tripo, and SAM-3D [51] able to plug into its closed-loop paradigm.Source: §5 · pp. 14–15

Supplementary Material · A Evaluation Settings · A.1 Quantitative Settings

The evaluation settings define VIGA’s iteration and candidate-selection procedures, describe BlenderBench’s domain-shifted tracks and success metrics, and use established protocols for BlenderGym and SlideBench.

A.1 Quantitative Settings

  • VIGA uses max_iterate_round=10 to limit iterations to the search budget.In best-of-N evaluation, it explores multiple reasoning branches and propagates the candidate modification with the highest CLIP score.Source: §A.1 · p. 21
  • The one-shot baselines use standard VLM system prompts from for fair comparison.Source: §A.1 · p. 21
  • GPT, Claude, and Gemini use native tool_calls, while Qwen is prompted for strict JSON arguments to mitigate formatting errors.The JSON approach bypasses brittle API wrappers.Source: §A.1 · p. 21
  • BlenderBench tests Camera Adjustment, Fixed-View Editing, and Exploratory Editing under reference-image domain gaps such as style transfers and guiding arrows.These shifts are intended to test extraction of geometric intent rather than pixel-level pattern matching.Source: §A.1 · p. 21
  • BlenderBench measures generation quality with Photometric Loss (PL) and Negative-CLIP Score (N-CLIP), and task success with VLM Score.VLM Score covers task completion, spatial accuracy, detail accuracy, and visual quality, prioritizing structural layout over task-irrelevant stylistic artifacts.Source: §A.1 · p. 21
  • BlenderGym evaluates single-step 3D scene editing, while SlideBench evaluates programmatic 2D document generation using their established protocols and metrics.BlenderBench comparisons include one-shot generation and the memory-less iterative baseline BlenderAlchemy; both iterative methods use best-of-N evaluation.Source: §A.1 · p. 21

A.2 Qualitative Settings

For complex 3D reconstruction and 4D physics simulation, the setup expands VIGA’s memory and uses GPT-5; observed model differences include reduced-fidelity reconstruction by Claude-Sonnet-4 and Gemini-2.5-Pro, and active spatial-exploration failures by GPT-4o.

A.2 Qualitative Settings

  • Complex 3D reconstruction and 4D physics simulation use context_window=12, max_iterate_round=100, and GPT-5 as the base VLM.Claude-Sonnet-4 and Gemini-2.5-Pro completed reconstruction-from-scratch tasks with reduced visual fidelity.Source: §A.2 · p. 22
  • GPT-4o frequently fails to adjust the camera to observe global scene layout, preventing completion of most complex reconstruction tasks.The tasks expose generational VLM differences as observable failures in active spatial reasoning.Source: §A.2 · p. 22

B Extended Results · B.1 Extended Qualitative Results: 3D Assets and 4D Videos

The supplementary results provide interactive 3D scene files and 4D simulation videos to assess VIGA’s geometric and temporal fidelity. They show compositional 3D construction and physically constrained, temporally consistent interactions.

B.1 Extended Qualitative Results: 3D Assets and 4D Videos

  • The supplementary archive adds qualitative material for evaluating the geometric and temporal fidelity of VIGA’s generations.Source: §B.1 · p. 22
  • Raw .blend files let reviewers freely navigate reconstructed 3D scenes and inspect their fine-grained geometry, spatial layouts, and materials.Source: §B.1 · p. 22
  • Interactive inspection shows VIGA constructs robust, compositional 3D assets rather than merely overfitting to 2D pixel patterns.Source: §B.1 · p. 22
  • For 4D physics tasks, the archive supplies raw .blend files and rendered .mp4 videos because static frames cannot adequately capture temporal dynamics.Source: §B.1 · p. 22
  • The videos showcase temporal consistency and accurate physical interactions, including multi-object collisions, gravity, and glass shattering.Source: §B.1 · p. 22
  • VIGA executes programmatic commands in a physics engine to enforce physical constraints over time.Source: §B.1 · p. 22
  • This approach bypasses temporal flickering and physical hallucinations associated with traditional diffusion-based video generation models.Source: §B.1 · p. 22

B.2 Extended Quantitative Results · C System Prompts

VIGA reduces Gemini-2.5-Pro’s BlenderBench Task 2 photometric loss, while the extended evaluation covers four models and C System Prompts describe a unified, phase-specific prompting and evaluation scheme.

B.2 Extended Quantitative Results

  • The extended best-of-1 evaluation compares VIGA with the one-shot baseline across four models on BlenderGym and BlenderBench.It includes Claude-Sonnet-4 and Gemini-2.5-Pro alongside the models highlighted in the main paper; Tables 5 and 6 report PL and N-CLIP, with BlenderBench also reporting VLM Score.Source: §B.2 · pp. 22–23, Table 5 · p. 23, Table 6 · p. 23
  • 10.14 to 1.99: Gemini-2.5-Pro’s BlenderBench Task 2 Fixed-View Editing Photometric Loss falls under VIGA; Claude-Sonnet-4 also recovers from severe Task 1 camera misalignment.The reported Gemini loss change is from the one-shot baseline to VIGA; the passage does not give a numerical value for Claude-Sonnet-4’s recovery.Source: §B.2 · p. 23, Table 6 · p. 23

C System Prompts

  • Unlike prior methods’ task-specific prompt engineering, VIGA uses a concise unified prompt structure, changing only a few domain-specific keywords across tasks.The passage cites prior VLM-based procedural generation methods [29] [46] as relying heavily on complex, task-specific prompts.Source: C System Prompts · p. 24
  • For code synthesis, the agent is instructed to reproduce the target scene through successive tool calls, each paired with concise reasoning about the call’s purpose.The synthesis role is an expert, tool-driven coding agent given a target image and optional text description.Source: C System Prompts · p. 24
  • For visual inspection, the agent uses read-only perceptual tools to compare the render with the target and propose actionable code modifications over multiple rounds.The inspector receives scene context, prior edits and reasoning, full code, and current render(s).Source: C System Prompts · pp. 24–25
  • The BlenderBench VLM Score evaluator prompt dynamically supplies the scoring criterion and rubric, exact task description, and integer rating scale.The scale’s maximum is typically set at runtime.Source: C System Prompts · p. 25
  • The evaluator prioritizes task-relevant geometry, layout, object placement, and lighting over style artifacts, and returns an integer score with a justification.It uses the target as the layout and camera reference, ignores style-transfer artifacts that do not affect objectives, and is instructed to use the full 0–5 score spectrum.Source: C System Prompts · p. 25

D Semantic Skill Library Schemas

VIGA’s semantic skill library replaces low-level graphics APIs with phase-specific interfaces that compress the action space and insulate the agent from execution trivialities. The schemas divide scene construction and editing from read-only visual inspection.

Semantic skill library

  • The library separates execution-oriented state modification during code synthesis from read-only, information-seeking state observation during visual inspection.This phase-based split supports the interleaved reasoning loop.Source: D Semantic Skill Library Schemas · pp. 26, 29

State Modification Interfaces

  • Scene-construction skills guide an initial scene description and staged plan, then support asset acquisition and programmatic scene edits.The plan proceeds through preparation, rough layout, intermediate asset placement, and refinement.Source: D Semantic Skill Library Schemas · pp. 26–28

State Observation Interfaces

  • Observation skills provide multi-view renders and let the agent adjust its viewpoint or inspect scene differences across camera, objects, layout, environment, and dynamic animation.For dynamic scenes, the verifier recommends inspecting critical frames and checking motion and inter-object interactions.Source: D Semantic Skill Library Schemas · pp. 29, 32

Related papers

Loading 2601.11109v3…