Source-linked AI summary
Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning
Shaofeng Yin, Jiaxin Ge, Zora Zhiruo Wang, Chenyang Wang, Xiuyu Li, Michael J. Black, Trevor Darrell, Angjoo Kanazawa, Haiwen Feng
TL;DR
Vision-Language Models struggle to reconstruct complex images as editable programs because one-shot settings lack fine-grained spatial grounding. VIGA addresses this with an interleaved code-render-inspect agent using semantic skills and evolving multimodal memory, and reports substantial gains across diverse benchmarks.
Problem
Vision-Language Models lack fine-grained spatial grounding for accurately reconstructing complex scenes as editable programs in one-shot settings.
Method
VIGA interleaves symbolic program synthesis, rendering, visual discrepancy inspection, semantic skills, and evolving multimodal memory in a training-free agent.
Results
+124.70% on BlenderBench, +117.17% on SlideBench, and +35.32% on BlenderGym are reported improvements from VIGA across diverse benchmarks.
Takeaways & Limitations
VIGA provides a unified training-free framework for 2D document layout editing, 3D reconstruction, multi-step scene editing, and 4D physical interaction.
Abstract
from arXiv · showhide
Vision-as-inverse-graphics, the concept of reconstructing images into editable programs, remains challenging for Vision-Language Models (VLMs), which inherently lack fine-grained spatial grounding in one-shot settings. To address this, we introduce VIGA (Vision-as-Inverse-Graphics Agent), an interleaved multimodal reasoning framework where symbolic logic and visual perception actively cross-verify each other. VIGA operates through a tightly coupled code-render-inspect loop: synthesizing symbolic programs, projecting them into visual states, and inspecting discrepancies to guide iterative edits. Equipped with high-level semantic skills and an evolving multimodal memory, VIGA sustains evidence-based modifications over long horizons. This training-free, task-agnostic framework seamlessly supports 2D document generation, 3D reconstruction, multi-step 3D editing, and 4D physical interaction. Finally, we introduce BlenderBench, a challenging visual-to-code benchmark. Empirically, VIGA substantially improves accuracy compared with one-shot baselines in BlenderGym (35.32%), SlideBench (117.17%) and our proposed BlenderBench (124.70%).
1 Introduction
VIGA addresses the difficulty of inverse graphics by interleaving symbolic code generation with active visual verification. Its semantic skills and evolving multimodal memory support evidence-based edits across diverse tasks, with substantial benchmark gains.
- Motivation: Inverse graphics requires reconstructing images as editable programs, but accurate reconstruction of complex scenes remains difficult for one-shot generation.The target representation includes assets, layout, objects, materials, and lighting for applications such as robot training and interpretable digital twins.
- Approach: VIGA couples discrete program synthesis with continuous visual perception in a code-render-inspect loop that lets symbolic logic and visual evidence cross-verify each other.The agent synthesizes programs, renders scenes, inspects discrepancies, and uses visual feedback to guide subsequent edits.
- Approach: VIGA uses read-only perceptual interfaces for multi-angle observation and targeted execution interfaces for programmatic scene modification.This separates scene observation from modification while enabling discrepancy localization before edits.
- Architecture: Its high-level semantic skill library and evolving multimodal context memory sustain long-horizon cross-modal reasoning without finetuning.The architecture combines semantic interfaces with retained multimodal history for evidence-based modification.
- Scope: VIGA supports 2D document layout editing, 3D reconstruction, multi-step scene editing, and 4D physical interaction in a zero-shot, training-free framework.The framework is described as task- and model-agnostic and is evaluated under a unified protocol.
- Results: +124.70% on BlenderBench, +117.17% on SlideBench, and +35.32% on BlenderGym are reported improvements delivered by VIGA across diverse benchmarks.The paper also introduces BlenderBench, a 27-task suite covering spatial adjustments, progressive editing, and compositional generation.
2 Related Work
Prior work includes text-centric agents and scene-program generation methods, but these approaches have limitations in visual processing, generalization, or task specificity. VIGA instead unifies symbolic code generation with active visual verification through a versatile semantic skill library.
- LLM-based Agents: Most LLM-based agents operate effectively on text but remain limited when tasks require processing visual information.The related work discusses shortcomings in visually grounded tasks such as selecting products by style.
- Inverse Graphics: Earlier image-to-code methods achieved success on abstract multi-object scenes but struggled to generalize to real images.The cited approach inferred structured markup code for renderable scenes, including clip-art and Minecraft-like settings.
- VIGA: VIGA differs by unifying training-free code generation and active visual verification through dynamically invoked semantic tools rather than rigid task-specific pipelines.The agent probes the environment and executes changes using a versatile skill library.
3 Method
VIGA performs inverse graphics through an interleaved loop that plans symbolic actions, renders executable programs, inspects visual outcomes, and feeds discrepancies into later edits. High-level semantic interfaces and evolving multimodal memory support fine-grained grounding and long-horizon reasoning.
- Interleaved Multimodal Reasoning Framework: VIGA formulates inverse graphics as an iterative process that synthesizes an executable program rendering a state visually consistent with the target.
- Interleaved Multimodal Reasoning Framework: At each iteration, the agent plans symbolic actions from the target and current memory before synthesizing an executable program.
- Interleaved Multimodal Reasoning Framework: Read-only spatial tools let the agent locate inconspicuous targets and detect fine-grained discrepancies before mapping observations to precise code parameters.
- Interleaved Multimodal Reasoning Framework: The executed program renders a visual state that the agent actively explores, compares with the target, and converts into actionable feedback for subsequent code edits.
- Evolving Multimodal Memory for Long-Horizon Reasoning: The structured multimodal memory archives plans, programs, rendered states, and feedback, while a sliding window limits context bloat during long-horizon refinement.
- Skill Library as a Semantic Interface for Graphics: A high-level skill library compresses the graphics action space into semantic interfaces that separate non-destructive state observation from persistent state modification.
4 Experiments
VIGA is evaluated across 2D, 3D, and 4D generation and editing tasks, including the multi-step BlenderBench benchmark. The framework produces broad qualitative results and substantial gains over baselines on established and proposed benchmarks.
- Qualitative Results across Diverse Tasks: VIGA demonstrates broad adaptability across 2D document design, 3D scene reconstruction, multi-step scene editing, and 4D physical interaction.
- BlenderBench: BlenderBench evaluates active spatial exploration, fixed-view attribute editing, and exploratory editing through three progressive tracks.
- BlenderBench: +187.50% VLM Score improvement is reported for GPT-4o on BlenderBench, best-of-4, while Qwen shows a +312.20% relative gain over BlenderAlchemy, best-of-1.
- Evaluation on Existing Benchmarks: +35.32% average improvement is reported across BlenderGym settings, where VIGA edits fine-grained attributes while preserving unedited scene elements.
- Evaluation on Existing Benchmarks: +117.17% average Overall score improvement is reported on SlideBench across open-source and closed-source foundation models.
5 Conclusion
VIGA addresses inverse graphics through interleaved multimodal reasoning, coupling executable program synthesis with active visual verification across 2D, 3D, and 4D domains. BlenderBench provides a benchmark for evaluating multi-step spatial reasoning and progressive scene editing, while the framework remains bounded by underlying VLM perception and context limits.
- VIGA couples discrete program synthesis with active visual verification to map abstract spatial concepts into executable code across 2D, 3D, and 4D domains.
- BlenderBench evaluates multi-step spatial reasoning and progressive scene editing through a comprehensive benchmark.
- VIGA’s performance remains bounded by the spatial perception capabilities of underlying VLMs and context-window constraints in extremely long sequences.
- An evolving memory and semantic skill library support a unified, training-free agent for complex, long-horizon inverse graphics.
A Evaluation Settings
The evaluation uses controlled one-shot and iterative settings across BlenderGym, SlideBench, and BlenderBench. BlenderBench progressively tests camera adjustment, fixed-view editing, and exploratory editing under domain gaps that require extracting geometric intent.
- VIGA uses max_iterate_round=10 to limit iterative search, while best-of-N evaluation selects candidate modifications using rule-based CLIP-score comparison.
- BlenderBench contains three progressive tracks: Camera Adjustment, Fixed-View Editing, and Exploratory Editing.
- BlenderBench includes artistic style transfers and explicit visual prompts, forcing agents to extract geometric intent rather than overfit pixel-level patterns.
- BlenderGym evaluates single-step 3D scene editing, while SlideBench evaluates programmatic 2D document generation using their standard protocols and metrics.
A.2 Qualitative Settings
Long-horizon qualitative evaluations expand memory and iteration budgets and use GPT-5 as the base VLM. Other tested models show distinct weaknesses in reconstruction fidelity or active spatial exploration.
- For full 3D reconstruction and 4D physics demonstrations, evaluations use context_window=12 and max_iterate_round=100.
- GPT-5 serves as the base VLM for the long-horizon qualitative tasks.
- Claude-Sonnet-4 and Gemini-2.5-Pro complete reconstruction-from-scratch tasks with reduced visual fidelity, while GPT-4o struggles with active spatial exploration.
B.1 Extended Qualitative Results: 3D Assets and 4D Videos
Extended results provide interactive 3D assets and videos for inspecting geometric, spatial, material, temporal, and physical fidelity. These artifacts are intended to show compositional 3D grounding and temporally consistent physical interactions.
- Extended Qualitative Results: 3D Assets and 4D Videos: Supplementary .blend files allow reviewers to navigate cameras and inspect reconstructed scenes’ geometry, layouts, and materials.
- Extended Qualitative Results: 3D Assets and 4D Videos: Interactive inspection demonstrates that VIGA constructs robust, compositional 3D assets rather than merely overfitting 2D pixel patterns.
- Extended Qualitative Results: 3D Assets and 4D Videos: Supplementary .blend files and rendered .mp4 videos showcase temporal consistency and physical interactions including collisions, gravity, and glass shattering.
- Extended Qualitative Results: 3D Assets and 4D Videos: Programmatic execution within the physics engine enforces physical constraints over time, bypassing temporal flickering and physical hallucinations associated with traditional diffusion-based video generation.
B.2 Extended Quantitative Results
The extended evaluation reports VIGA against one-shot baselines across foundation models and benchmarks, while detailing the prompts and interfaces used for code synthesis, visual inspection, and scoring. Results show consistent cross-model gains, including reduced Photometric Loss for Gemini-2.5-Pro on BlenderBench.
- Comprehensive Benchmark Performance: The extended tables evaluate four foundation models on BlenderGym and BlenderBench under the best-of-1 setting against the standard one-shot baseline.BlenderGym reports Photometric Loss and Negative-CLIP Score, while BlenderBench additionally reports VLM Score.
- Cross-Model Generalization: Photometric Loss on BlenderBench Task 2 fell from 10.14 to 1.99 for Gemini-2.5-Pro with VIGA.Claude-Sonnet-4 also corrected severe viewpoint misalignment in Task 1 relative to its one-shot baseline.
- Prompting and Interfaces: VIGA uses a concise unified prompt across tasks, changing only limited domain-specific keywords such as 2D to 3D.This contrasts with prior procedural-generation methods that rely heavily on complex task-specific prompt engineering.
- Prompting and Interfaces: The agent alternates execution-oriented code synthesis with read-only visual inspection across multiple rounds.The synthesis instructions require tool calls with reasoning, while inspection uses current renders and scene context to analyze discrepancies.
- Evaluation Protocol: The VLM Score in BlenderBench is computed by GPT-4o using runtime-injected criteria and task descriptions.The evaluator compares target and generated images, prioritizing geometry, layout, object identity or pose, placement, and lighting over stylistic differences.
D Semantic Skill Library Schemas
The semantic skill library abstracts scene construction and observation into phase-specific interfaces. State-modification tools execute programmatic edits, while read-only observation tools gather visual evidence to guide those edits.
- Library Design: The skill library insulates VIGA from low-level graphics APIs by exposing high-level semantic interfaces.Its interfaces are explicitly divided according to whether the agent is modifying or observing the scene.
- State Modification Interfaces: State-modification interfaces support asset generation, scene construction, animation, and programmatic edits during code synthesis.The execution-oriented tools include asset-related operations and scene-editing functions such as execute_code.
- Edit Representation: The edit workflow requires minimal line-level removals and additions followed by complete updated Blender Python code.Removals precede additions, unrelated changes are avoided, and the final script includes both modified and unchanged lines.
- Planning Skills: The agent first stores an overall scene description and then develops a detailed construction plan before applying edits.The overall description captures the scene comprehensively, while the detailed plan structures construction stages such as preparation, rough layout, and intermediate asset placement.
- State Observation Interfaces: Read-only observation interfaces let the agent inspect objects, adjust viewpoints, and gather rendered evidence without altering geometric state.Available operations include scene information retrieval, viewpoint initialization, camera control, object observation, zooming, moving, focusing, and visibility inspection.
- State Observation Interfaces: Observation tools expose focused camera operations, including zoom, directional movement, and object-focused inspection.These operations use enumerated controls and require an object name when the focus operation is selected.