Source-linked AI summary
Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models
Guangzhao He, Rundong Luo, Wei-Chiu Ma, Hadar Averbuch-Elor
TL;DR
Single-image inverse graphics asks whether a pretrained vision-language model can reconstruct an editable 3D scene despite tightly coupled geometry, materials, composition, and lighting. SEIG decomposes this task into staged, verifiable Blender-code refinements and achieves the best score on five of six metrics across two scene sets, while supporting editable downstream graphics tasks.
Problem
The paper asks whether pretrained VLMs can recover geometry, materials, composition, and lighting as an editable Blender program from one image without specialized models or multi-view supervision.
Method
SEIG constructs scenes through additive generator–verifier stages that progressively refine geometry, materials, composition, and lighting in executable Blender code.
Results
SEIG achieves the best score on five of six metrics on both the NeRF synthetic and Edit3D scenes, outperforming executable inverse-graphics baselines.
Takeaways & Limitations
The reconstructed editable Blender scenes support downstream graphics tasks including relighting, scene editing, and physics simulation.
Takeaways & Limitations
Errors introduced in early stages may propagate, while repeated generator–verifier inference calls substantially increase runtime and API cost.
Abstract
from arXiv · showhide
Inverse graphics is a longstanding and highly underconstrained problem that seeks to reconstruct images as editable 3D scenes which can be rendered, relit, and manipulated. In this work, we investigate whether pretrained vision-language models (VLMs) can perform executable inverse graphics directly from a single image by reconstructing a scene as an editable Blender program, without relying on specialized 2D or 3D foundation models, differentiable rendering, or multi-view supervision. We introduce Staged Executable Inverse Graphics (SEIG), an agentic framework that reconstructs a 3D scene from a single image by progressively refining scene factors including geometry, materials, composition, and lighting directly in executable Blender code space. We evaluate our framework across diverse scenes using a range of reconstruction metrics spanning pixel-level, perceptual, and semantic fidelity. Our experiments show that staged reconstruction substantially improves reconstruction fidelity, highlighting the importance of task decomposition for executable inverse graphics with general-purpose VLMs. Finally, we showcase various downstream applications enabled by the reconstructed editable Blender scenes.
1 Introduction
The paper investigates whether pretrained vision-language models can reconstruct editable 3D scenes from a single image as executable Blender programs. It introduces SEIG, which stages reconstruction to improve fidelity across synthetic and in-the-wild scenes.
- Motivation: Professional 3D scene creation follows an iterative sequence of geometry, materials and textures, composition, lighting, and camera adjustment.Artists inspect intermediate results throughout this staged workflow.
- Motivation: Pretrained VLMs are evaluated for executable inverse graphics directly from a single image, leveraging their visual reasoning, instruction following, and code generation abilities.The work asks whether these models encode useful latent knowledge about 3D scenes and their underlying structure.
- Method: SEIG decomposes executable inverse graphics into sequential, semantically meaningful stages that mirror professional artists’ workflows.The framework is built on top of a pretrained vision-language model.
- Results: Staged reconstruction substantially improves reconstruction fidelity across synthetic and in-the-wild scenes compared with monolithic executable inverse graphics baselines.The comparisons include baselines with and without specialist 2D and 3D foundation models.
2 Related Work
Related work spans classical inverse graphics, neural scene representations, and VLM-based executable 3D generation. The paper positions its staged executable reconstruction framework against prior approaches that recover scenes or generate Blender programs but do not directly address the same decomposition challenge.
- Inverse Graphics: Classical inverse graphics sought to recover scene geometry, illumination, and reflectance from images through analysis-by-synthesis, intrinsic decomposition, and shape-from-shading.These directions trace back to early formulations such as Roberts’ “Blocks World.”
- Inverse Graphics: Neural approaches extend inverse graphics to sparse-view and single-image reconstruction, including differentiable rendering and primitive-based structured scene representations.Recent work also increasingly uses NeRF and 3D Gaussian Splatting for scene reconstruction.
- Inverse Graphics: Neural scene representations effectively reconstruct scenes but typically encode geometry, materials, and lighting in latent forms that are not directly editable as structured graphics programs.Several works pursue partial disentanglement within this neural-representation paradigm.
- Vision-Language Models for 3D Reasoning: VLMs provide visual understanding, instruction following, code generation, and non-trivial spatial and geometric reasoning from 2D observations.Prior studies show that appropriate prompting enables coarse 3D grounding, spatial reasoning, and geometric reasoning.
- Executable Scene Generation with Vision-Language Models: Recent systems use VLMs to generate or manipulate executable 3D content, including Blender programs from text, code-based meshes from point clouds, and compositional 3D scenes.The cited systems include SceneCraft, LL3M, MeshCoder, and BrickGPT.
- Executable Scene Generation with Vision-Language Models: VIGA reconstructs executable 3D scenes through an iterative write–render–compare–revise loop, whereas the paper identifies jointly reasoning about geometry, materials, composition, and lighting as challenging for monolithic VLM generation.The paper contrasts this entangled process with explicit sequential decomposition.
3 Method
SEIG reconstructs a reference image as executable Blender code by decomposing inverse graphics into staged, verifiable scene-editing tasks. The pipeline progressively refines scene structure, geometry, materials, composition, and lighting through agentic generator-verifier iterations.
- Framework: SEIG uses Blender’s unified Python interface to reconstruct a reference image as executable scene code whose render matches the input.The agent modifies the scene and observes rendered feedback through API calls.
- Staged decomposition: The method decomposes tightly coupled geometry, material, layout, and lighting inference into independently verifiable substages, reducing the burden of solving the entire scene jointly.The staged approach enables more grounded reasoning while additively constructing the scene.
- Scene representation and initialization: Initialization prompts the VLM to build a hierarchical scene graph, then instantiates a coarse Blender scaffold containing all visible objects and parts.The graph stores descriptions of geometry, materials, spatial relations, and reconstruction strategies; rollout sampling selects a structurally plausible candidate.
- Object refinement: After initialization, object-centric geometry and material stages refine shapes, structures, surface appearance, textures, and Blender PBR materials.Geometry edits include local shape changes, transforms, and structural additions, supported by tools for alternate-view rendering, isolation, and reverting edits.
- Verification loop: Each stage runs a multi-round generator-verifier loop in which generated Blender edits are executed, rendered, and compared against the reference image.The loop supports iterative refinement because a single code-generation pass is insufficient for matching the reference.
- Scene refinement: Scene-level composition aligns object scale, position, rotation, contact, spatial organization, and camera, while lighting then adjusts illumination with scene content fixed.Lighting cues include direction, height, shadow softness, color temperature, exposure, and contrast.
4 Experiments
Experiments evaluate SEIG against VIGA variants on synthetic and object-centric scenes using pixel, perceptual, and semantic metrics. SEIG achieves the best score on five of six metrics across both datasets, while its editable Blender outputs support relighting, object editing, and physics simulation, with staged errors and higher runtime/API cost remaining limitations.
- Quantitative Results: SEIG achieves the best score on five out of six metrics on both the NeRF synthetic and Edit3D scenes.The evaluation reports PSNR, SSIM, LPIPS, DreamSim, DINO, and CLIP between reconstructed renderings and reference images.
- Baselines and Attribution: SEIG outperforms VIGAfull without specialist 2D or 3D foundation models, while also outperforming VIGAVLM-only, isolating gains from harness design and per-stage decomposition.VIGAfull uses SAM and SAM-3D, whereas VIGAVLM-only disables these specialist models; the same base VLM is used throughout experiments without fine-tuning or task-specific supervision.
- Qualitative Results: Across representative scenes, SEIG produces structured Blender outputs matching reference geometry, surface appearance, and composition, while avoiding the Janus artifact shown by VIGAfull.VIGAfull duplicates frontal facial features onto the back of a humanoid character, whereas VIGAVLM-only and SEIG avoid this failure mode through compositional reconstruction.
- Staged Reconstruction: The four reconstruction stages progressively refine a coarse primitive scaffold through geometry, materials, composition, and lighting.Intermediate outputs demonstrate how object shapes, surface details, reference layout, and illumination are refined sequentially.
- Downstream Applications: Editable Blender scenes enable relighting, per-object editing, and physics simulation without retraining or post-processing.Examples include changing illumination, duplicating or retexturing objects, manipulating shapes and compositions, and simulating rigid-body and soft-body interactions.
- Limitations: Early-stage errors may propagate and trap later stages in local minima, while multiple generator–verifier stages incur substantially higher runtime and API cost than single-pass generation.Global refinement passes that revisit and jointly optimize scene factors are proposed as a possible mitigation for propagated errors.
5 Conclusion
The paper introduces a staged executable inverse graphics framework that reconstructs editable Blender scenes from a single image using only a pretrained vision-language model. It progressively solves individually verifiable subproblems through generator–verifier stages, producing coherent editable intermediate scenes.
- SEIG reconstructs editable Blender scenes directly from a single image using only a pretrained vision-language model, without task-specific training, specialized foundation models, or differentiable rendering.
- Decomposing reconstruction into individually verifiable subproblems enables progressive recovery through generator–verifier stages.
- Each pipeline stage commits its output before the next begins, making every intermediate scene coherent and editable.The stages are Initialization, Geometry, Material, Composition, Lighting, and Camera-adjustment.
- SEIG creates Blender scenes from both in-the-wild and synthetic reference images, with input and novel views visualized alongside the reference image.