Source-linked AI summary
3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models
Ling Liu, Bingchen Gong, Amal Dev Parakkat, Maks Ovsjanikov
TL;DR
Existing evaluations often test fixed-input perception or spatial judgments rather than whether VLM agents can acquire evidence and generate 3D structure. 3DHarnessBench evaluates executable Blender reconstructions across four progressively richer access settings. Richer access improves results overall, but agent gains are uneven, and Full 3D Interaction performs best overall.
Problem
Prior 3D evaluations largely probe perception or fixed inputs, leaving unclear whether VLM agents can acquire evidence, self-correct, and actively generate 3D structure.
Source: §1 · p. 2Method
3DHarnessBench evaluates Blender Python reconstruction across Single-view, Multi-view, Active Visual, and Full 3D Interaction harnesses, scoring appearance, geometry, topology, and interaction cost.
Source: §1 · p. 2Results
Takeaways & Limitations
The benchmark reveals uneven frontier-VLM ability to acquire and exploit visual and geometric evidence for 3D-to-code reconstruction.
Source: §1 · p. 2, §5 · p. 8Reference images are unavailable at runtime.
Source: §12 · p. 25
Abstract
from arXiv · showhide
We introduce 3DHarnessBench, a benchmark that evaluates the agentic ability of frontier vision-language models (VLMs) to recover 3D geometry as Blender Python code from a variety of inputs. Unlike previous frameworks that prompt the VLMs with a fixed input (e.g., a single rendering or a text description), 3DHarnessBench evaluates four separate harness settings that progressively enable active agentic exploration, facilitated by recent Blender MCP functionality. Our hierarchy from Single-view, Multi-view, Active Visual (arbitrary viewpoint access), and Full 3D Interaction (complete access to the target object through Blender function calls) probes the models' abilities in both visual perception and active inference, tool calling, and self-correction. We observe that the ability of all frontier models to recover 3D geometry improves significantly with richer function call access, although the improvements are strongly model-dependent, revealing highly uneven agentic 3D-to-code capabilities. We will release the benchmark, code, outputs, and agent trajectories for reproducible 3D evaluation.
1. Introduction
3DHarnessBench evaluates frontier VLMs’ agentic 3D-to-code reconstruction under progressively richer access to a target. Results improve with access overall, but vary substantially across models, especially when agents must actively inspect the target.
1. Introduction
- Earlier 3D evaluations largely tested recognition, question answering, or spatial reasoning rather than active generation of 3D structure.Fixed-input reconstruction benchmarks also entangle weak inference, poor view integration, and missing evidence, without testing evidence acquisition.Source: §1 · p. 2
- The benchmark holds executable reconstruction fixed while expanding target access from one image to fixed views, arbitrary viewpoints, and Blender geometric queries.These harnesses probe single-image inference, multi-view integration, active viewpoint selection, metric grounding, and interactive self-refinement.Source: §1 · p. 2
- The evaluation uses 100 objects derived from 3DCodeBench and seven frontier VLMs in their official agent CLIs connected to Blender MCP.Reconstructions are assessed on appearance, geometry, topology, and interaction cost.Source: §1 · p. 2
- Full 3D Interaction yields the strongest geometry across agents, while Active Visual helps some agents but leaves others below their fixed-view baselines.Moving from Multi-view to Active Visual shifts Uni3D by up to +16% for the strongest agents and −17% for the weakest; the authors attribute much of the gap to observation-interface handling.Source: §1 · p. 2
- The authors analyze controlled ablations and plan to release the benchmark, code, outputs, and agent trajectories.Ablations cover camera-pose cues, observation normalization, and measurement granularity.Source: §1 · p. 2
2. Related Work
Prior 3D VLM work studies spatial understanding, reconstruction, executable inverse graphics, and programmatic modeling benchmarks. 3DHarnessBench uses executable reconstruction to measure geometry, topology, and appearance while extending fixed-input evaluation with agent-directed exploration and geometric queries.
2. Related Work
- Earlier work includes 3D-grounded representations and supervision [63], spatially grounded supervision, and other grounding or predictive approaches [45] [27].A separate line develops image-conditioned 3D prediction [41] [49].Source: §2 · pp. 2–3
- Prior VLM evaluations mainly measure spatial judgments, metric reasoning, or question answering, whereas 3DHarnessBench makes geometry, topology, and appearance measurable through executable reconstruction.Its executable program externalizes inferred structure and allows measurement as richer evidence becomes available.Source: §2 · p. 3
- Executable inverse graphics generates structured 3D programs [29], while systems combine VLM reasoning, code, and visual feedback [46] [19] [56].Most related programmatic modeling benchmarks use fixed images; 3DHarnessBench adds agent-directed Blender MCP exploration and geometric queries.Source: §2 · p. 3
3. 3DHarnessBench
3DHarnessBench evaluates whether vision-language model agents can reconstruct 3D targets as executable Blender Python code under progressively richer target access. Across the harnesses, agents acquire evidence, update and execute their programs, and refine reconstructions through interaction.
3. 3DHarnessBench
- 3DHarnessBench probes agentic 3D-to-code ability by requiring VLMs to infer target structure and reconstruct it as executable Blender Python code.Target access expands from passive visual observations to agent-directed exploration and explicit geometric interaction.Source: §3 · p. 3
3.1. Benchmark Formulation
- The benchmark varies accessible target evidence while keeping the target and executable reconstruction objective fixed.Agents condition tool choices on interaction history and iteratively update and execute a Blender Python program.Source: §3.1 · pp. 3–4
3.2. Target Access Settings
- The four harnesses progress from passive observation to active visual exploration and explicit 3D interaction, differing in available tools and observations.All settings share the same executable reconstruction objective.Source: §3.2 · p. 4
3.2.1. Single-view and Multi-view
- Single-view supplies one fixed target image, while Multi-view supplies four fixed views; both keep visual evidence fixed throughout reconstruction.Multi-view reduces ambiguity by showing the same object from four views, while preserving passive observation.Source: §3.2.1 · p. 4
3.2.2. Active Visual
- Active Visual lets the agent choose target views, revisit ambiguous regions, and use new observations to refine its reconstruction.The agent can change viewing direction, inspect unseen regions, and zoom in or out as its reconstruction evolves.Source: §3.2.2 · pp. 4–5
3.2.3. Full 3D Interaction
- Full 3D Interaction adds target-side geometric queries to visual exploration, with the agent deciding what to measure and when.Queries can return properties such as bounding boxes, transformations, dimensions, and distances; the reconstruction must remain procedurally agent-generated rather than copying the target.Source: §3.2.3 · p. 5
4. Experiments
The experiments compare seven frontier VLMs across four progressively interactive target-access settings, scoring reconstruction quality and interaction efficiency. Richer access improves geometry overall, but agents differ in how well they use active observations, geometric cues, and refinement.
4.1. Experimental Setup
- The benchmark samples 100 objects from 3DCodeBench and evaluates seven agents using appearance, geometry, topology, and interaction-efficiency measures.Agents run in their official CLIs with native tool-use and context management; target source programs and authoring information are withheld.Source: §4.1 · p. 5
4.2. Evaluation Protocol
- Single-view and Multi-view provide fixed images, Active Visual permits constrained viewpoint inspection, and Full 3D Interaction exposes the full Blender MCP interface.Fixed-view settings refine successful reconstructions for three iterations; Active Visual restricts target-side tools to approved camera and viewport operations.Source: §4.2 · pp. 5–6
4.3. Main Results
- Full 3D Interaction yields every agent’s best geometry, while Active Visual separates agents most sharply.Across appearance, geometry, and topology, richer access shows broadly similar trends; interactive settings also reveal larger cross-agent differences than fixed-view settings.Source: Table 1 · p. 6, §4.3 · p. 6
- Opus 5 is strongest overall under both interactive settings, while Gemini 3.1 Pro and MiniMax M3 remain weaker, particularly under Active Visual.Fable 5, GPT-5.6 Sol, Kimi K3, and Qwen 3.8 Max form a competitive second group.Source: §4.3 · p. 6
- From Multi-view to Active Visual, Uni3D changes by +16.4% for Opus 5 but −16.5% for MiniMax M3, and the weakest-to-strongest gap reaches 44.7%.The gap is 24.9% under Multi-view and 26.8% under Full 3D Interaction; Active Visual is the sharpest discriminator.Source: §4.3 · p. 6
4.4. Analysis
- Camera-pose cues give the most consistent Multi-view gains; grid cues help less reliably, and combining them is not additive.The ablation tests ten instances with GPT-5.6 Sol; explicit camera pose information is more useful than viewport grid cues.Source: §4.4 · p. 7, Table 2 · p. 8
- Normalized viewport screenshots improve MiniMax M3, while Fable 5’s inspection views help Gemini 3.1 Pro recover Multi-view performance on several metrics.The results attribute both agents’ Active Visual degradation largely to observation acquisition and quality.Source: §4.4 · pp. 7–8, Table 3 · p. 8
- Explicit global and part measurements improve nearly all metrics, with part cues best for Uni3D and topology and combined cues lowest in Chamfer distance.On ten instances with Opus 5, global measurements achieve the highest SigLIP-2 score.Source: §4.4 · p. 8, Table 4 · p. 8
5. Conclusion and Future Work
The conclusion presents 3DHarnessBench as an evaluation of frontier VLMs’ 3D-to-code ability under progressively richer target access. It reports that access, geometric grounding, and explicit measurements relate to reconstruction outcomes, while proposing broader scenes and trace-based fine-tuning as future directions.
5. Conclusion and Future Work
- 3DHarnessBench evaluates frontier VLMs’ agentic 3D-to-code capabilities under progressively richer target access.Multi-view generally improves over Single-view, while Active Visual helps only when observations are effectively acquired and integrated; Full 3D Interaction performs best overall.Source: §5 · p. 8
- Ablations show gains from geometric grounding and explicit measurements, and identify initial reconstruction and interactive self-correction as distinct capabilities.Source: §5 · p. 8
- Future work will extend evaluation to multi-object placement and affordance constraints and explore fine-tuning on benchmark agentic traces for smaller models.Source: §5 · p. 9
7. Overview
The overview describes supplementary material that extends the paper’s protocol, results, analyses, qualitative examples, prompts, and code documentation.
7. Overview
- Section 8 adds per-harness protocols, the Blender MCP interface specification, experimental settings, and metric definitions.Source: §7 · p. 13
- Sections 9–11 provide detailed results, ablations and analyses, and additional qualitative reconstruction comparisons.These include per-Betti-number errors, interaction-efficiency statistics, and supplementary Active Visual and refinement analyses.Source: §7 · p. 13
- Sections 12–13 reproduce prompts and agent-skill materials and present a complete agent-generated Blender Python reconstruction program.Source: §7 · p. 13
8. Evaluation Protocol
3DHarnessBench holds executable 3D reconstruction fixed while progressively expanding target access, from fixed images to agent-directed viewing and geometric interaction. Its protocol, Blender interface, and evaluation settings define how agents explore, reconstruct, and are scored.
8.2. Blender MCP Interface
- Active Visual restricts target-side tools to screenshots and approved viewport or camera operations, while Full 3D Interaction exposes the complete Blender MCP function set.Active Visual validates submitted code against an AST allow-list; its target and reconstruction use separate Blender processes, unlike Full 3D Interaction's single process.Source: §8.2 · p. 13, Table 6 · p. 14, §8.3 · p. 14
8.3. Experimental Settings
- The benchmark samples 100 objects from 3DCodeBench, reuses those targets across harnesses, and evaluates seven frontier VLMs in their official agent CLIs.The original procedural source program is withheld; all ablations use the same fixed 10-instance subset.Source: §8.3 · pp. 13–14
8.1. Harness Protocols
- The four harnesses progress from one fixed image and four fixed views to arbitrary camera inspection and full Blender interaction with geometric queries.The fixed-view settings add three render-and-refine iterations; interactive settings replace fixed refinement rounds with open-ended interaction.Source: §8.1 · p. 13, Table 5 · p. 14
8.3. Experimental Settings
- Appearance, geometry, topology, and interaction efficiency are measured using canonical rendered views, point-cloud metrics, mesh Betti numbers, and agent/tool usage statistics.Appearance views use identical lighting; geometry metrics include symmetric Chamfer distance and Uni3D similarity, while topology uses normalized Betti-number error.Source: §8.3 · pp. 14–15
- Controlled ablations vary camera-pose and grid cues, viewport normalization and inspection views, and global or part-level geometric measurements.The VLM-judge evaluation scores Full 3D Interaction reconstructions on six criteria, averaging scores across 100 instances per agent.Source: §8.3 · p. 15
9. Main Results
Richer target access generally improves topology, but interactive efficiency and gains vary across agents. Comparisons with 3DCodeBench and a perceptual judge further show that model rankings depend on the metric and whether agents can direct their observations.
9.1. Topology and Interaction Efficiency
- Full 3D Interaction reduces all three Betti-number errors relative to Single-view for every evaluated agent, with especially pronounced gains for β0 and β1.Opus 5's β0 error falls from 0.140 to 0.015, while GPT-5.6 Sol's β1 error falls from 0.097 to 0.006.Source: §9.1 · pp. 15–16, Table 7 · p. 16
- Interactive harnesses show greater cross-agent variation in interaction cost, and richer Full 3D Interaction requires fewer model and MCP calls for most agents than Active Visual.MiniMax M3 averages 431 model API calls, 156 MCP calls, and 23.87 reconstruction versions under Active Visual; GPT-5.6 Sol is the exception to the call reduction, increasing from 55 to 70 model API calls.Source: §9.1 · pp. 16–17
9.2. Comparison with 3DCodeBench
- Fable 5 retains its geometric lead over Gemini 3.1 Pro in both benchmarks, but their Active Visual Uni3D results diverge: 0.764 to 0.832 for Fable 5 and 0.753 to 0.691 for Gemini.The 3DCodeBench comparison is contextual, not directly comparable, because the object subset, refinement rounds, and Chamfer protocols differ.Source: §9.2 · pp. 17–18, Table 8 · p. 17
9.3. VLM-based Hexagonal Evaluation
- Under Full 3D Interaction, Opus 5 ranks strongest overall in the six-criterion judge evaluation, followed by Fable 5 and GPT-5.6 Sol.The judge is Gemini 3.1 Pro; its own reconstructions nevertheless receive the second-lowest scores.Source: Figure 5 · p. 17, §9.3 · p. 18
- Agents cluster more closely on 3D plausibility than on fidelity, geometry detail, or texture detail, where matching the specific target remains harder.Even weaker agents often produce a coherent, plausible object but are less successful at matching its identity, proportions, and fine-grained structure.Source: §9.3 · p. 18
10. Ablations and Analyses
The ablations show that reconstruction quality depends on the kind of geometric evidence provided and on agents’ ability to acquire and use observations. Active inspection and self-correction differ markedly across agents, and more interaction alone does not ensure better evidence or refinement.
10. Ablations and Analyses
- Additional metrics broadly support the main conclusions, while changes in geometric fidelity and visual similarity do not always align across feature spaces.The extended analyses include DINOv2, DINOv3, and Uni3D image–3D similarity.Source: §10 · p. 18
10.1. Geometric Grounding in Multi-view
- Camera-pose cues are the most reliable geometric grounding signal, while grid cues provide less consistent gains and combining pose and grid is not additive.The combined cues can degrade some appearance and topology metrics.Source: §10.1 · p. 18
- Explicit geometric measurements improve most metrics over image-only input; part-level cues lead on Uni3D and topology, while combined global and part-level cues achieve the lowest Chamfer.Their clearest benefits are in geometric fidelity and topology.Source: §10.1 · p. 18
10.2. Active Visual
- Viewport normalization improves MiniMax M3 across reported metrics but has mixed effects on Gemini 3.1 Pro; Fable 5 inspection views benefit Gemini more than MiniMax.For Gemini, normalization improves appearance and geometry but worsens topology; for MiniMax, Fable 5's views help versus standard Active Visual, but less than normalization.Source: Table 11 · p. 19, §10.2 · pp. 19–20
- Active Visual success depends on informative, diverse observations as well as inspection volume: MiniMax M3 makes many largely repetitive inspections, while Gemini 3.1 Pro explores little and terminates early.GPT-5.6 Sol and Opus 5 explore more diverse viewpoints; Opus also adjusts the clipping plane to expose occluded geometry.Source: §10.2 · p. 20
10.3. Initialization versus Interactive Refinement
- Interactive refinement is not merely local polishing: agents differ in when they form a usable hypothesis and whether later versions refine or correct earlier structure.MiniMax M3's early reconstructions are far from the target structure but later iterations substantially reorganize the geometry; Kimi K3 establishes structure early and adds detail over a longer trajectory.Source: §10.3 · pp. 20, 22, Figure 8 · p. 21
11. Additional Qualitative Results
Qualitative examples show that richer access particularly helps stronger agents recover objects with ambiguous, thin, repeated, or highly structured geometry. Full access is more robust overall, but substantial agent-dependent differences remain.
11. Additional Qualitative Results
- For stronger agents, interactive access improves complete part layouts and proportions on challenging objects, including a crustacean, auger, and flower plant.The examples show better recovery of articulated appendages, repeated helical structure, and the arrangement of stems, leaves, and blossoms.Source: §11 · p. 22, Figure 10 · p. 23, Figure 11 · p. 23
- Active Visual most clearly separates agents: some improve with directed viewpoints, while others still simplify or misrepresent structure; Full 3D Interaction is more robust overall.Large differences remain especially on fine, repeated, and multi-part structures.Source: §11 · p. 22, Figure 12 · p. 24
- Even with explicit geometric information, agents differ substantially in preserving silhouettes, repeated structures, and major part relationships.Weaker agents often simplify thin structures, miss repeated components, or recover only the coarse object category.Source: §11 · p. 22, Figure 12 · p. 24
12. Prompts and Agent Skill
This section specifies the prompts and Agent Skill governing reconstruction and refinement across the harness settings. It describes how models inspect references and produce Blender scripts, with workflow and task constraints fixed while available Blender MCP capabilities are controlled by the harness.
Single-view and Multi-view
- Fixed-view harnesses use 3DCodeBench’s initial reconstruction prompt and BlenderGym’s editing prompt for iterative refinement.The initial prompt requests a self-contained Blender 5.0 Python script that reconstructs geometry and visible surface appearance; the script must run without reading reference images at runtime.Source: §12 · pp. 24–25
- The reconstruction prompt directs models to cross-reference views of one object and reproduce its proportions, parts, symmetries, ornament, materials, colors, and surface effects.Visible depth and silhouette-changing details should be modeled as geometry or displacement, not only color.Source: §12 · pp. 25–26
- For refinement, models compare current renders with every ground-truth view, correct the largest visible errors, preserve useful detail, and return the complete corrected script.The prompt explicitly asks for a complete script rather than a diff.Source: §12 · p. 26
Active Visual and Full 3D Interaction
- Interactive sessions begin with a preamble and task prompt invoking the staged Agent Skill; the preamble specifies one official Blender MCP server and takes precedence over conflicting task text.The prompt identifies the configured server and connection facts, and instructs models to use only that server.Source: §12 · p. 26
- The reproduced skill text describes separate reference-viewing and geometry-editing Blender instances, with a loop of viewing the target from arbitrary angles before editing.It identifies the viewport-only scene as ground truth and says not to change it, while the full-access instance is used to build and edit geometry.Source: §12 · p. 26
Prompts and Agent Skill
- The shared Agent Skill fixes reconstruction workflow and task constraints across Active Visual and Full 3D Interaction, while the harness controls target-side Blender MCP capabilities.The same instruction is used in both settings.Source: §12 · p. 26
- The skill requires determining the access mode and reading its reference before inspection, distinguishing same-scene single-server access from two-server screenshot-only access.The two-server inspection instance is read-only and exposes viewport or camera changes and screenshots, but not code execution or scene editing.Source: §12 · p. 27
- The ground truth must remain unmodified; geometry is stage 1, with textures and materials added at stage 2 after stage 1 passes checks unless the task limits scope to geometry.The preservation rule covers its objects, meshes, materials, modifiers, hierarchy, collections, transforms, and visibility.Source: §12 · p. 27
- Inspect the ground truth before scripting from multiple viewpoints, assessing dimensions, parts, proportions, transforms, symmetry, attachments, silhouettes, curvature, and visible materials and colors.The required views include front and rear, left and right, top and bottom, three-quarter views, and close-ups of ambiguous or distinctive geometry.Source: §12 · p. 27
- The workflow decomposes the model into coherent geometry units and calls for repeatedly inspecting, modifying, and rerunning the Blender Python script until it closely matches.Units are characterized by geometry type, dimensions, position and orientation, attachment, and symmetry or repetition.Source: §12 · p. 27
13. Example of Agent-Generated Code
The example reconstructs an auger shell in Blender with procedural geometry and appearance, using hard-coded parameters and sampled target features.
13. Example of Agent-Generated Code
- The Opus 5 example combines procedural modeling, hard-coded geometric parameters, and sampled points to reproduce the auger’s geometry and appearance.The code labels the target an “AugerFactory” sea shell.Source: §13 · p. 32
- The script creates a Blender collection and shell object, removes its own reconstruction data before reruns, and returns vertex count and dimensions.The object is parented to a root and assigned the generated material.Source: §13 · pp. 32–34
- The mesh is built by sampling successive coil rings, connecting them with faces, and closing the protoconch with a capped sweep.The generated mesh is converted to a Blender mesh and smooth-shaded.Source: §13 · p. 32
- The material uses GT-sampled colors along the coil, measured roughness and metallic values, and noise for mottling and fine growth-line relief.The code adds the relief through a bump node.Source: §13 · p. 33