PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Adaptive Grounded 3D-Programmers Using 2D VLMs

Arman Raayatsanati, Sombit Dey, Anna-Maria Halacheva, Jan-Nico Zaech, Luc Van Gool, Danda Pani Paudel

arXiv:2610.02021v1cs.CVcs.AI

TL;DR

Specialized 3D systems rely on extensive 3D annotations, domain expertise, and task-specific training, fragmenting the ecosystem and limiting scalability across diverse real-world tasks. 3D-Prog combines CCF, which grounds inputs and outputs in a shared Euclidean frame, with TAF, an iterative feedback loop for task-specific analysis and refinement. Across all evaluated benchmarks, 3D-Prog consistently outperforms or matches proprietary and open-source VLM baselines.

  • Problem

    Specialized 3D systems rely on extensive 3D annotations, domain expertise, and task-specific training, fragmenting the ecosystem and limiting scalability across diverse real-world tasks.

    Source: §1 · p. 2

  • Method

    3D-Prog combines CCF, which grounds inputs and outputs in a shared Euclidean frame, with TAF, an iterative feedback loop for task-specific analysis and refinement.

    Source: §1 · p. 2

  • Results

    Across all evaluated benchmarks, 3D-Prog consistently outperforms or matches proprietary and open-source VLM baselines.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    Experiments show consistent, reliable, high-quality outputs and precise 3D understanding across diverse tasks and real and synthetic assets.

    Source: §5 · p. 8

    Performance depends on the underlying VLM, the Judge may accept suboptimal results, and CCF scene size and detail are limited by the VLM context.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent vision-language models (VLMs) exhibit remarkable generalization and reasoning abilities, yet 3D understanding in these models is limited by data scale, training diversity, and reasoning capacity. Instead of naively extending these models into 3D, we take a different approach: we enable powerful 2D VLMs to operate reliably in 3D by introducing 3D grounding and iterative feedback loops with two novel concepts: Canonical Coordinate Framing (CCF) and Task-Adaptive Feedback (TAF). CCF serves as a unified visual representation that anchors both inputs and outputs to a shared Euclidean coordinate system, solving common challenges in 3D grounding such as axis ambiguity, inconsistent metric scale, and floating references. Complementary to this structured framing of the 3D inputs, TAF closes the reasoning loop with task-adaptive dynamic feedback that enables 2D VLMs to perform varied open-vocabulary tasks within their native visual context. Building on this foundation, we introduce 3D-Prog, a 3D understanding, reasoning, and generation framework that jointly employs the capabilities of CCF and TAF together with powerful VLMs. Without requiring any retraining, 3D-Prog performs open-vocabulary 3D understanding, manipulation, and generation across both object-level and scene-level tasks. Our experiments show that the joint use of CCF and TAF transforms 2D VLMs into geometry-aware 3D programmers, achieving consistent, interpretable, and high-quality results across diverse 3D tasks.

1. Introduction

The introduction frames 3D-Prog as a way to use pretrained 2D VLMs for grounded 3D tasks, combining canonical spatial framing with task-adaptive feedback. It reports open-vocabulary performance across object- and scene-level tasks without retraining.

1. Introduction

  • Specialized 3D systems depend on extensive annotations, domain expertise, and task-specific training, fragmenting the ecosystem and limiting scalability.They are tailored to particular geometric tasks, from scanned-object conversion to part segmentation and language-based scene composition.Source: §1 · p. 2
  • 2D VLMs lack explicit metric 3D grounding, while 3D VLMs are constrained by limited data scale and diversity.Directly prompting 2D VLMs for 3D quantities yields unstable predictions, while 3D VLMs lack diverse capabilities.Source: §1 · p. 2
  • CCF and TAF ground 2D VLMs in a shared Euclidean frame and provide iterative feedback for task-specific reasoning over 3D content.CCF anchors inputs and outputs; TAF supplies an execution-based feedback loop without retraining or architectural modification.Source: §1 · p. 2
  • 3D-Prog turns natural-language instructions and 3D assets into executable, interpretable manipulation programs whose outputs use USD.The programs invoke geometric analysis, manipulation, or visual-feedback modules and form self-contained reasoning chains.Source: §1 · p. 2
  • Without retraining or fine-tuning, 3D-Prog performs open-vocabulary 3D understanding, manipulation, and synthesis across object- and scene-level tasks.The paper presents it as unifying 3D perception, reasoning, and generation in a visually grounded framework.Source: §1 · p. 2

2. Related Work

Related work describes strong 2D VLM capabilities but notes that their spatial reasoning lacks explicit 3D consistency. 3D-aware VLMs and visual-programming approaches address parts of the problem, but the cited work identifies limits in generality and task coverage.

2. Related Work

  • 2D VLMs show spatial and physical understanding, but their reasoning remains implicit in 2D coordinates and lacks explicit 3D consistency.This limits geometry-grounded outputs for 3D reasoning and manipulation.Source: §2 · p. 2
  • 3D-aware VLMs use 3D backbones or multi-view consistency for 3D perception and generation, but limited datasets and task-specific objectives constrain generalization.They typically specialize in either understanding 3D assets or generating 3D outputs rather than combining both flexibly.Source: §2 · pp. 2–3
  • Visual-programming frameworks compose executable programs for complex 2D vision tasks, while prior 3D visual-program synthesis remains domain- or asset-specific.3D-Prog is presented in contrast as supporting diverse assets and tasks spanning generation, editing, and understanding.Source: §2 · p. 3

3. Method

The method combines canonical visual grounding, task-adaptive feedback, and a USD-based programming framework. It applies this design across generation, understanding, and reasoning tasks on varied assets.

3. Method

  • 3D-Prog combines CCF grounding, TAF feedback, and a unified framework for diverse 3D programming tasks.The method section introduces the framing protocol first, then task-adaptive feedback and the integrated system.Source: §3 · p. 3

3.1. Canonical Coordinate Framing

  • CCF renders six orthographic projections and a perspective view with shared labeled axes, metric scale cues, and a fixed bounding-box origin.The shared frame supports grounded coordinate and spatial-relation predictions; the paper reports that ungrounded designs struggle with orientation, scale, and metric consistency.Source: §3.1 · p. 3, Table 1 · p. 4
  • CCF supports consistent spatial queries, geometry-aware visual grounding, interpretable scene editing, and multi-view alignment without pose metadata.The authors conclude that explicit canonical framing is a prerequisite for robust geometry-aware multimodal interaction.Source: §3.1 · pp. 3–4

3.2. Task-Adaptive Feedback

  • TAF links the VLM to Python execution so it can inspect an asset, generate task-specific diagnostic visualizations, and refine its output from the resulting evidence.The VLM chooses intermediate analyses, ranging from material-property plots to DBSCAN clustering for part segmentation.Source: §3.2 · p. 4

3.3. 3D-Prog Framework

  • 3D-Prog uses CCF-rendered views and task instructions to generate interpretations and executable code, with a Judge VLM returning completion status and diagnostic feedback for iterative refinement.The loop continues until completion or a predefined iteration limit; the system uses USD to decompose complex transformations into interpretable programmatic steps.Source: §3.3 · pp. 4–5, Figure 3 · p. 4

3.4. Generalization of 3D-Prog

  • 3D-Prog spans generation, understanding, and reasoning across synthetic and real-world assets at object and scene scales.Its tasks include object completion, Scan2Sim, detection, segmentation, editing, question answering, captioning, articulation prediction, and scene composition; USD serves as a composable representation across these objectives.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate 3D-Prog across object- and scene-level generation, understanding, and reasoning, using task benchmarks, ablations, VLM-family comparisons, and spatial-robustness tests. Results report broad gains over standalone VLMs, while ablations distinguish CCF’s geometric grounding from TAF’s iterative refinement.

4. Experiments

  • Evaluation spans generation, understanding, and reasoning on synthetic and real-world assets at object and scene scales, using task-specific metrics and human judgment.The setup compares standalone VLMs with Claude- and Kimi-backed 3D-Prog variants; refinement is capped at 20 iterations with judge-approved early stopping.Source: §4 · pp. 5–6

4.1. Results and Analysis

  • 3D-Prog consistently matches or outperforms proprietary and open-source VLM baselines across the evaluated benchmarks.Generative results include 0.19 Chamfer distance for synthetic completion and 0.08 for Scan2Sim; baselines could not generate valid USD files from real-world Scan2Sim data.Source: §4.1 · p. 6

4.2. Ablation Study

  • CCF and TAF are complementary: removing CCF worsens completion Chamfer distance from 0.19 to 0.33 and detection mAP from 0.38 to 0.01, while removing TAF degrades completion threefold, from 0.19 to 0.58.Without CCF, the visual prompt has one unframed orthographic view; without TAF, the system loses task-adaptive analysis and iterative correction.Source: §4.2 · pp. 6–7, Table 3 · p. 6
  • Under geometric perturbations, 3D-Prog retains 0.02 Chamfer distance for every translation and 15° rotation and 0.07 at 30°, while the baseline degrades by up to 22×.At 45°, both methods degrade similarly.Source: §4.2 · p. 7, Table 4 · p. 7

4.3. 3D Programming Across VLM Families

  • Among the tested open-source VLMs, only Kimi K2.5 provides the code generation and USD adaptability needed for these 3D programming tasks.The benchmark includes Kimi K2.5, Llama 4, and Qwen3-VL-30B-A3B-Instruct; the other models’ visual understanding was not matched by adequate programming expertise for these tasks.Source: §4.3 · p. 7, Table 5 · p. 7

4.4. Comparisons to 3D VLMs

  • The training-free approach outperforms GaussianVLM on both SQA3D scene-level accuracy and Nr3D object-captioning METEOR.The comparisons use the Claude 2D VLM baseline and GaussianVLM, a generalist 3D VLM.Source: Table 6 · p. 7, §4.4 · p. 8

4.5. Discussion of Experiments

  • CCF is especially relevant to geometric reasoning, while TAF specializes feedback to the query and refines analysis over successive iterations.Across tasks, convergence takes 7–20 iterations (mean 11), with about 22 seconds per iteration and total runtimes of roughly 2–8 minutes.Source: §4.5 · p. 8, Figure 5 · p. 8

5. Conclusion

3D-Prog brings 2D VLM capabilities to diverse 3D tasks through CCF and TAF, with experiments reporting reliable, high-quality outputs across real and synthetic assets. Its performance and scope remain bounded by underlying VLM capabilities, judge-module behavior, and context limits.

5. Conclusion

  • 3D-Prog combines CCF grounding with TAF task-specific feedback to support 2D VLMs on 3D understanding, reasoning, editing, composition, and generation.The paper describes it as a generalizable approach to free-form 3D programming.Source: §5 · p. 8
  • Across diverse tasks on real and synthetic assets, 3D-Prog consistently produces high-quality 3D outputs and precise 3D understanding.The conclusion characterizes the framework as versatile and generalizable.Source: §5 · p. 8
  • Performance depends on underlying VLM capabilities, which still struggle with very complex reasoning and fine-grained details.The paper also notes that CCF scene size and detail are limited by VLM context, and the judge evaluates convergence rather than correctness.Source: §5 · p. 8
  • The convergence judge may accept suboptimal results, although conservative iteration limits and visual-comparison feedback are used as mitigation.The authors report 100% motion-type accuracy on articulation as suggestive of a self-correcting loop.Source: §5 · p. 8

Supplementary Material

The supplementary material provides task, runtime, user-study, feedback, error, and qualitative-evaluation details, along with design and implementation specifications.

Supplementary Material

  • Section A provides a detailed task taxonomy and dataset licenses.The supplement also points to dataset terms of use.Source: Supplementary Material · p. 12
  • Sections B and C cover per-task runtime statistics and the user-study methodology, respectively.Source: Supplementary Material · p. 12
  • Sections D through G provide TAF analysis, qualitative error analysis, expanded qualitative results, and design choices including the core algorithm and prompts.Source: Supplementary Material · p. 12

A. Task Overview

The evaluation covers nine tasks spanning generation, understanding, and reasoning, with task descriptions and input asset types tabulated. It also reports dataset licenses and terms of use.

A. Task Overview

  • The evaluation demonstrates 3D-Prog across nine tasks spanning generation, understanding, and reasoning.Tab. S1 breaks down each task, its description, and input asset types.Source: A. Task Overview · p. 12
  • All datasets used in the evaluation are publicly available, and their licenses and terms of use are reported in Tab. S2.Source: A. Task Overview · p. 12
  • ScanNet and ScanNet++ have custom terms requiring separate inspection, while the Pixar Kitchen Set uses Pixar’s non-commercial USD asset license.The passage cites ScanNet [12] and ScanNet++ [52].Source: A. Task Overview · p. 12

B. Runtime Analysis

The supplementary runtime analysis reports measurements across selected tasks, and a four-evaluator user study compares compositional reasoning and object editing. The study finds stronger compositional performance for 3D-Prog, while visual quality in object editing was comparable and the baseline failed at physical-property corrections.

B. Runtime Analysis

  • Runtime statistics for selected evaluation tasks are reported in Tab. S3, with measurements collected on an NVIDIA RTX 5090 GPU with 32 GB of memory.Source: B. Runtime Analysis · p. 12

C. User Study

  • Four independent evaluators scored generated assets from 0 to 5 using five task-specific success criteria, and their average determined final task performance.For object editing, evaluators also received the input asset to assess geometry preservation; compositional criteria included semantics, orientation, color/material accuracy, structure, and composition, while editing criteria included geometry preservation, material accuracy, physical-property realism, scope accuracy, and visual realism.Source: C. User Study · p. 12
  • In compositional reasoning, 3D-Prog performed consistently highly across criteria, while the baseline struggled with intersections, orientation, and semantics, especially for non-cubic primitives.The baseline succeeded only on color and material attributes; object-editing visual quality was comparable, but the baseline consistently failed to correct physical properties.Source: C. User Study · p. 12

D. TAF Analysis

TAF uses task-adaptive analysis and visual feedback in an iterative loop to support geometric refinement and open-ended tasks. The section inspects its outputs and documents task criteria and runtime.

D.1. Structure of TAF Outputs

  • TAF supports semantic knowledge extraction, iterative task execution, geometric refinement, and open-ended tasks.The authors manually inspect its logs and describe analysis, 3D programming, reasoning, and visual validation as distinct output stages.Source: §D.1 · pp. 12–13
  • The described TAF output combines visual and geometric analysis with task-specific operations, corrective reasoning, and final visual validation.Operations may include filtering, zooming, PCA, and clustering; reasoning can identify flaws and propose corrections.Source: §D.1 · pp. 12–13
  • The supplementary tables summarize demonstrated task coverage, dataset licenses, and runtime; segmentation tasks use more iterations than semantic tasks.Per-iteration time comprises approximately 60% VLM inference, 25% code execution, and 15% rendering.Source: Table S3 · p. 13, Table S1 · p. 13, Table S2 · p. 13
  • Tables S4 and S5 specify success criteria for compositional reasoning and object editing, respectively.Source: Table S4 · p. 13, Table S5 · p. 13

D.2. Iterative Refinement Case Study: Semantic Segmentation

A tea-kettle segmentation case study shows how TAF supports iterative analysis and refinement. Geometric segmentation improved, but the VLM retained an initial semantic labeling error.

D.2. Iterative Refinement Case Study: Semantic Segmentation

  • The first iteration proposed five kettle parts and used height and radial distance as primary segmentation features.The multi-view point-cloud visualization supported the initial component hypothesis.Source: §D.2 · p. 14
  • The second iteration filtered the base and lid by height, separated the body by radial distance, and clustered outer points to distinguish spout and handle.A six-panel visualization was generated for inspection.Source: §D.2 · p. 14
  • TAF inspection revealed under-segmentation of the lid and body and over-segmentation of the handle, prompting adjusted thresholds, DBSCAN settings, and geometric heuristics.The handle was distinguished by its arc shape and vertical dominance, while the spout was compact and linear.Source: §D.2 · p. 14
  • The third iteration confirmed successful geometric segmentation, but the VLM retained incorrect semantic labels despite the visible legend error.The authors say this self-correction failure was rare and may relate to task complexity and VLM hallucinations.Source: §D.2 · p. 14

E. Error Analysis

Manual inspection identifies invalid programs, insufficient geometric fidelity, and hallucinations as error modes in 3D-Prog.

E. Error Analysis

  • Incorrect programs are the most common failure mode and can produce erroneous code or invalid USD assets that cannot be rendered.This occurs more often in the baseline system; iterative feedback significantly reduces these issues in 3D-Prog.Source: E. Error Analysis · p. 14
  • Some outputs lack fine-grained geometric detail or oversimplify geometry; the independent judge helps reduce this through stronger critical feedback.Source: E. Error Analysis · p. 14
  • In a few runs, VLM hallucinations caused a mismatch between correct scene reasoning and the generated output.Source: E. Error Analysis · p. 14

F. Qualitative Results

Qualitative examples cover editing, detection, scan processing, and real-image programming. The supplemental design details describe an iterative generation-and-feedback workflow with task-specific prompts.

F. Qualitative Results

  • Object editing changes appearance and physical characteristics, detection identifies chairs in cluttered incomplete scans, and scan simplification and completion produce simulation-ready assets.The examples span diverse object classes and sizes.Source: F. Qualitative Results · pp. 14–15, Figure S2 · p. 15

F.1. 3D-Prog for 3D Programming in the Wild

  • On self-captured phone images, 3D-Prog generates and places a table and creates a second apple with metallic texture.The first apple was reconstructed using SAM 3D [10].Source: §F.1 · p. 15

G. Design Choices

G.1. 3D-Prog Algorithm

  • The algorithm analyzes geometry, generates and executes USD code, compares the result with a reference when available, and uses feedback until approval or the iteration limit.Without a reference, it evaluates output quality instead.Source: §G.1 · p. 15, §G.2 · p. 16

G.2. Full Prompts for 3D-Prog

  • From the second iteration, prompts combine six-view comparative overlays with task-dependent TAF visualizations and judge feedback.The textual prompt also includes system instructions, USD-Core modeling rules, and the user task.Source: §G.2 · pp. 15–16, Figure S4 · p. 16
Loading 2610.02021v1…