PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Agentic AI for Spatial Reasoning with a Dynamic API

Damiano Marsili, Rohun Agrawal, Yisong Yue, Georgia Gkioxari

arXiv:2502.06787v2cs.CV

TL;DR

VADAR addresses the decline of vision-language models on multi-step 3D spatial reasoning by dynamically generating a Pythonic API and synthesizing executable programs. It outperforms static visual-programming APIs and performs competitively with monolithic VLMs, while oracle specialists reveal a vision-model bottleneck.

  • Problem

    Vision-language models perform less well on 3D spatial reasoning, while static human-defined APIs restrict visual program synthesis to predefined functionality.

  • Method

    VADAR uses collaborating LLM agents to dynamically generate reusable API functions and synthesize executable Python programs for complex image questions.

  • Results

    VADAR outperforms ViperGPT and VisProg by more than 20% on CLEVR and OMNI3D-BENCH, while remaining competitive with state-of-the-art VLMs.

  • Takeaways & Limitations

    Dynamic APIs support a wider variety of 3D queries and provide interpretable execution traces for complex reasoning.

  • Takeaways & Limitations

    VADAR often struggles with queries requiring five or more inference steps.

Abstract

from arXiv · show

Visual reasoning -- the ability to interpret the visual world -- is crucial for embodied agents that operate within three-dimensional scenes. Progress in AI has led to vision and language models capable of answering questions from images. However, their performance declines when tasked with 3D spatial reasoning. To tackle the complexity of such reasoning problems, we introduce an agentic program synthesis approach where LLM agents collaboratively generate a Pythonic API with new functions to solve common subproblems. Our method overcomes limitations of prior approaches that rely on a static, human-defined API, allowing it to handle a wider range of queries. To assess AI capabilities for 3D understanding, we introduce a new benchmark of queries involving multiple steps of grounding and inference. We show that our method outperforms prior zero-shot models for visual reasoning in 3D and empirically validate the effectiveness of our agentic framework for 3D spatial reasoning tasks. Project website: https://glab-caltech.github.io/vadar/

1. Introduction

VADAR targets multi-step 3D spatial reasoning, where current vision-language models decline beyond category-level understanding. It dynamically builds reusable functions and programs, outperforming static APIs and showing that specialized vision models are a key bottleneck.

  • Current vision-language models perform well on category-level semantics but decline on spatial understanding, with GPT4o producing a wrong answer on the motivating query.
  • 3D spatial queries require locating objects, estimating pixel dimensions and depth, and combining measurements through multiple grounding and inference steps.
  • VADAR uses collaborating LLM agents to dynamically define and expand a domain-specific language across API generation and program synthesis.
  • The evaluation combines CLEVR with OMNI3D-BENCH, whose queries emphasize relative depth, size, location, grounding, and 3D inference.
  • VADAR outperforms a no-API agent by 6% and exceeds static human-defined APIs by more than 20% on CLEVR.
  • 83.0% accuracy with oracle vision specialists indicates that VADAR is bottlenecked by its component vision models.

2. Related Work

Prior work establishes gaps in VLM spatial reasoning, static visual program synthesis, and 3D benchmarks. VADAR relates these areas through dynamic library creation and training-free neuro-symbolic reasoning.

  • VLMs show limited ability beyond category-level semantics, including grounding spatial relationships and inferring object-centric attributes.
  • Visual program synthesis composes vision specialists symbolically, but existing methods use specified or static APIs and can ignore parts of complex queries.
  • Library-learning methods dynamically propose programs and synthesize reusable components across domains including gaming, graphics, theorem proving, and symbolic regression.
  • VADAR achieves the same CLEVR performance as LEFT trained with at least 10,000 samples while remaining training-free and scaling to OMNI3D-BENCH.
  • Existing benchmarks often test single-step recognition, localization, 2D relations, absolute measurements, or depth ordering rather than complex 3D understanding.

3. Method

VADAR first generates a dynamic API of reusable functions, then synthesizes and executes Python programs for individual image questions. Agents and deterministic Python-based testing iteratively implement dependencies and repair errors.

  • API Generation: The dynamic API decomposes complex reasoning into reusable modules and can be extended with functions for novel queries.
  • API Generation: Vision specialists initialize the API with localization, 2D size, depth, and visual question-answering functions.
  • API Generation: The Signature Agent receives batches of 15 unanswered queries and proposes general method signatures, which the Implementation Agent implements in Python.
  • API Generation: The Signature Agent avoids duplication using the current API state, while omitting in-context examples yields a more diverse API.
  • API Generation: Implementation uses other signatures to form an API hierarchy, and weak in-context examples improve implementation accuracy.
  • API Generation: The Test Agent runs implementations with placeholder inputs and sends runtime exceptions back for revision, traversing dependencies when prerequisite methods are incomplete.
  • Program Synthesis: The Program Agent generates Python code from the API and question, while the Execution Agent runs it line by line and supports up to five error-repair attempts.
  • Program Synthesis: API-agnostic natural-language instructions and a planning step guide program generation because the dynamically created API cannot support fixed full-program examples.

4. Experiments

VADAR is evaluated on CLEVR and OMNI3D-BENCH, a new benchmark targeting multi-step 3D grounding and inference. Across comparisons and ablations, dynamic API generation improves program-based spatial reasoning, while vision-specialist quality remains a central limitation.

  • Comparisons: On OMNI3D-BENCH, VADAR trails GPT4o by 2% and exceeds all other evaluated VLMs by more than 5%.On CLEVR, VADAR exceeds VLMs on numeric answers by 1.0% and yes/no answers by 2.3%, while providing interpretable execution traces.
  • Comparisons: VADAR outperforms ViperGPT and VisProg on both CLEVR and OMNI3D-BENCH by more than 20%.Both VADAR and VisProg use GPT4o, whereas ViperGPT uses GPT-3.5.
  • Oracle analysis: With oracle vision specialists, VADAR reaches 83.0% on CLEVR and 94.4% on OMNI3D-BENCH, exceeding ViperGPT and VisProg.The oracle evaluation separates program correctness from execution accuracy and uses a smaller OMNI3D-BENCH subset of 50 queries.
  • Oracle analysis: Oracle results indicate that VADAR’s dynamically generated API supports a wider variety of queries than the static APIs in ViperGPT and VisProg.The API flexibly integrates vision specialists and avoids fixed design choices such as defining “behind” through image cropping.
  • Oracle analysis: VADAR’s oracle accuracy is more than 20% above Claude-Sonnet on CLEVR and more than 40% above GPT4o on OMNI3D-BENCH.The authors identify improving specialized vision models as a promising path because these models address simpler tasks with more accessible training data.
  • Additional evaluation: On GQA, VADAR and VisProg perform comparably, while ViperGPT shows a drop in accuracy.GQA emphasizes appearance and attributes with primarily one-step inference, limiting its usefulness for evaluating 3D spatial reasoning.
  • Ablations: The No-API Agent performs worst, while weak in-context examples and instructions produce the best ablation performance.The No-API variant directly creates programs without defining reusable API methods, supporting the value of the agentic API design.

5. Limitations & Future Work

VADAR remains limited on queries requiring five or more inference steps and on execution accuracy when vision specialists make errors. Future work targets prompting and dynamic selection of vision modules.

  • Limitations: Queries requiring 5 or more inference steps remain difficult for VADAR.The paper suggests advanced prompting strategies such as Chain-of-Thought and prompt chaining.
  • Limitations: VADAR achieves 83.0% program accuracy on CLEVR but only 53.6% execution accuracy because of vision-specialist errors.
  • Future Work: A proposed enhancement is dynamically selecting vision modules from a pool according to empirical performance.

Supplementary Material

The supplementary material documents experimental variability, runtime, prompts, predefined modules, and qualitative analyses for VADAR across its evaluated benchmarks.

  • Supplementary Material: Supplementary materials report standard deviations across three experimental runs and qualitative examples on CLEVR, OMNI3D-BENCH, and GQA.
  • Supplementary Material: Runtime is reported separately for each VADAR agent in seconds on an A100 GPU.
  • Supplementary Material: Predefined module docstrings initialize the dynamic API, with a benchmark-specific omission for CLEVR’s binary size representation.
  • Supplementary Material: The Signature Agent generates only necessary new methods, while the Implementation Agent uses weak ICL examples to implement them correctly.
  • Supplementary Material: Program Agent prompts use planning, natural-language instructions, and benchmark-specific definitions for attributes and dimensions.

B. Additional Quantitative Analysis

Across three runs, VADAR’s variability is comparable to VLMs on OMNI3D-BENCH but somewhat higher than program-synthesis methods. Signature and Implementation Agent costs become negligible when processing more than 1000 questions.

  • Experimental Variability: VADAR’s variation is comparable to VLMs on OMNI3D-BENCH but slightly higher than program-synthesis methods on both benchmarks.Despite this variation, VADAR significantly outperforms ViperGPT and VisProg.
  • Experimental Variability: VADAR significantly outperforms ViperGPT and VisProg even after accounting for experimental variation.
  • Runtime: For 1000+ questions, the Signature and Implementation Agents run only once, making their runtime negligible relative to total inference runtime.

C. More information on OMNI3D-BENCH

OMNI3D-BENCH is a free-form benchmark designed to test diverse, challenging 3D spatial reasoning. It complements CLEVR and remains difficult for current models, while VADAR outperforms Gemini1.5-Pro on a VSI-Bench image subset.

  • Benchmark Construction: OMNI3D-BENCH uses human-authored free-form questions focused on scene layout, object sizes, and complex 2D and 3D inference.
  • Benchmark Comparison: 50.1% accuracy on a 75-query VSI-Bench image subset exceeded 40.4% on OMNI3D-BENCH, underscoring the latter’s challenging nature.
  • Reasoning Coverage: OMNI3D-BENCH covers relative size, depth, spatial relationships, proportions, alignments, and object interactions.
  • Benchmark Construction: The benchmark contains 500 unique image-question-answer tuples computed from Omni3D’s 3D annotations.
  • Benchmark Design: CLEVR and OMNI3D-BENCH together provide a comprehensive test of 3D spatial reasoning using templated and free-form questions.

D. Comparison to VSI-Bench

The paper contrasts OMNI3D-BENCH with VSI-Bench, emphasizing image-based hypothetical 3D reasoning rather than video frame retrieval or absolute attribute estimation. VADAR performs comparably to Gemini1.5-Pro on VSI-Bench-img, while OMNI3D-BENCH is more challenging.

  • Benchmark differences: VSI-Bench focuses on video understanding, frame retrieval, absolute measurements, and templated queries, unlike OMNI3D-BENCH’s hypothetical 3D reasoning.OMNI3D-BENCH asks questions such as how tall a chair would be if a table were two meters wide.
  • Evaluation setup: VSI-Bench-img contains 75 queries paired with the frame containing the information needed for image-based reasoning.This variant decouples frame retrieval from the reasoning task.
  • Results: 10% higher performance on VSI-Bench-img than OMNI3D-BENCH: 50.1 versus 40.4.VADAR performs on par with the industry-leading Gemini1.5-Pro on VSI-Bench-img.

E. Qualitative Examples on CLEVR

Qualitative CLEVR examples show VADAR using API methods for repeated tasks and spatial relations, while also exposing failures in reference-object handling.

  • Qualitative Examples on CLEVR: VADAR correctly uses API methods for repeated tasks and determines spatial relations in a CLEVR example.The example illustrates executable program use for recurring operations.
  • Qualitative Examples on CLEVR: A CLEVR failure occurs when VADAR does not exclude the original reference object while searching for “another” object.The error concerns same-object handling in the generated reasoning process.
  • Qualitative Examples on CLEVR: The CLEVR examples therefore pair successful spatial reasoning with a concrete failure on reference-object exclusion.Both outcomes appear in the qualitative examples.

F. Qualitative Examples on OMNI3D-BENCH

Qualitative OMNI3D-BENCH examples show VADAR estimating 3D distances by scaling depth using a reference measurement, while also revealing incorrect scaling cases.

  • Qualitative Examples on OMNI3D-BENCH: VADAR correctly estimates 3D distances by scaling depth according to the reference scale specified in the question.The method uses a measurement hypothesis to convert visual depth into the queried spatial quantity.
  • Qualitative Examples on OMNI3D-BENCH: An incorrect depth-scaling instance appears in the final OMNI3D-BENCH qualitative example.The example identifies a failure mode in applying the reference-scale operation.
  • Qualitative Examples on OMNI3D-BENCH: The OMNI3D-BENCH examples demonstrate both successful reference-based depth scaling and an associated scaling error.The qualitative results expose the mechanism and its failure case.

G. Qualitative Examples on GQA

GQA examples show VADAR explicitly locating objects needed for answers, including nearest-object reasoning, while also exposing ambiguity in some questions and spatial references.

  • Qualitative Examples on GQA: VADAR identifies and locates key objects needed to answer GQA questions.The programs make the grounding steps explicit rather than relying only on an unstated visual judgment.
  • Qualitative Examples on GQA: In one GQA example, VADAR locates the nearest person using pixel distance from a tree.This illustrates explicit distance-based spatial reasoning.
  • Qualitative Examples on GQA: Some GQA questions have ambiguous answers because descriptions such as “round” and spatial frames of reference can be unclear.The cited examples include a pot’s generic shape and uncertainty about which man is the reference object.
Loading 2502.06787v2…