PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models

Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, David Acuna

arXiv:2409.09788v1cs.CVcs.CL

TL;DR

Quantitative reasoning about object sizes and distances remains underexplored in vision-language models. The paper introduces Q-Spatial Bench and SpatialPrompt, which encourages models to use reference objects as visual cues. SpatialPrompt improves success rates across several VLMs without additional data, architectural modifications, or fine-tuning.

  • Problem

    Quantitative reasoning about object sizes and distances remains underexplored, despite VLMs’ ability to describe image relationships in natural language.

    Source: §1 · p. 1

  • Method

    The paper introduces Q-Spatial Bench and a zero-shot SpatialPrompt technique that guides VLMs to reason about quantitative spatial questions using reference objects.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    SpatialPrompt raises success rates by 47, 22, and 30 points for Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4V, respectively, on Q-Spatial Bench.

    Source: §5.2.1 · p. 7

  • Takeaways & Limitations

    The reported gains are obtained through natural-language instructions without more data, architectural modifications, or fine-tuning.

    Source: §1 · p. 2

    Q-Spatial Bench is limited in size by the costly labeling process, and SpatialPrompt may be constrained by inference speed, image context, and model scale.

    Source: §6 · p. 9

Abstract

from arXiv · show

Despite recent advances demonstrating vision-language models' (VLMs) abilities to describe complex relationships in images using natural language, their capability to quantitatively reason about object sizes and distances remains underexplored. In this work, we introduce a manually annotated benchmark, Q-Spatial Bench, with 271 questions across five categories designed for quantitative spatial reasoning and systematically investigate the performance of state-of-the-art VLMs on this task. Our analysis reveals that reasoning about distances between objects is particularly challenging for SoTA VLMs; however, some VLMs significantly outperform others, with an over 40-point gap between the two best performing models. We also make the surprising observation that the success rate of the top-performing VLM increases by 19 points when a reasoning path using a reference object emerges naturally in the response. Inspired by this observation, we develop a zero-shot prompting technique, SpatialPrompt, that encourages VLMs to answer quantitative spatial questions using reference objects as visual cues. By instructing VLMs to use reference objects in their reasoning paths via SpatialPrompt, Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4V improve their success rates by over 40, 20, and 30 points, respectively. We emphasize that these significant improvements are obtained without needing more data, model architectural modifications, or fine-tuning.

1 Introduction

Quantitative spatial reasoning about object sizes and distances remains underexplored, while existing benchmarks emphasize qualitative relations or rely on potentially inaccurate labels. The paper introduces Q-Spatial Bench and SpatialPrompt, a reference-object prompting technique that improves VLM success rates without additional data or model changes.

1 Introduction

  • Quantitative spatial reasoning remains underexplored, as existing benchmarks primarily test qualitative relations such as left versus right or near versus far.Estimating sizes and distances from monocular images is ill-posed, although humans use contextual clues to make such estimates.Source: §1 · p. 1
  • Q-Spatial Bench is a high-precision question-answering corpus designed to evaluate VLMs on quantitative spatial reasoning.The benchmark is manually annotated.Source: §1 · p. 2
  • The authors propose SpatialPrompt after observing that VLM responses are more effective when their reasoning paths involve reference objects.The technique guides VLMs to use reference objects and avoids requiring direct model access, task-specific data, architectural modifications, or fine-tuning.Source: §1 · p. 2
  • Over 47, 22, and 30 points, SpatialPrompt improves success rates for Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4V, respectively, without more data, architectural changes, or fine-tuning.The authors contrast this with SpatialVLM’s reported improvement of less than 4 points after fine-tuning on 2 billion spatial QA pairs; the evaluation benchmarks are not directly comparable.Source: §1 · p. 2

2 Related Works

Prior spatial benchmarks largely assess qualitative relations, while quantitative size and distance estimation remains relatively underexplored. The paper positions Q-Spatial Bench as a manually annotated, high-precision benchmark and SpatialPrompt as a prompting alternative to training- or architecture-based approaches.

2.1 Spatial Reasoning Benchmarks

  • Many existing benchmarks test qualitative relations such as left versus right or behind versus in front, and some address linguistic complexity or spatial biases.The passage also notes that spatial reasoning is entangled with other challenges in many datasets.Source: §2.1 · p. 2
  • Quantitative size and distance estimation is relatively underexplored; Q-Spatial Bench provides a human-annotated benchmark with high-precision measurements and images captured to reduce test-data leakage risk.Spatial VQA is not publicly available, while SpatialRGPT-Bench is automatically generated and human-verified.Source: §2.1 · p. 2

2.2 Spatial Reasoning in VLMs

  • Prior approaches use depth information, sequential RGB images, fine-tuning, or relative-depth plugins; this work focuses on quantitative spatial reasoning from a single monocular image using prompting.SpatialPrompt is presented as avoiding direct model access, additional task-specific data, architectural modifications, and fine-tuning.Source: §2.2 · pp. 2–3
  • Q-Spatial Bench has two splits, Q-Spatial-ScanNet and Q-Spatial++, and covers five categories of quantitative spatial questions.Figure 1 presents examples from both splits and the five categories; Q-Spatial-ScanNet is repurposed from ScanNet and Q-Spatial++ uses additional captured images.Source: Figure 1 · p. 3

3 Quantitative Spatial Reasoning

The paper frames direct quantitative spatial reasoning as estimating physical-object sizes or distances from a 2D image, then introduces Q-Spatial Bench for evaluating it. The benchmark combines annotated ScanNet examples with freshly captured images, and evaluates estimates using ratio-threshold success rates.

3.1 Preliminaries

  • Direct quantitative spatial reasoning asks a VLM to estimate physical-object quantities such as sizes or distances from a 2D image without external tools or large models.The task is formulated as question answering, with the estimated quantity output in text.Source: §3.1 · p. 3

3.2 Benchmark: Q-Spatial Bench

  • Q-Spatial Bench contains 271 questions across five categories, with two splits: Q-Spatial-ScanNet and Q-Spatial++.The categories cover object width, object height, horizontal and vertical distances between objects, and direct distance between objects.Source: §3.2 · pp. 3–4
  • Q-Spatial-ScanNet uses 99 ScanNet images and 170 expert-annotated questions, with annotators measuring distances in dataset point clouds.The annotation pipeline selects images, lets experts mark distance endpoints in downsampled point clouds, converts image-distance-description triplets into QA pairs with Llama 3-70b, and manually verifies the pairs.Source: §3.2 · p. 4
  • The caption reports that GPT-4o outperforms other commercial VLMs in quantitative spatial reasoning; success rate δ≤2 was evaluated on each Q-Spatial Bench split.The caption notes that Gemini 1.5 Pro consistently refuses to provide the measurements.Source: Table 2 · p. 4
  • Q-Spatial++ uses 101 diverse iPhone-captured images across indoor, outdoor, daytime, and nighttime settings, with 101 annotated questions; real-world object distances were physically measured, focusing on horizontal distances.Source: §3.2 · p. 4

3.3 Evaluation metrics

  • Success is measured by thresholding the maximum ratio between an estimate and its ground truth, with δ≤2 as the main metric.The δ≤2 criterion counts estimates within 0.5× to 2× the ground truth; δ≤1.25 uses the narrower 0.8× to 1.25× range.Source: §3.3 · p. 4

4 Analysis

The analysis finds that distance questions challenge state-of-the-art VLMs, while GPT-4o leads the benchmark. GPT-4o answers were more often accurate when they used reference objects, and regression analysis supports a significant association.

4.1 Strengths and Weaknesses

  • GPT-4o leads both Q-Spatial Bench splits, with success rates of 69.41 and 62.44 and a margin of over 41 points over GPT-4V.The success-rate metric is δ≤2.Source: §4.1 · p. 5
  • Horizontal distance is especially challenging: it has the lowest success rates across models, and GPT-4o leads the second-best model by over 35 points.Unspecified measurements from Gemini 1.5 Pro are counted as incorrect answers.Source: §4.1 · p. 5, Table 3 · p. 5

4.2 Understanding the dominance of GPT-4o

  • GPT-4o used reference objects in 23.4% of instances; success was 83% (45/54) with them versus 64% (114/177) without them.A logistic regression controlling for dataset split and ground-truth distance found reference-object use associated with 2.7 times the odds of an accurate estimate (p-value < 0.05).Source: Table 4 · p. 6, §4.2 · p. 6

5 SpatialPrompt: Eliciting Reasoning Paths with Reference Objects

SpatialPrompt asks VLMs to identify reference objects as visual cues, using either a compact prompt or explicit steps. It improves success rates across the tested commercial models and produces mixed results on LLaVA.

5 SpatialPrompt: Eliciting Reasoning Paths with Reference Objects

  • SpatialPrompt explicitly encourages VLMs to identify reference objects in images, without additional data, fine-tuning, or architectural modifications.The approach is presented as orthogonal to prior work.Source: §5 · p. 6

5.1 Prompt Design

  • SpatialPrompt-Single is compact, while SpatialPrompt-Steps breaks the solution into more detailed instructions.Gemini 1.5 Pro follows detailed instructions more effectively; GPT-4V’s performance depends less on prompt complexity when encouraged to find a reference object.Source: §5.1 · p. 7

5.2 Experiments

  • The experiments compare SpatialPrompt with a standard prompt containing only the spatial question and task-agnostic zero-shot CoT, which appends “Let’s think step by step.”The paper also tracks how often responses use reference objects and evaluates open-source VLMs.Source: §5.2 · p. 7

5.2.1 Main results

  • SpatialPrompt improves success rates by 47, 22, 30, and 2 points for Gemini 1.5 Pro, Gemini 1.5 Flash, GPT-4V, and GPT-4o, respectively, reducing GPT-4o’s lead over the second-best model from 41 to 13 points.The tested prompts were SpatialPrompt-Single for GPT-4V and GPT-4o, and SpatialPrompt-Steps for Gemini 1.5 Pro and Gemini 1.5 Flash.Source: §5.2.1 · p. 7

5.2.2 Does SpatialPrompt encourage the use of references?

  • Reference-object frequency and success rate δ≤2 are positively correlated, with Spearman coefficients of 0.69 for Q-Spatial-ScanNet and 0.61 for Q-Spatial++.The authors interpret this as empirical corroboration of their hypothesis.Source: §5.2.2 · p. 7

5.2.3 Open-sourced VLMs

  • LLaVA scores 60.59 on Q-Spatial-ScanNet but drops by more than 20 points on Q-Spatial++; SpatialPrompt-Steps decreases its ScanNet score by around 9 points and improves its Q-Spatial++ score by 9 points.The authors report that its outputs appear to predict numbers rather than show intuitive reasoning; they offer possible explanations for its mixed prompting results.Source: §5.2.3 · p. 8

6 Limitations

The authors identify constraints in both Q-Spatial Bench and SpatialPrompt: costly annotation limits benchmark size, while prompting may trade inference speed for performance and may be less reliable across image contexts and model scales.

6 Limitations

  • Time-consuming annotation, especially precise distance measurement, leaves Q-Spatial Bench around four times smaller than prior benchmarks.Finding endpoints for ambiguous distances can require annotators to inspect nearby points in the point cloud.Source: §6 · p. 9
  • SpatialPrompt adds tokens by eliciting detailed reasoning paths, making it less suitable for real-time inference.Source: §6 · p. 9
  • Images with clean backgrounds may provide too few visual cues, so SpatialPrompt can hurt or confuse VLMs.Source: §6 · p. 9
  • SpatialPrompt’s gains are inconsistent for open-source VLMs, and whether it works only at sufficiently large model scales remains unclear.Source: §6 · p. 9

7 Conclusion

The paper introduces Q-Spatial Bench for evaluating quantitative spatial reasoning across five question categories and analyzes current VLMs’ strengths and weaknesses. It also reports that prompting models to use reference objects improves performance without additional data or fine-tuning.

7 Conclusion

  • Q-Spatial Bench is a benchmark for quantitative spatial reasoning with five question categories.Source: §7 · p. 9
  • Systematic analysis identifies strengths and weaknesses of current state-of-the-art VLMs on the task.Source: §7 · p. 9
  • SpatialPrompt leverages naturally emerging reasoning paths by instructing VLMs to use reference objects, significantly boosting performance without additional data or fine-tuning.Source: §7 · p. 9

A Table of Contents

The appendix contents list full experimental results, text prompts, qualitative results, implementation details, and licensing information.

B Full experimental results

The appendix reports human and model evaluations, prompt-condition results, and reference-object use, then documents the full prompt templates. Its results include a substantial human–GPT-4o gap and model-specific differences in which prompting procedures work best.

B Full experimental results

  • Gemini 1.5 Pro works only with detailed procedures, Gemini 1.5 Flash responds well to abstract and detailed procedures, and detailed procedures hurt GPT-4V and GPT-4o.The authors hypothesize that detailed constraints limit the reasoning paths of the more capable GPT models.Source: B Full experimental results · p. 12, Table 8 · p. 13
  • LLaVAv1.6-34b performs strongly on Q-Spatial-ScanNet but drops more than 20 points on Q-Spatial++; LLaVA variants also prefer different prompts (Liu et al., 2024, 2023b).The authors connect these inconsistent preferences to prior findings that models under 100B parameters exhibit unreliable reasoning paths.Source: B Full experimental results · p. 12, Table 9 · p. 13
  • Standard deviations generally increase under the stricter δ≤1.25 evaluation threshold compared with δ≤2.Source: B Full experimental results · p. 12
  • SpatialPrompt increases reference-object use the most by explicitly asking models to use objects in the image.Zero-shot CoT also increases the likelihood of reference-object use.Source: B Full experimental results · p. 12
  • Humans exceed GPT-4o’s success rate δ≤2 by more than 30 points on a sample of 40 questions, indicating the benchmark is easy for humans but difficult for VLMs.Source: B Full experimental results · p. 12

C Full text prompt

  • The prompt appendix includes a reference-object detection prompt, standard and LLaVA-adapted prompts, SpatialPrompt-Steps, and a prompt converting image-distance-description triplets into QA pairs.Source: C Full text prompt · p. 12

D Qualitative results

Prompting changed how GPT-4V approached quantitative spatial questions: zero-shot CoT and SpatialPrompt elicited use of surrounding objects, unlike the rough estimates under the standard prompt. SpatialPrompt was the only evaluated prompt to elicit quantitative spatial reasoning from Gemini 1.5 Pro; GPT-4o sometimes used floor tiles as references, leading to inaccurate estimates.

D Qualitative results

  • With the standard prompt, GPT-4V tended to make rough estimates, while zero-shot CoT and SpatialPrompt elicited reasoning using surrounding objects.The passage describes these behaviors in Fig. 11.Source: D Qualitative results · p. 13
  • Among evaluated prompts, SpatialPrompt alone successfully elicited quantitative spatial reasoning in Gemini 1.5 Pro.The passage describes these responses in Fig. 12.Source: D Qualitative results · p. 13
  • GPT-4o tended to use floor tiles as references, leading at times to inaccurate estimates.The passage identifies this as a common failure case shown in Fig. 13.Source: D Qualitative results · p. 13

E Implementation details

The experiments used deterministic sampling with model-specific seed counts and API endpoints. The authors also report sampling settings for VLM experiments and default statsmodels parameters for logistic regression.

E Implementation details

  • Experiments used deterministic sampling with five seeds, except GPT-4V and GPT-4o, which used three, and specified dated API endpoints.The endpoints were gpt-4o-2024-05-13, gpt-4-turbo-2024-04-09, gemini-1.5-pro-001, and gemini-1.5-flash-001; these experiments used the provided API.Source: E Implementation details · p. 13
  • The authors report that top_p=1−14 and temperature=1 performed better than top_p=1 and temperature=0, and used the former settings for experiments.They note that most VLM APIs remain random even at temperature 0.Source: E Implementation details · p. 13
  • LLaVA experiments used temperature=0 and top_p=1 on a machine with two A100 GPUs.Source: E Implementation details · p. 13
  • Logistic regression used the default statsmodels parameters.Source: E Implementation details · p. 13
Loading 2409.09788v1…