PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models

Mengfei Du, Binhao Wu, Zejun Li, Xuanjing Huang, Zhongyu Wei

arXiv:2406.05756v1cs.AIcs.CLcs.CVcs.MM

TL;DR

Spatial understanding in embodied environments is important for agents, but existing benchmarks do not adequately evaluate it. EmbSpatial-Bench assesses six egocentric relations in embodied scenes, and experiments find weak current LVLM performance; instruction tuning with EmbSpatial-SFT improves accuracy across settings.

  • Problem

    Spatial relationships are vital to embodied interaction, but existing benchmarks are not suitable for accurately assessing LVLMs’ spatial understanding in embodied environments.

    Source: §1 · p. 1

  • Method

    EmbSpatial-Bench is automatically derived from annotated embodied 3D scenes and evaluates six egocentric relationships; EmbSpatial-SFT provides instruction-tuning data for spatial understanding.

    Source: §1 · pp. 1–2, §2.1 · p. 2

  • Results

    EmbSpatial-SFT increases overall accuracy by 34.25% under likelihood evaluation and +9.04% overall under generation evaluation, while current LVLMs show weak embodied spatial understanding.

    Source: §1 · p. 2, §4.3 · p. 4

  • Takeaways & Limitations

    The study advances embodied spatial understanding by constructing benchmark and instruction-tuning datasets from well-annotated 3D embodied datasets.

    Source: Limitations · p. 5

    The datasets draw on indoor embodied environments, which may limit suitability for outdoor environments, and the study investigates only English, limiting generalizability to other languages.

    Source: Limitations · p. 5

Abstract

from arXiv · show

The recent rapid development of Large Vision-Language Models (LVLMs) has indicated their potential for embodied tasks.However, the critical skill of spatial understanding in embodied environments has not been thoroughly evaluated, leaving the gap between current LVLMs and qualified embodied intelligence unknown. Therefore, we construct EmbSpatial-Bench, a benchmark for evaluating embodied spatial understanding of LVLMs.The benchmark is automatically derived from embodied scenes and covers 6 spatial relationships from an egocentric perspective.Experiments expose the insufficient capacity of current LVLMs (even GPT-4V). We further present EmbSpatial-SFT, an instruction-tuning dataset designed to improve LVLMs' embodied spatial understanding.

1 Introduction

Spatial understanding is vital to embodied agents, yet existing benchmarks do not adequately evaluate it because they often use image-centered relationships and scenes weakly related to embodied tasks. EmbSpatial-Bench evaluates six egocentric relationships in embodied scenes, and experiments report weak performance by current LVLMs alongside improvements after instruction tuning on EmbSpatial-SFT.

1 Introduction

  • Spatial relationships are vital for embodied agents’ environmental interaction, but existing benchmarks are unsuitable for accurately assessing this ability.Prior benchmarks tend to describe relationships from the depicted subject’s perspective and use image datasets weakly related to embodied scenarios.Source: §1 · p. 1
  • EmbSpatial-Bench evaluates six egocentric spatial relationships—above, below, left, right, close, and far—in images from embodied 3D scenes.The benchmark uses multiple-choice questions and images from MP3D, AI2-THOR, and ScanNet.Source: §1 · pp. 1–2
  • Current LVLMs, including GPT-4V and Qwen-VL-Max, show poor embodied spatial understanding, while models fine-tuned on EmbSpatial-SFT improve across scenarios.The paper introduces EmbSpatial-SFT as an instruction-tuning dataset to improve this capability.Source: §1 · p. 2

2 EmbSpatial-Bench

EmbSpatial-Bench is built from annotated 3D embodied scenes by extracting spatial relations, generating multiple-choice questions, and filtering unsuitable examples. It contains 3,640 question-answer pairs across 294 object categories and six spatial relations.

2 EmbSpatial-Bench

  • The construction pipeline generates target images from 3D scenes, extracts object relations, then generates and filters QA pairs.The benchmark differs from prior 2D-image benchmarks by deriving its data from 3D scenes.Source: §2 · p. 2

2.1 Dataset Construction

  • Relations are extracted from annotated 3D coordinates projected into image bounding boxes, using box positions for four relations and average depth for close or far.The six relations are above, below, left, right, close, and far; the first four require non-overlapping boxes.Source: §2.1 · pp. 2–3
  • Questions use templates for above, below, left, and right, while close and far questions ask for the closest or farthest object in an image.QA pairs with overly large or small boxes are filtered, then samples with unclear objects or incorrect relations are removed.Source: §2.1 · p. 3

2.2 Dataset Statistics

  • 3,640 QA pairs cover 294 object categories and six spatial relationships in EmbSpatial-Bench.The object set is collected across three embodied datasets, with indoor objects such as chairs, bowls, and windows among the most frequent.Source: §2.2 · p. 3
  • The benchmark’s common spatial-relation triples show diverse object-relation combinations, and the relation distribution is balanced.The paper states that this diversity and balance enhance benchmark reliability.Source: §2.2 · p. 3

3 EmbSpatial-SFT

EmbSpatial-SFT supplies instruction-tuning QA data for spatial relation identification and object localization. Its relation-identification training data follows the benchmark pipeline, and localization answers use textual bounding boxes.

3 EmbSpatial-SFT

  • EmbSpatial-SFT combines spatial relationship identification with object localization, which serves as an auxiliary grounding task.The paper describes localization as a foundational skill for relationship identification and builds the dataset on MP3D’s training split.Source: §3 · p. 3
  • 25K training samples are constructed for spatial relation identification using the automatic benchmark pipeline.The training split supports zero-shot evaluation on data from the other two scenes in EmbSpatial-Bench.Source: §3 · p. 3
  • Object-localization examples ask the model to give an inquired object’s location as a textual bounding box.The format follows Chen et al. (2023a).Source: §3 · p. 4

4 Experiments

Zero-shot evaluations expose weak embodied spatial understanding in current LVLMs, while instruction-tuning on EmbSpatial-SFT improves MiniGPT-v2 across evaluation settings.

4.1 Experimental Setup

  • Zero-shot evaluation uses accuracy and compares generation-based predictions with a likelihood strategy that selects the highest-probability option.The generation strategy reads predicted options from model text outputs.Source: §4.1 · p. 4

4.2 Zero-shot Performance

  • 49.11% (Generation) and 43.85% (Likelihood) are the best LVLM accuracies, versus 90.33% human performance.The evaluated models include GPT-4V and Qwen-VL-Max.Source: §4.2 · p. 4

4.3 Instruction Tuning on EmbSpatial-SFT

  • 34.25% overall-accuracy increase under likelihood and +9.04% overall under generation follow instruction-tuning on EmbSpatial-SFT.The improvements span in-domain and out-of-domain environments under likelihood evaluation.Source: Table 3 · p. 4, §4.3 · p. 4
  • LoRA tuning significantly contributes across scenarios, while auxiliary object-localization data add 0.47% overall under generation and 0.76% under likelihood.The LoRA comparison is against a variant with a frozen LLM backbone.Source: §4.3 · p. 4

5 Related Works

Prior work explored LVLMs for embodied tasks but did not evaluate or enhance their spatial understanding; existing spatial benchmarks do not align well with embodied scenarios.

Large Vision-Language Models

  • Further LVLM-based embodied-task work (Tsai et al., 2023; Zheng et al., 2023) did not evaluate or empower spatial understanding.The cited work preliminarily indicates LVLMs’ potential as embodied intelligence.Source: §5 · p. 5

Benchmarks for Spatial Understanding

6 Conclusion

The paper proposes EmbSpatial-Bench to evaluate LVLMs’ embodied spatial understanding and EmbSpatial-SFT to enhance that capacity; experiments report weak current-model performance and tuning-data effectiveness.

6 Conclusion

  • EmbSpatial-Bench evaluates embodied spatial understanding, and EmbSpatial-SFT is an instruction-tuning dataset whose data components are experimentally validated for enhancing LVLM capacity.The evaluation results reveal weak spatial understanding among current popular LVLMs.Source: §6 · p. 5

Limitations

The study’s scope is limited by its indoor embodied datasets and English-only investigation.

Limitations

  • The datasets come from three widely used indoor embodied datasets, which may restrict suitability for outdoor environments.The passage identifies this as a scope boundary for the datasets.Source: Limitations · p. 5
  • The study investigates only English, limiting generalizability of the benchmark and findings to other languages.Source: Limitations · p. 5

Ethical Considerations

The benchmark and instruction-tuning data use publicly available embodied datasets, and the authors report that the sources contain no personal data or offensive content. For AI2-THOR, key images are selected after sub-goals to avoid redundancy in observation sequences.

Ethical Considerations

  • The data sources are publicly available embodied datasets, including photorealistic or generated rendered scenes, and are reported to have no copyright issues.The authors also report no personal data, uniquely identifiable individuals, or offensive content in the data source.Source: Ethical Considerations · p. 5
  • The authors report that the data source contains no personal data, uniquely identifiable individuals, or offensive content.Source: Ethical Considerations · p. 5

A.1 AI2-THOR Image Selection

  • For AI2-THOR, key images showing content changes after each sub-goal are selected using detailed ALFRED PDDL annotations, addressing similarity across observation sequences.Source: §A.1 · p. 8

A.2 Dataset Statistics

The benchmark statistics describe a varied, balanced set of questions and samples drawn from three embodied datasets. The section also details sample verification, filtering, model evaluation, and experimental setup.

A.2 Dataset Statistics

  • The benchmark’s question distribution spans spatial relations, and the object-category distribution is diverse and balanced, which the authors say enhances reliability.The most frequent objects across scenes include chairs, bowls, and windows.Source: §A.2 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

A.3 Data Cases

  • Examples are shown from MP3D, AI2-THOR, and ScanNet, illustrating samples drawn from all three embodied datasets.Source: §A.3 · p. 8, Figure 7 · p. 9

A.4 Filtering and Verification

  • Filtering removes spatial-relation triples with bounding-box dimensions below 50 pixels or above half the corresponding image dimension; human review checks object clarity, relation correctness, and negative options.Samples failing any human-verification condition are discarded.Source: §A.4 · p. 8

B.1 Experimental Details

  • MiniGPT-v2 is the baseline LVLM, with its connection layer and language model fine-tuned using LoRA; training runs for 25,000 steps on four V100 GPUs.The reported setup uses AdamW, a cosine learning-rate schedule, and a global batch size of 16.Source: §B.1 · p. 8

B.2 Evaluation Strategy

  • Evaluation uses generation and likelihood strategies; likelihood scores the options given an image and question, while generation extracts an option mark from text.Source: §B.2 · pp. 8–9

B.3 Models

  • The evaluation covers 12 LVLMs—10 open-source and two closed-source—including GPT-4V and Qwen-VL-Max; the open-source models are approximately 7B parameters.The selected GPT-4V version is gpt-4-1106-vision-preview.Source: §B.3 · p. 9

B.4 Main Results of Each Spatial Relation

After instruction-tuning on EmbSpatial-SFT, MiniGPT-v2 improves or maintains comparable accuracy across spatial-relation categories and environments. The results also identify weaker depth and generation performance, while GPT-4V cases show errors in both localization and relation judgment.

B.4 Main Results of Each Spatial Relation

  • Likelihood performance is significantly lower for depth relations; the authors point to missing depth-estimation training and the need to identify four objects, while generation remains poor after tuning.Improving open-source models’ generation performance remains an open question.Source: §B.4 · p. 9
  • MiniGPT-v2 significantly improved or maintained comparable accuracy across spatial-relation categories and environments after EmbSpatial-SFT instruction-tuning.Source: §B.4 · p. 9, Table 4 · p. 10

Appendix C GPT-4V Cases

Loading 2406.05756v1…