PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied Scene Understanding for Vision Language Models via MetaVQA

Weizhen Wang, Chenda Duan, Zhenghao Peng, Yuxin Liu, Bolei Zhou

arXiv:2501.09167v1cs.CVcs.RO

TL;DR

VLMs have limited spatial and embodied understanding, and standardized evaluation across VQA and closed-loop driving is lacking. MetaVQA combines Set-of-Mark question generation from real and simulated traffic with closed-loop simulation. The authors report improved VQA and driving performance after fine-tuning, including transfer from simulation to real-world VQA.

  • Problem

    General-purpose VLMs have limited spatial awareness and embodied interaction training, while existing driving benchmarks lack consistent conventions and scalable closed-loop evaluation.

    Source: §1 · pp. 1–2

  • Method

    MetaVQA uses Set-of-Mark prompting and scene-graph-generated VQA from real and simulated traffic, alongside closed-loop driving simulation.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2.3 · p. 4

  • Results

    Fine-tuning improves VQA accuracy and closed-loop driving performance, with simulated-data learning transferring to real-world VQA.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Fine-tuned VLMs show reasonable decision-making in an unseen driving task, supporting the generalizability of learned embodied knowledge.

    Source: §1 · p. 2

    MetaVQA assumes perception is mostly solved and directly highlights relevant traffic objects.

    Source: §3.2.2 · p. 4

Abstract

from arXiv · show

Vision Language Models (VLMs) demonstrate significant potential as embodied AI agents for various mobility applications. However, a standardized, closed-loop benchmark for evaluating their spatial reasoning and sequential decision-making capabilities is lacking. To address this, we present MetaVQA: a comprehensive benchmark designed to assess and enhance VLMs' understanding of spatial relationships and scene dynamics through Visual Question Answering (VQA) and closed-loop simulations. MetaVQA leverages Set-of-Mark prompting and top-down view ground-truth annotations from nuScenes and Waymo datasets to automatically generate extensive question-answer pairs based on diverse real-world traffic scenarios, ensuring object-centric and context-rich instructions. Our experiments show that fine-tuning VLMs with the MetaVQA dataset significantly improves their spatial reasoning and embodied scene comprehension in safety-critical simulations, evident not only in improved VQA accuracies but also in emerging safety-aware driving maneuvers. In addition, the learning demonstrates strong transferability from simulation to real-world observation. Code and data will be publicly available at https://metadriverse.github.io/metavqa .

1. Introduction

MetaVQA targets limits in VLM spatial and embodied understanding and the lack of comparable, scalable evaluation across open-loop questions and closed-loop driving. It combines a driving VQA benchmark with simulation-based evaluation, and reports improvements after fine-tuning, including transfer to real-world observations.

1. Introduction

  • General-purpose VLMs have limited spatial awareness after offline text-and-image pretraining and limited embodied understanding because interaction data are scarce.Source: §1 · p. 1
  • Existing driving VQA benchmarks use inconsistent, unintuitive object references, making poor performance hard to attribute to scene understanding rather than prompt interpretation.Source: §1 · p. 1
  • Prior evaluations often use open-loop VQA, while scalable closed-loop tests with consequential actions and safety-critical scenarios remain needed.Source: §1 · pp. 1–2
  • MetaVQA combines zero-shot VLM evaluation, fine-tuning, and closed-loop simulation in safety-critical driving scenarios.It uses real and simulated observations annotated with Set-of-Mark prompting and traffic scenarios from nuScenes and Waymo.Source: §1 · p. 2, Figure 1 · p. 2
  • Fine-tuning improves VQA accuracy and closed-loop driving performance, with learned capabilities transferring from simulation to real-world VQA.The paper reports better nuScenes real-world VQA after training on synthetic VQA data and reasonable decision-making in an unseen driving task.Source: §1 · p. 2

2. Related Work

Prior work spans driving-scene datasets, VLM driving agents, and simulator-based evaluation, but the paper identifies limitations in data diversity and interactive assessment. MetaVQA aims to address these gaps with varied question types and interpretable scene-understanding measures.

Driving scene understanding datasets

  • Driving-scene datasets cover descriptions, instructions, object importance, and attributes, but often rely on limited traffic data and lack diverse question types.Examples include BDD-X and Talk2Car for descriptions and directions, Rank2Tell for object importance, and nuScenes-QA for road-element attributes.Source: §2 · pp. 2–3

Vision language models as driving agents; Closed-loop evaluation of end-to-end driving agents

  • Many driving VLMs are evaluated offline, while simulator metrics such as success and route completion often lack interpretability.The paper motivates closed-loop evaluation and more meaningful measures that include scene understanding and behavioral reasonableness.Source: §2 · p. 3

3. Constructing MetaVQA Dataset

MetaVQA is designed to make VLM scene-understanding evaluation more intuitive and comparable, while covering spatial relations and action consequences. Its pipeline aggregates real-world traffic, annotates objects for visual grounding, and generates question-answer pairs from scene graphs.

3.1. Our Design Principles

  • Set-of-Mark prompting, single-correct-option multiple choice, and discretized spatial language aim to make questions clear and zero-shot evaluation fair.Fine-grained numerical descriptions supplement common phrases such as “front,” and the benchmark covers 30 question types across real and simulated observations.Source: §3.1 · pp. 3–4

3.2. VQA Generation Pipeline

  • A search-based pipeline programmatically extracts answers for template-generated questions from scene graphs and human-checks dataset quality.Source: §3.2 · p. 4

3.2.1. Scenario Aggregation from Multiple Sources.

  • MetaVQA draws on nuScenes and Waymo traffic, reconstructing Waymo scenarios in MetaDrive because WOMD lacks RGB observations.nuScenes scenarios are also represented as digital twins to augment appearance diversity.Source: §3.2.1 · p. 4

3.2.2. Set-of-Mark Prompting

  • Set-of-Mark boxes highlight traffic objects, using projected 3D boxes for nuScenes images and instance segmentation for simulated images.The design assumes perception is mostly solved and uses the marks to support clear visual grounding.Source: §3.2.2 · p. 4, Figure 2 · p. 4

3.2.3. Question-Answer Generation

  • Question generation selects a scene-graph object and uses question-bounded queries to produce a correct answer and multiple-choice candidates.An explanation field with dense captions is included in fine-tuning targets to deepen scene understanding; it is not used in evaluation.Source: §3.2.3 · p. 4, Figure 3 · p. 5

4. MetaVQA Dataset

MetaVQA is evaluated as a dataset for zero-shot VLM assessment and fine-tuning, with experiments examining answerability, transfer between simulated and real observations, and training-set scaling.

4.1. Dataset Composition

  • The corpus contains 4,305,450 questions across 442,102 annotated frames; its 9,725-question test set uses both reconstructed-simulation and real images.Questions cover spatial, embodied, and grounding categories; the training-only describe-scenario type asks models to describe all labeled objects.Source: §4.1 · p. 5

4.2. Zero-shot Answerability with Set-of-Mark Prompting

  • Novice participants averaged 88% accuracy on 35 sampled questions, supporting the answerability and directness of the question wording.Participants were six novice human respondents.Source: §4.2 · p. 5
  • 69.6% average zero-shot accuracy across 467 grounding questions; the best-performing VLM reached 87.4%.The grounding questions ask models to identify the randomly labeled object inside a white box.Source: §4.2 · p. 5, Table 1 · p. 6

4.4. Data Scalability of Learning

  • InternVL2-8B test performance scales with training-set size, supporting MetaVQA's large-corpus design.The experiment compares training sets of 150,000, 37,500, and 9,375 questions.Source: Table 3 · p. 6, §4.4 · p. 6

4.3. Transfer learning with simulated observations

  • InternVL2-8B performs best after mixed real-and-simulated training, while each observation domain also benefits the other's test performance.Training on simulated observations alone significantly increases real-world image test accuracy.Source: Table 2 · p. 6, §4.3 · p. 6

5. Benchmark Results

Fine-tuning on MetaVQA improved VQA performance, especially on spatial questions, and improved closed-loop driving metrics. The results also show above-random zero-shot performance for most models and qualitative differences in collision avoidance.

5.1. Visual Question Answering

  • Most models generated valid answers and performed above random guessing zero-shot, while LLaVA-NeXT fell below random and had a 27% failure rate.The paper also attributes LLaVA-NeXT’s low metrics to frequent refusals to answer.Source: §5.1 · p. 7
  • Fine-tuned models performed better across question supertypes, with the strongest improvements on spatial questions; InternVL2-8B achieved the best fine-tuned performance.The authors report comparable gains across models and illustrate improved pedestrian-heading understanding, multi-object reasoning, and grounding.Source: Figure 5 · p. 7, §5.1 · p. 8, Figure 7 · p. 8

5.2. Closed-loop Evaluation

  • The closed-loop evaluation used 120 scenarios and recorded collision, off-road, ADE, FDE, route-completion, and VQA-accuracy metrics.The scenarios included nuScenes examples and safety-critical cases generated from WOMD’s training split.Source: §5.2 · pp. 8–9
  • Fine-tuned VLMs improved route completion, off-road ratio, FDE, and VQA accuracy; ADE and collision rates generally improved, with some exceptions.The authors interpret improved zero-shot closed-loop performance as evidence that general-purpose VLMs become stronger adaptive embodied agents after learning MetaVQA.Source: §5.2 · p. 9, Table 5 · p. 9
  • In paired closed-loop cases, fine-tuned Llama3.2 avoided hazards more successfully than its zero-shot counterpart and fine-tuned Qwen2, which collided in the latter comparison.Both models received the same “forward” navigation command, so the reported evasions were made without behavioral hints from that command.Source: §5.2 · p. 9

6. Conclusion

MetaVQA combines real and simulated driving scenes with object-grounded question generation, then evaluates VLMs across VQA and closed-loop driving. The authors report transfer across observation domains and improved simulated driving, while noting limits in observation history and viewpoint.

6. Conclusion

  • MetaVQA supports VQA evaluation, sim-to-real transfer tests, and closed-loop driving evaluation, with improved driving capabilities reported for fine-tuned VLMs.The authors also identify image-only and single-perspective observations as limitations.Source: §6 · p. 9

A.1. Scenario Aggregation from Multiple Sources

  • Scenarios combine nuScenes and Waymo through MetaDrive and ScenarioNet, while nuScenes-real frames are filtered for object visibility and relevance.The nuScenes collection uses CAM FRONT images and removes objects with less than 50% visible projected area.Source: §A.1 · p. 14

A.2. Set-of-Mark Prompting

  • Set-of-Mark annotations place object labels on approximated boxes in nuScenes images and precise masks in simulated images, using a labeling algorithm and a fixed bounding-box style.The final style uses black text backgrounds and text size 1.00 to reduce label occlusions.Source: §A.2 · p. 15

A.3.1. Question Generation

  • Question templates are either parameterized, with concrete values sampled from parameter spaces, or non-parameterized, requiring examination of all visible objects.Parameterized templates may contain one or multiple parameter types.Source: §A.3.1 · pp. 15–16

A.3.2. Answer Generation

  • A question-specific query program extracts concrete answers from scenario information before the pairs are reformatted as multiple-choice questions.At this stage, question-answer pairs are not yet multiple choice.Source: §A.3.2 · p. 16

A.3.3. Post-processing

  • Post-processing creates distractor candidates, multiple-choice descriptions, and optional explanations, with candidate search spaces varying by question type.When applicable, candidate selection is designed to challenge evaluated VLMs.Source: §A.3.3 · p. 16

B.1. Dataset Composition

  • The question taxonomy groups question types by the objects they require models to analyze and the embodied scene-understanding facet they evaluate.Dotted contours indicate questions with similar formulations.Source: §B.1 · p. 16

B.2. Zero-shot Answerability with Set-of-Mark Prompting

A pilot questionnaire tested whether MetaVQA questions were answerable and clear, and its findings informed refinements to the final dataset. The appendix also reports a prompting-style comparison and illustrates question formats and taxonomy.

B.2.1. Human Evaluation

  • 88.05% average accuracy (SD 7.54%) across six participants on 35 pilot questions supported the authors’ assessment that the questions were intuitive and answerable.The best participant scored 94.2% and the worst 74.2%.Source: §B.2.1 · pp. 16–17
  • Questions 19 and 29 were difficult for participants, with five of six and four of six answering incorrectly, respectively.The authors speculate that wording ambiguity and distant objects contributed, then refined phrasing and visibility constraints.Source: §B.2.1 · pp. 16–17
  • Figures 13–15 illustrate question-answer generation for single-parameter, multiple-parameter, and non-parameterized questions.Source: Figure 13 · p. 17, Figure 14 · p. 17, Figure 15 · p. 17

B.3. Effect of Set-of-Marks Prompting Scheme

  • Bounding boxes with black text backgrounds and size 1.25 achieved Qwen2’s best overall and grounding performance, while the final dataset uses size 1.00 to reduce label occlusions.The authors report text size had a trivial impact on final performance.Source: §B.3 · p. 17
  • The question taxonomy organizes question types by required objects and evaluated facets, with dotted contours marking similar formulations.Source: Figure 16 · p. 18
  • Figure 17 presents a sample questionnaire question whose correct answer is choice C.Source: Figure 17 · p. 18

C. Benchmark Results

The appendix describes benchmark reporting across overall, simulated, and real-observation test splits, alongside parsing rules for model responses. It also includes naming conventions and examples of participant errors.

C.1. Definitions

  • The appendix introduces the work’s naming conventions and includes an example in which ambiguous wording and distant objects led participants to common errors.Source: §C.1 · p. 18, Figure 18 · p. 18

C.2. Visual Question Answering

  • Baseline VQA results are reported for the withheld overall test set and separately for simulated and real observations.The appendix also includes a question example noting that limited object visibility led to common errors.Source: §C.2 · pp. 18–19, Figure 19 · p. 19
  • Table 6 describes comparisons of annotation styles, text sizes, and background colors with Qwen2 and the base images, labels, and grounding questions fixed.Source: Table 6 · p. 19

C.2.1. Response Parsing

  • A regex parser searches generated tokens and multiple-choice keywords, selecting the last matching option; closed-loop parse failures trigger a randomized action.Source: §C.2.1 · p. 19

C.2.2. Benchmarks on Test Set

The appendix points readers to withheld-test benchmarks for spatial reasoning, embodied understanding, and grounding performance.

C.2.2. Benchmarks on Test Set

  • Table 8 reports baseline VLM performance in spatial reasoning on the withheld test set.Source: §C.2.2 · p. 19
  • Table 11 reports baseline VLM performance in embodied understanding on the withheld test set.Source: §C.2.2 · p. 19
  • Table 12 reports baseline VLM grounding performance categorized by test-set composition.Source: §C.2.2 · p. 19

C.2.3. Benchmarks on Real Test Split

The subsection reports baseline VLM performance in spatial reasoning and embodied understanding on the withheld test set’s real split.

C.2.3. Benchmarks on Real Test Split

  • Baseline VLM performance is reported for spatial reasoning and embodied understanding on the withheld test set’s real split.The referenced tables are Tables 9 and 13, respectively.Source: §C.2.3 · p. 19

C.2.4. Benchmarks on Simulated Test Split

The subsection reports baseline VLM performance in spatial reasoning and embodied understanding on the withheld test set’s simulated split.

C.2.4. Benchmarks on Simulated Test Split

  • Baseline VLM performance is reported for spatial reasoning and embodied understanding on the withheld test set’s sim split.The referenced tables are Tables 10 and 14, respectively.Source: §C.2.4 · p. 19

C.3.1. Task Formulation

The task formulation evaluates VLMs in closed-loop driving simulations and uses navigation commands and a calibrated action space to control the ego vehicle.

C.3.1. Task Formulation

  • VLMs receive a Set-of-Mark front-camera observation and driving prompt every five simulation steps, then select an action repeated until the next inference.The MetaDrive simulation terminates at the time horizon or if the ego vehicle leaves drivable regions; invalid parsed responses are assigned KEEP STRAIGHT.Source: §C.3.1 · p. 19
  • Navigation commands are recomputed for the ego vehicle’s current position and specify destination distance and position using discrete spatial vocabulary.The command takes the form “your final destination is at <distance> to <position> at this moment.”Source: §C.3.1 · pp. 19–20
  • The supplied tables report per-question VQA accuracies for overall, real, and simulated splits across spatial, embodied, and grounding benchmarks.The captions identify the corresponding evaluation scopes but provide no accuracy values in these passages.Source: Table 8 · p. 20, Table 9 · p. 20, Table 10 · p. 20
  • The discrete driving actions map to normalized steering and acceleration-or-braking controls, with identical vehicle configurations used across trials.The action space was selected by greedily reconstructing real-world trajectories at the evaluation’s inference frequency and choosing the set with the best reconstruction quality.Source: §C.3.1 · pp. 20–21
  • The evaluation uses 120 scenarios, including 60 nuScenes scenarios and 60 safety-critical scenarios with an adversarial agent attempting to hit the ego vehicle.The adversarial agents are ensured to be observable.Source: §C.3.1 · p. 21

C.3.2. Metrics

The section defines driving outcomes and describes question families that test spatial reasoning, predicted interactions, embodied motion, and visual grounding.

C.3.2. Metrics

  • Route completion measures traveled distance as a ratio of complete route length, while collision and off-road rates count scenarios with a collision or departure from drivable regions.Source: §C.3.2 · p. 21
  • ADE averages per-step L2 trajectory error, appending the final ego position after premature off-road termination, while FDE measures final-position distance from the destination.Source: §C.3.2 · pp. 21–22
  • Embodied questions ask whether an action at a stated speed and duration causes collision, how far the ego moves, or whether it moves left or right.Source: §C.3.2 · pp. 23–24
  • Spatial question families probe object distance, position, heading, type, color, pairwise relations, relative distance, and heading alignment.Color questions are generated only from simulated observations because nuScenes lacks color annotations.Source: §C.3.2 · pp. 25–28
  • Other spatial questions ask models to predict collisions, compare object proximity, order objects, identify extrema, or describe all labeled objects.The describe-scenario question is train-only, intended to boost learning performance and avoid VLM collapse.Source: §C.3.2 · pp. 29–32
  • Grounding questions test whether the VLM can identify the numerical label associated with a highlighted area, using valid labels as distractors.Source: §C.3.2 · p. 32

Related papers

Loading 2501.09167v1…