PaperlayerView exact revision on arXiv

Source-linked AI summary

UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces

Baining Zhao, Jianjie Fang, Zichao Dai, Ziyou Wang, Jirong Zha, Weichen Zhang, Chen Gao, Yue Wang, Jinqiang Cui, Xinlei Chen, Yong Li

arXiv:2503.06157v1cs.CVcs.AI

TL;DR

Embodied cognitive abilities in urban open 3D spaces remain insufficiently assessed. UrbanVideo-Bench evaluates Video-LLMs with urban embodied videos and multiple-choice tasks. The best current models achieve 45.5% accuracy, and the paper reports correlations across abilities and gains from simulation-data fine-tuning.

  • Problem

    Video-LLMs have been assessed mostly on disembodied third-person clips, while embodied abilities in urban open 3D spaces remain insufficiently defined and assessed.

    Source: §1 · p. 1

  • Method

    UrbanVideo-Bench combines manually collected real and simulated urban motion videos with 16 tasks across recall, perception, reasoning, and navigation.

    Source: §1 · pp. 2–3

  • Results

    The best current Video-LLMs reach 45.5% accuracy; causal reasoning is highly correlated with recall, perception, and planning, and simulation-data fine-tuning enhances real-world task performance.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The findings indicate substantial remaining limitations in current Video-LLMs' urban embodied cognition.

    Source: §5 · p. 8

    Generated questions can be invalid, ambiguous, or incorrect, requiring human refinement.

    Source: §3.2.4 · p. 5

Abstract

from arXiv · show

Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban 3D space remain to be explored. We introduce a benchmark to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. We have manually control drones to collect 3D embodied motion video data from real-world cities and simulated environments, resulting in 1.5k video clips. Then we design a pipeline to generate 5.2k multiple-choice questions. Evaluations of 17 widely-used Video-LLMs reveal current limitations in urban embodied cognition. Correlation analysis provides insight into the relationships between different tasks, showing that causal reasoning has a strong correlation with recall, perception, and navigation, while the abilities for counterfactual and associative reasoning exhibit lower correlation with other tasks. We also validate the potential for Sim-to-Real transfer in urban embodiment through fine-tuning.

1 Introduction

UrbanVideo-Bench addresses the limited assessment of embodied cognition in urban open 3D spaces with a benchmark spanning multiple task types and real and simulated videos. It evaluates Video-LLMs on recall, perception, reasoning, and navigation.

Motivation

  • Existing Video-LLM benchmarks mostly use disembodied third-person clips, while embodied video work focuses on manipulation or indoor and ground-level movement, leaving urban open 3D cognition underassessed.The paper also notes that drone data collection is difficult and costly because of signal loss, limited range, and obstruction-related crashes.Source: §1 · pp. 1–2

Benchmark

  • UrbanVideo-Bench covers four ability categories and 16 tasks with 1.5k video clips and 5.2k multiple-choice questions.The benchmark targets embodied motion cognition in urban airspace.Source: §1 · pp. 2–3

Data and evaluation

  • The videos draw on real cities in Guangdong, the Beijing-based EmbodiedCity simulator, and the virtual-city AerialVLN simulator.The authors say combining real devices and simulators helps increase the number of videos rapidly.Source: §1 · p. 2
  • The question-generation pipeline combines trained human annotators, over 800 hours of effort, and LMM expertise to produce multiple-choice questions.The authors evaluate proprietary and open-source Video-LLMs in zero-shot settings and attempt supervised fine-tuning on two models.Source: §1 · p. 2
  • The authors evaluate 17 popular LMMs and explore correlations among embodied cognitive abilities and the potential for Sim-to-Real transfer.The task set assesses recall, perception, reasoning, and navigation.Source: §1 · p. 3

2 Related Work

Prior video and embodied-understanding benchmarks largely focus on video content or non-urban settings. The paper positions UrbanVideo-Bench around embodied cognition in urban 3D aerial motion.

Embodied Capabilities of LMMs

  • Earlier LMM spatial-understanding work primarily used 2D images, static point clouds, or language, whereas human world comprehension is grounded in continuous visual perception.The authors argue for video benchmarks from diverse sources to evaluate embodied scenarios.Source: §2 · p. 3

Video Benchmarks for LMMs

  • Traditional video benchmarks cover content understanding but lack an embodied, egocentric perspective, while embodied benchmarks have focused on indoor or ground-level scenes.The paper identifies insufficient exploration of embodied abilities in urban open 3D spaces.Source: §2 · p. 3
  • UrbanVideo-Bench differs from popular video benchmarks in its sources and scenarios, focusing on Video-LLM embodied cognition in urban 3D aerial motion.The paper compares its benchmark with popular video-LLM benchmarks in Table 2.Source: Table 2 · p. 4

3 Benchmark Design and Construction

UrbanVideo-Bench defines 16 tasks across recall, perception, reasoning, and navigation, then constructs a dataset of urban embodied-motion videos and multiple-choice questions. Its generation pipeline combines video curation, question generation, blind filtering, and human refinement.

3.1 Task Set

  • The benchmark evaluates four abilities—recall, perception, reasoning, and navigation—through 16 tasks designed for urban open-ended scenarios.The tasks cover remembering urban environments, perceiving spatial and temporal relationships, reasoning about actions and surroundings, and planning or generating navigation actions.Source: §3.1 · p. 4
  • The benchmark’s four abilities include memory-oriented recall, static and dynamic spatial perception, action-and-surroundings reasoning, and route- or goal-oriented navigation.Navigation tasks assess route progress, control actions, high-level planning, and mapping plans to control actions.Source: §3.1 · p. 4

3.2 Dataset Generation Pipeline

  • The dataset pipeline has four stages: video curation, multiple-choice question generation, blind filtering, and human refinement.Figure 2 also summarizes frame counts, path lengths, and question word counts.Source: §3.2 · p. 4, Figure 2 · p. 5

3.2.1 Video Curation

  • Video collection combines drone footage from Shenzhen and Zhaoqing with aerial-agent videos from EmbodiedCity and AerialVLN.Experienced pilots collected first-person footage while following designed navigation goals or routes.Source: §3.2.1 · p. 5

3.2.2 MCQ Generation

  • MCQs are generated with a chain-of-thought prompt that guides narration, key-information extraction, role-playing, and question-option templates.Gemini-1.5 Flash assists with generation.Source: §3.2.2 · p. 5

3.2.3 Blind Filtering

  • Blind filtering removes questions that multiple Video-LLMs answer correctly without video, retaining items that require video input.The procedure removes an item if all n models guess its answer correctly without video.Source: §3.2.3 · p. 5

3.2.4 Human Refinement

  • Human refinement addresses invalid or ambiguous questions, urban-element hallucinations, inaccurate directions, and flawed answer choices.The refinement process required over 800 person-hours.Source: §3.2.4 · pp. 5–6

3.3 Dataset Statistics

  • The dataset contains 1,547 clips and over 5.2K questions, spanning brief movements to long-distance navigation across diverse 3D trajectories.Clips range from 10 seconds to 10 minutes; low-level and high-level questions each account for approximately 50% of the total.Source: §3.3 · p. 6

4 Experiments

Experiments compare Video-LLMs on embodied urban tasks, examine relationships among abilities, test simulated-to-real fine-tuning, and analyze common errors. Performance remains limited overall, while task correlations and error patterns vary.

4.1 Experimental Setup

  • The evaluation calculates accuracy for each task and overall using multiple-choice questions, and compares proprietary and open-source Video-LLMs.The comparison includes models with adjustable frame rates or fixed frame-count inputs.Source: §4.1 · p. 6

4.2 Model Comparison

  • Some open-source Video-LLMs outperform some proprietary models, and video-optimized models outperform image-focused LMMs.Source: §4.2 · p. 6
  • 45.5% average accuracy is achieved by the best-performing model, Qwen-VL-Max, while proprietary and open-source models both show relatively poor urban embodied cognition.Table 3 reports accuracy for existing Video-LLMs and fine-tuned models.Source: §4.2 · p. 6, Table 3 · p. 7
  • Smaller-parameter models appear less stable: at equivalent average accuracy, an open-source small model tends to have a lower minimum task accuracy than a commercial large model.Source: §4.2 · p. 7

4.3 Correlation of Cognitive Abilities

  • Causal reasoning correlates highly with almost all other tasks, while recall tasks are strongly intercorrelated and navigation correlates highly with recall and perception.The analysis treats similar model performance across two tasks as evidence that the tasks may require similar embodied cognitive abilities.Source: §4.3 · pp. 7–8
  • Counterfactual and association reasoning correlate weakly with other task types, which the authors interpret as potentially distinct cognitive processes.The authors suggest targeted training for tasks involving these two abilities.Source: §4.3 · p. 8

4.4 Sim-to-Real

  • Fine-tuning on EmbodiedCity and AerialVLN data improves real-world test performance on Goal Detection and Association Reasoning by approximately 7%.The mean improvements for the two fine-tuned models were 3.2% and 5.2%, respectively.Source: Table 3 · p. 7, §4.4 · p. 8

4.5 Error Analysis

  • Error analysis identifies urban-scene understanding errors, motion-orientation errors, and failures in egocentric reasoning such as route planning and extrapolation.Recall and perception are affected by scene and motion understanding errors, while reasoning and navigation exhibit various errors.Source: §4.5 · p. 8, Figure 5 · p. 8

5 Conclusion

UrbanVideo-Bench evaluates urban embodied cognition across 1.5k video clips and 5.2k questions; the best evaluated Video-LLMs achieve 45.5% accuracy, and the analysis reports task correlations and simulation-to-real fine-tuning gains.

5 Conclusion

  • 45.5% accuracy is the highest achieved by the 17 evaluated Video-LLMs on UrbanVideo-Bench.The benchmark contains 1.5k video clips and 5.2k multiple-choice questions.Source: §5 · p. 8
  • Causal reasoning is highly correlated with recall, perception, and planning, while counterfactual and associative reasoning have lower correlations with other tasks.Source: §5 · p. 8
  • Fine-tuning large models with simulation data can enhance performance on real-world embodied video tasks.Source: §5 · p. 8

A Dataset Examples

The appendix presents video and multiple-choice-question examples from real-world footage and two simulators, covering all task types.

A Dataset Examples

  • Figures 6–8 show videos and MCQ examples from the real world, EmbodiedCity, and AerialVLN, encompassing all task types.Source: A Dataset Examples · p. 12

B Dataset Generation Pipeline

The dataset pipeline combines drone and simulator video curation, staged LLM question generation, and human refinement of generated questions.

B.1 Details of Video Curation

  • EmbodiedCity and AerialVLN provide urban environments, aerial-agent support, and route references, while experienced pilots and a gimbal-equipped camera support purposeful real-drone data collection.The camera tilts up to 90 degrees downward; pilots have over 1,000 hours of real drone flight time.Source: §B.1 · p. 12

B.2 Details of MCQ Generation

  • Question generation proceeds from trajectory narration to structured movement and object extraction, then role-based prompts and task-specific question and option templates.Source: §B.2 · p. 12
  • Because LLMs struggle with complicated city environments, pregenerated object lists and movement series assist question generation, which uses task-specific instructions, templates, and examples.Source: §B.2 · p. 12

B.3 Details of Human Refinement

  • Human refinement addresses unclear questions, hallucinated urban elements, inconsistent directions, and ambiguous options to improve factual accuracy and question reliability.Annotators standardize directions relative to the drone and ensure one unambiguously correct answer.Source: §B.3 · p. 13

C.1 Evaluation Metric Calculation

Evaluation combines automated answer processing and alignment with ground truth, followed by accuracy calculation and manual checks for unresolved outputs.

C.1 Evaluation Metric Calculation

  • The evaluation workflow combines regular expressions and model inference with manual validation to assess video-based multiple-choice answers.Source: §C.1 · pp. 13–14
  • Answers are extracted from model outputs; GPT-4o infers options for ambiguous text, and unresolved entries are flagged for manual verification.Source: §C.1 · p. 14
  • Corrected answers are standardized and matched to ground truth by video ID and question category before invalid entries are filtered out.Source: §C.1 · p. 14
  • Accuracy is computed by category and aggregated into an overall score, with results exported in structured tables.Source: §C.1 · p. 14

C.2 Brief Introduction on Baselines

The appendix documents the evaluated model baselines and question and evaluation prompts, then reports variation in proprietary-model format adherence and higher input than output evaluation costs.

C.2 Brief Introduction on Baselines

  • The passages describe Gemini-1.5-Flash, Gemini-1.5-Pro, Gemini-2.0-Flash, and GPT-4o; the three Gemini configurations use 1 fps, while GPT-4o uses 32 f.Source: §C.2 · p. 15
  • Question generation uses a teacher role, video-specific multiple-choice requirements, and 14 categories covering video, spatial, temporal, and causal understanding.Most categories use five choices; Proximity, Duration, Counterfactual, and Viewpoint Invariance questions use fewer choices.Source: §C.2 · p. 16

InternVL2 series (2B, 4B, 8B, 26B, 40B,

  • InternVL2 models span 2B to 76B parameters, with context lengths from 32K to 256K; the described experiment uses 448 × 448 resolution and 32 f.The supplied passage identifies the model series and its evaluation settings.Source: InternVL2 series (2B, 4B, 8B, 26B, 40B · p. 17

C.3 Responses of Video-LLMs

  • Proprietary models show varying levels of adherence to the required output format, as assessed by format correctness rates.Table 6 presents rates for six proprietary models; no rate values are supplied here.Source: §C.3 · pp. 17–18

C.4 Cost of Proprietary Models

  • Input cost is significantly greater than output cost in the proprietary-model evaluations, which the authors link to excessive token amounts for video input.Table 7 reports evaluation costs across models, but the supplied text gives no individual cost values.Source: §C.4 · p. 18

C.5 Prompt

  • Evaluation prompts pair each video with an MCQ and background and output-format instructions; some models receive all questions for a video together to avoid repeated uploads.For models such as GPT-4o and Qwen, batching questions with each video is described as reducing time consumption and token number.Source: §C.5 · pp. 18–19

D Fine-Tuning

The fine-tuning setup adapts InternVL2-4B and InternVL2-8B with LoRA while freezing the visual encoder and language-model backbone.

D Fine-Tuning

  • InternVL2-4B and InternVL2-8B are fine-tuned with LoRA while the visual encoder and language-model backbone are frozen.LoRA uses rank 128, alpha 256, and dropout 0.05.Source: D Fine-Tuning · p. 19
  • Training uses 70% of the multiple-choice samples, with associated video data totaling 39.56 GB; visual features are concatenated with text embeddings.The maximum input sequence length is 8192 to accommodate long video segments and complex text.Source: D Fine-Tuning · pp. 19–20
  • The setup trains for one epoch with AdamW at learning rate 1e−6, weight decay 0.05, and an effective batch size of 8.Linear warm-up and cosine annealing are used, with gradient clipping at maximum norm 1.Source: D Fine-Tuning · p. 20

E Error Cases

The error examples use question categories, prompts, ground-truth answers, and model responses to illustrate shortcomings in embodied abilities.

E Error Cases

  • The examples are organized by question category and include the question and options, a human-refined ground truth, and the Video-LLM answer with its reasoning.The described categories include Action Generation and High-level Planning.Source: E Error Cases · p. 20

Related papers

Loading 2503.06157v1…