PaperlayerView exact revision on arXiv

Source-linked AI summary

NaviTrace: Evaluating Embodied Navigation of Vision-Language Models

Tim Windecker, Manthan Patel, Moritz Reuss, Richard Schwarzkopf, Cesar Cadena, Rudolf Lioutikov, Marco Hutter, Jonas Frey

arXiv:2510.26909v3cs.RO

TL;DR

Navigation evaluation remains limited by costly real-world trials, simplified simulations, and benchmarks that do not cover trace-level plans across embodiments. NaviTrace introduces a real-world VQA benchmark and semantic-aware score for embodiment-specific image-space navigation traces. Across embodiments and task categories, VLMs remain substantially below human and oracle-like baselines, with goal localization a dominant failure mode.

  • Problem

    Navigation evaluation is constrained by costly real-world trials, simplified simulations, and VQA datasets that typically limit outputs to text and cover only legged or wheeled robots.

    Source: §I · pp. 1–2

  • Method

    NaviTrace evaluates 2D image-space navigation traces across four embodiments in real-world scenarios, using a score that combines path distance, goal endpoint error, and semantic embodiment-conditioned penalties.

    Source: §I · p. 2

  • Results

    Across four embodiments and task categories, VLM scores are substantially worse than human and oracle-like baselines, and goal localization is the dominant failure mode.

    Source: D. Summary of Key Findings · p. 7

  • Takeaways & Limitations

    NaviTrace provides a systematic real-world framework that extends pointing tasks to sequential navigation prediction, bridging high-level VLM reasoning and low-level robotic control.

    Source: §V · p. 7

    NaviTrace is geographically concentrated in Switzerland, limited to single-image scenarios and ground vehicles, and its segmentation-based score may introduce systematic evaluation bias.

    Source: §VI · p. 7

Abstract

from arXiv · show

Vision-language models demonstrate unprecedented performance and generalization across a wide range of tasks and scenarios. Integrating these foundation models into robotic navigation systems opens pathways toward building general-purpose robots. Yet, evaluating these models' navigation capabilities remains constrained by costly real-world trials, overly simplified simulations, and limited benchmarks. We introduce NaviTrace, a high-quality Visual Question Answering benchmark where a model receives an instruction and embodiment type (human, legged robot, wheeled robot, bicycle) and must output a 2D navigation trace in image space. Across 1000 scenarios and more than 3000 expert traces, we systematically evaluate eight state-of-the-art VLMs using a newly introduced semantic-aware trace score. This metric combines Dynamic Time Warping distance, goal endpoint error, and embodiment-conditioned penalties derived from per-pixel semantics and correlates with human preferences. Our evaluation reveals consistent gap to human performance caused by poor spatial grounding and goal localization. NaviTrace establishes a scalable and reproducible benchmark for real-world robotic navigation. The benchmark and leaderboard can be found at https://leggedrobotics.github.io/navitrace_webpage/.

I. INTRODUCTION

The introduction frames rigorous evaluation of VLM navigation as important but underserved, then presents NaviTrace as a real-world, multi-embodiment trace benchmark with a human-aligned score.

I. INTRODUCTION

  • Existing VQA navigation datasets typically require text answers and evaluate only legged or wheeled robots, leaving trace-level, cross-embodiment evaluation unaddressed, [10].Real-world rollouts are costly, time-consuming, and hard to reproduce, while simulations constrain scenario diversity and physical realism.Source: §I · pp. 1–2
  • NaviTrace evaluates embodiment-specific navigation in 1,000 real-world scenarios by asking for 2D image-space paths across four embodiment types.The benchmark tests instruction following, spatial understanding, and physical understanding across humans, legged robots, wheeled robots, and bicycles.Source: §I · p. 2
  • Its semantic-aware score combines ground-truth Dynamic Time Warping distance, goal endpoint error, and pixelwise embodiment-conditioned penalties, and aligns with human preferences.The score is presented as inexpensive to compute and annotate while competitive with more expensive human-derived metrics.Source: §I · p. 2

II. RELATED WORK

  • Prior VLN benchmarks use specialized action spaces or simulated indoor environments, while real-world trajectory datasets may lack language-conditioned tasks,, [4],,.NaviTrace instead uses manually collected real-world images and VLM-accessible 2D trace prediction across embodiments.Source: §II · pp. 2–3
  • Existing social-navigation VQA datasets ask about interactions, perceptions, or future movements, but do not directly evaluate navigation tasks or embodiment differences, [10].The introduction also notes that it remains unclear which VLM system performs best for navigation –[15].Source: §I · p. 1, §II · p. 3

III. NAVITRACE BENCHMARK

NaviTrace pairs real-world images and instructions with embodiment-specific trace annotations, organized to capture diverse navigation challenges and support systematic evaluation.

III. NAVITRACE BENCHMARK

  • The benchmark contains 1,000 scenarios with more than 3,000 traces, split evenly into validation and test sets, with test annotations kept secret for leaderboard evaluation.Models receive an image, task description, and embodiment type, then predict a path scored with a task-specific function.Source: §III · p. 3

A. Data Collection

  • Each scenario pairs a distinct first-person real-world image with a manually written instruction solvable from visual information, framed as either a goal or directional route.Most images are crowd-sourced from consumer devices, with 164 curated samples from GrandTour.Source: A. Data Collection · p. 3
  • Scenarios are tagged by navigation challenge, including terrain geometry and semantics, accessibility, visibility, and social norms.Examples include stairs or closed doors, sidewalks or friction, wheelchair ramps, occlusion or poor lighting, and crosswalk rules.Source: A. Data Collection · p. 3
  • The dataset includes dynamic and stationary obstacle avoidance as navigation challenges, such as moving people or vehicles and fixed debris or road closures.The trace annotations represent routes around these obstacles.Source: A. Data Collection · p. 3
  • A trace is a sequence of 2D image-coordinate points describing a navigation path, with multiple annotations when equally valid and fast alternatives exist.The representation is independent of robot-specific controls.Source: A. Data Collection · p. 3
  • The four modeled embodiments are humans, legged robots, wheeled delivery robots, and bicycles, each associated with distinct mobility or route preferences.Cars are excluded because their viewpoint differs fundamentally from the other modeled embodiments.Source: A. Data Collection · p. 3

B. Data Quality

The dataset balances urban and rural scenarios but favors daylight and clear or cloudy imagery. Score evaluation finds gains from DTW, endpoint error, and semantic penalties, with limited additional gains from manual semantic labeling.

B. Data Quality

  • Images are concentrated in Switzerland but include other countries; scenarios balance urban and rural settings while favoring structured, daylight, clear or cloudy conditions.The benchmark primarily targets navigation challenges rather than difficult visual-perception conditions.Source: B. Data Quality · pp. 3–4, Figure 2 · p. 4

C. Score

  • The score combines DTW trace similarity, endpoint error, and embodiment-conditioned semantic costs along predicted paths, then scales scores using Straight Forward as a reference.Semantic costs penalize traces crossing undesired regions; Straight Forward is the upper-bound reference for scaling.Source: C. Score · p. 4
  • DTW outperforms RMSE and discrete Fréchet distance across trace-similarity metrics; adding FDE and Mask2Former semantic penalties each improves performance, while manual labels add limited gains.Table II compares score variants with human rankings.Source: C. Score · p. 5, Table II · p. 5

IV. EXPERIMENTS

The experiments assess VLM trace prediction, variation by embodiment or task category, and the aspects of navigation that are most challenging. They establish baselines before evaluating state-of-the-art VLMs on the test split.

IV. EXPERIMENTS

  • The experiments ask how well current VLMs predict navigation traces.This is the first of three questions guiding the experiments.Source: §IV · p. 5
  • The experiments examine whether performance varies by embodiment or task category.This is the second stated experimental question.Source: §IV · p. 5
  • The experiments investigate which aspects of navigation tasks pose the greatest challenges.They establish five baselines, deploy state-of-the-art VLMs, and analyze test-split results.Source: §IV · p. 5

A. Baselines

The section defines baseline comparisons and evaluates eight VLMs. Human experts outperform all models; the results also point to challenges in goal localization, path shaping, and spatial grounding.

A. Baselines

  • The baselines include human experts, a centerline Straight Forward strategy, goal-informed straight lines, and Gemini 2.5 Pro goal-only and path-only predictions.Human performance provides an upper bound; the goal-only and path-only variants isolate aspects of trace prediction.Source: A. Baselines · p. 5

B. Models

  • The evaluation includes five proprietary and three open-weight VLMs, with prompts specifying the task, output format, expected behavior, and embodiment.Models return traces as normalized 2D points in JSON.Source: B. Models · p. 6

C. Performance

  • Human experts outperform all VLMs, while Gemini 2.5 Pro ranks first among models and Straight Forward ranks close behind o3.Figure 4 compares rankings by embodiment; higher scores are better.Source: C. Performance · p. 6, Figure 4 · p. 6
  • Performance varies only slightly across task categories, but this uniformity does not establish balanced competence.The overall weakness of models may mask category- and embodiment-specific differences; Straight Forward’s competitiveness highlights the deficit.Source: C. Performance · p. 6, Figure 4 · p. 6
  • Predicting only the goal and connecting it by a straight line is only slightly worse than Gemini 2.5 Pro’s full-trace prediction.Baselines with explicit goal access perform significantly better, indicating that locating the goal area is a major challenge.Source: C. Performance · p. 6, Table III · p. 6
  • With the goal provided, Gemini 2.5 Pro’s path-only prediction beats Oracle-Goal Straight Line but remains below human performance.The comparison indicates difficulty in both goal recognition and meaningful path shaping.Source: C. Performance · p. 6, Table III · p. 6
  • o3’s textual reasoning can identify the correct path option even when its predicted trace fails to align with that reasoning.The authors describe this as a common qualitative pattern and suggest a gap in spatial grounding, particularly localization of traversable structures.Source: C. Performance · p. 6

D. Summary of Key Findings

VLMs substantially trail human and oracle-like baselines across embodiments, with goal localization the dominant failure mode. NaviTrace provides a real-world evaluation framework, but its geographic, scenario, embodiment, and scoring scope remains bounded.

D. Summary of Key Findings

  • An o3 example shows correct textual reasoning about path options alongside a predicted trace that fails to match that reasoning.Source: Figure 6 · p. 7
  • Across embodiments, VLMs trail human and oracle-like baselines; goal localization dominates failure, while path shaping still lags even with correct goals.Aggregate performance differences across human, legged robot, wheeled robot, and bicycle embodiments are small.Source: D. Summary of Key Findings · p. 7
  • The semantic-aware trace score correlates more strongly with human preference than DTW alone, with manual segmentation adding a modest gain.The score combines DTW distance, endpoint error, and embodiment-conditioned semantic penalties.Source: D. Summary of Key Findings · p. 7

V. CONCLUSION

  • NaviTrace presents a real-world, four-embodiment framework with 1,000 diverse images, extending pointing tasks to sequential navigation prediction.The benchmark is intended as a bridge between high-level VLM reasoning and low-level robotic control.Source: §V · p. 7

VI. LIMITATIONS

  • The dataset is geographically concentrated in Switzerland, limited to single-image scenarios and ground vehicles, and its automated segmentation may bias scores.Source: §VI · p. 7
  • The scoring function may miss nuanced human preferences, recognizes specific goal points rather than broader targets, and cannot account for precise robot dimensions.Source: §VI · pp. 7–8

A. Prompts

The prompts ask models to turn an image, navigation task, and embodiment into normalized image-space predictions. Separate prompt variants elicit a complete trace, a goal point, or a path between supplied endpoints.

A. Prompts

  • Three prompt variants distinguish full-trace prediction, goal-point prediction, and path prediction between a supplied start and goal.The goal-point prompt returns one normalized point; the path prompt uses provided endpoints.Source: A. Prompts · pp. 9–10
  • The prompts encode embodiment-specific constraints, including stair access, surface preferences, and bicycle traffic behavior.Source: A. Prompts · p. 9
  • The model receives an image, embodiment, and task, then predicts a feasible path as normalized 2D image points.The prompts specify embodiment movement characteristics and request a structured point-list output.Source: A. Prompts · p. 9
  • The full-trace prompt starts near the image bottom center and asks the path to reach the visible or reasonably inferred goal, stopping where an occluded route cannot be inferred.Source: A. Prompts · p. 9

C. Model Configuration

The model evaluation uses API-based access with common token and temperature settings while otherwise retaining provider defaults.

C. Model Configuration

B. Semantic Penalty

The semantic penalty component assigns embodiment-specific costs to semantic classes along predicted traces.

B. Semantic Penalty

Loading 2510.26909v3…