Source-linked AI summary
EgoPathBench: Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language Models
Yang Zhao, Zhuo Chen, Xubo Yang
TL;DR
EgoPathBench addresses the lack of benchmarks for integrated first-person navigation decisions that combine target recognition, action consequences, embodiment, and route planning. It introduces a five-task waypoint benchmark with geometry-grounded evaluation and training supervision. Across nine VLMs, the best EgoPath Score is 28.3, while fine-tuning Qwen 3.5 4B raises its score from 3.9 to 38.9.
Problem
Existing spatial-intelligence benchmarks mainly test isolated relations, directions, or targets rather than integrated, goal-reaching waypoint navigation under agent geometry.
Method
EgoPathBench presents first-person RGB images, natural-language goals, and numbered waypoints, then evaluates candidate feasibility, route legality, and goal arrival across five tasks.
Results
28.3 is the highest EgoPath Score across nine VLMs, while fine-tuning Qwen 3.5 4B raises its score from 3.9 to 38.9 and improves all four reported external evaluations.
Takeaways & Limitations
The benchmark exposes a gap between locally plausible actions and complete embodied routes, while its released data provides effective supervision for this ability.
Abstract
from arXiv · showhide
Zero-shot waypoint navigation requires vision-language models to select, from the current first-person observation, a sequence of spatial actions that is feasible for the agent and reaches the goal, placing joint demands on the integrated spatial intelligence of today's foundation VLMs. Existing spatial-intelligence benchmarks primarily evaluate isolated judgments of relations, directions, or targets and therefore do not directly measure the integrated navigation ability required to combine target recognition, action-consequence assessment, distance estimation, and path planning. To fill this evaluation gap, we introduce EgoPathBench, a dataset and five-task benchmark for first-person waypoint decision-making. Each question presents an egocentric RGB image, a natural-language goal, and numbered visible waypoints; a model returns traversable candidates or an ordered route. Predictions are evaluated for candidate feasibility, adjacent-edge legality, and goal arrival under point-agent or embodied geometry. EgoPathBench contains 31,852 training, 1,345 validation, and 1,111 benchmark questions and retains at least one geometrically verified reference route for every route question. Across nine VLMs, the highest EgoPath Score is only 28.3. The top-ranked model reaches 35.9% success on Point Path, but only 2.9% and 4.0% on Embodied Path and Intent Path, respectively, showing that current models remain limited in forming complete, goal-consistent routes under embodiment constraints. Beyond the evaluation data, we release the corresponding training resource. Fine-tuning Qwen 3.5 4B on the released training split raises its EgoPath Score from 3.9 to 38.9 and improves all four reported evaluations across three external spatial benchmarks, with gains of 1.4--9.6 points.
Introduction
EgoPathBench addresses the gap between component spatial judgments and integrated, embodiment-aware waypoint navigation. It introduces a five-task benchmark and releases geometry-grounded supervision, while results show substantial difficulty in complete route formation.
- Motivation: EgoPathBench evaluates integrated first-person waypoint decisions rather than isolated recognition or spatial-relation judgments, requiring target grounding, action consequences, route planning, and goal consistency.Models return traversable candidates or ordered routes, which are evaluated as action-valued outputs.
- Benchmark scope: The benchmark organizes five waypoint-decision tasks around candidate feasibility, route legality, goal arrival, embodiment, target grounding, and multi-step planning.Point and embodied route decisions share the scene and candidate vocabulary while testing different feasibility and goal requirements.
- Results: 28.3 is the highest EgoPath Score across nine VLMs, while the leader reaches 35.9% Point Path success but only 2.9% Embodied Path and 4.0% Intent Path success.Performance declines when routes must satisfy embodiment and goal conditions jointly.
- Results: Fine-tuning Qwen 3.5 4B on the released training split raises EgoPath Score from 3.9 to 38.9 and improves all four reported evaluations across three external spatial benchmarks by 1.4–9.6 points.The results support the released data as supervision for first-person spatial decision-making.
- Contributions: 31,852 training, 1,345 validation, and 1,111 benchmark questions are released with aligned observations, waypoints, feasibility graphs, legal routes, and Spatial CoT supervision.The resource supports both unified model comparison and training experiments.
Related Work
Prior benchmarks measure complementary spatial, embodied, video, or 3D reasoning components, whereas EgoPathBench evaluates their joint use in scene-grounded waypoint actions. Its geometry-backed scoring differs from trajectory agreement with human demonstrations.
- Closest comparison: NaviTrace is the closest evaluation, using a real RGB image, navigation instruction, embodiment description, and continuous image-space trace scored against human demonstrations.Its score combines Dynamic Time Warping, endpoint error, and embodiment-conditioned semantic penalties.
- EgoPathBench distinction: EgoPathBench registers each observation to a 3D scene and evaluates predicted waypoint actions through geometric feasibility, consecutive-edge legality, and goal attainment.This scene backing permits arbitrary predicted routes to be scored by their action outcomes.
- Existing evaluations: Existing benchmarks assess distance, direction, relations, video spatial memory, viewpoint localization, 3D structure, or embodied observations as complementary component abilities.These evaluations do not constitute the same ordered waypoint-action decision.
- EgoPathBench distinction: EgoPathBench connects target, candidate locations, agent constraints, and route structure in one explicit action interpretation with either traversable sets or ordered routes.The resulting representation links low-level embodied cues to ordered first-person decisions.
EgoPathBench Dataset and Benchmark
EgoPathBench represents navigation as selections over visible numbered waypoints and evaluates them against scene-grounded point or embodied geometry. Its construction fixes targets, action spaces, routes, and quality checks before generating questions and supervision.
- Benchmark formulation: Each example supplies an egocentric image, numbered waypoint vocabulary, target specification, and task-specific feasibility graph; outputs are unordered traversable sets or ordered routes.Route tasks additionally require the specified start, legal consecutive edges, and an acceptable endpoint.
- Geometry: Point and embodied feasibility graphs share observations and waypoints but differ because the embodied graph accounts for agent footprint, allowing identical visual actions to have different consequences.This paired design isolates the effect of embodiment within the same scene view.
- Data construction: Construction anchors visible waypoints, targets, goal regions, and reference routes in shared scene geometry before generating natural-language questions and Spatial CoT.Point and embodied free-space searches produce geometry-validated route waypoints projected into the current view.
- Task design: The five tasks progressively test target grounding, embodiment, and route composition through two traversability tasks and three route tasks.Point and Embodied Traversability differ by clearance, while Point Path, Embodied Path, and Intent Path add routing and target-resolution requirements.
- Supervision: Every retained route question has at least one scene-verified reference route, while training data additionally includes reasoning text verbalizing fixed targets, candidates, feasibility labels, legal edges, and routes.The reasoning text is generated downstream of formal geometric annotations and checked against them.
- Quality and benchmark selection: Quality control removes disconnected, colliding, target-missing, unavailable-waypoint, ambiguous, or unsupported examples and audits alignment among images, prompts, waypoints, routes, and answers.The formal benchmark then selects challenging bundles emphasizing clutter, feasibility boundaries, competing targets, and route composition.
Experiments
Experiments show that zero-shot VLMs perform substantially better on local waypoint feasibility than on complete, goal-reaching routes, especially under embodied constraints. Fine-tuning on EgoPathBench improves in-domain performance and transfers gains to external spatial benchmarks.
- Zero-Shot VLM Results: 28.3 was the highest zero-shot EgoPath Score, with Gemini 3.1 Pro ranking ahead of GPT-5.5 at 27.3 and Claude Opus 4.8 at 25.6.
- Zero-Shot VLM Results: 77.9% and 77.3% were the best Point and Embodied Traversability BA values, but Point Path success reached only 35.9% and SPL 28.7%.The results indicate that plausible local actions do not by themselves yield legal, goal-reaching routes.
- Zero-Shot VLM Results: Embodied Path achieved only 12.0% VPR, 2.9% SR, and 2.4% SPL, while Intent Path reached 14.4% VPR, 4.0% SR, and 3.3% SPL.The VPR–SR gaps indicate complementary failures in edge legality and endpoint selection, while low SPL reflects scarce complete successes.
- Zero-Shot VLM Results: Humans reached an EgoPath Score of 54.2 versus 28.6 for the strongest VLM on the same questions, with higher performance across all five task axes.The gap spans both candidate-feasibility judgments and all three route tasks.
- Where Complete Routes Fail: Route diagnostics show evaluable outputs at 96.3–96.8%, but goal-consistent endpoints at 28.9%, 4.8%, and 5.2% for Point, Embodied, and Intent Path.Legal first actions remain relatively common, whereas full-route legality falls to 46.8%, 4.8%, and 6.5%, and joint success to 16.7%, 1.0%, and 1.4%.
- Training-Resource Evaluation: Fine-tuning Qwen3.5-4B raised its EgoPath Score to 38.9 and improved all reported external evaluations, including gains of 9.6 points on SpatialEval-VTQA and 1.4 points on 3DSRBench.On VSI-Bench Route Planning, Full-setting performance increased from 29.38% to 33.51% and Debiased performance from 20.18% to 24.56%.
Conclusion
EgoPathBench evaluates whether VLMs can convert first-person observations and goals into feasible, goal-reaching waypoint actions across five scene-grounded tasks. Results expose a large gap between local action plausibility and complete embodied routes, while fine-tuning improves both benchmark and external spatial performance.
- Conclusion: EgoPathBench evaluates first-person waypoint decision-making across feasibility, grounding, route legality, goal arrival, and efficiency.
- Conclusion: Across nine VLMs, the best EgoPath Score was 28.3, with Point Path success of 35.9% versus 2.9% on Embodied Path and 4.0% on Intent Path.
- Conclusion: Fine-tuning Qwen 3.5 4B increased EgoPath Score from 3.9 to 38.9 and improved all four reported evaluations across three external spatial benchmarks.The results identify sustained geometric and goal consistency across multi-step decisions as a central direction for future VLM research.
Evaluation Protocol and Reproducibility
The benchmark uses fixed JSON waypoint outputs evaluated by ordered geometric checks, with reproducible model and fine-tuning protocols plus aggregate-score sensitivity controls.
- Route tasks require ordered sequences that start at the specified waypoint, use legal consecutive edges, and end within an acceptable goal set.
- The evaluator separately checks candidate validity, route start, direct-edge legality, goal membership, and applies SPL only after route success.
- All model runs use identical task-specific prompts and JSON contracts, fixed benchmark questions, deterministic decoding where exposed, and 8,192-token completion budgets.
- Qwen3.5-4B fine-tuning uses LoRA with a frozen vision tower, staged cosine learning-rate schedules, bfloat16, and fixed image and token limits.
- Alternative aggregate definitions preserve the complete nine-model ordering, with Spearman ρ = 1.0 for both tested alternatives.
Analyses
Supporting analyses test dependence on the visual interface, reference-route validity, visibility and geometry assumptions, evaluator robustness, and human–model differences.
- The analyses separately probe image-overlay dependence, reference-route admission, visibility, scene consequences, and geometry sensitivity.
- The scene-consequence audit recovers registered obstruction evidence for 4,934 of 5,563 predictions containing geometry-defined illegal edges, or 88.7%.
- The same-question human reference scores 54.2 versus 28.6 for the strongest VLM, with higher route valid-path and success rates for humans.
Complete Dataset and Route Statistics
The released route questions are structurally nontrivial: models can fail after initially correct decisions, and longer references worsen—but do not fully explain—illegal transitions.
- Route questions contain a median of four same-family objects, while Embodied and Intent references require a median of two edges.
- Embodied routes spend a median 40% of their length in the reported constrained segment, indicating route structure beyond direct, unambiguous cases.
- 42.2% of Point, 75.8% of Embodied, and 69.4% of Intent routes fail on a later edge despite a legal first edge and acceptable endpoint.
- Illegal-edge incidence rises with reference length, reaching 94.0% for Embodied and 92.4% for Intent references with at least three edges.
Same-Question Human and Model Outputs
The supplementary figures visualize same-question human and VLM waypoint selections and routes over identical first-person inputs and candidate interfaces.
- Each panel repeats the identical first-person image and visual candidate interface, then overlays recorded human or VLM waypoint responses.
- Figures S1–S4 mark selected feasible points with green rings, incorrect selections with red crosses, and omitted feasible points with dashed orange rings.
- Figures S5–S10 color complete routes green when all consecutive edges are legal and red otherwise, while endpoint color distinguishes acceptable from wrong goals.