Source-linked AI summary
KilometerVision: A New Frontier for Large-Scale Spatial Intelligence in VLMs
Aravindh Mahendran, Michael King, Matthew Koichi Grimes, Antoine Yang, Tyler Zhu, Joseph Heyward, Tengda Han, Shiry Ginosar, Chen Sun, Dima Damen, Simon Osindero, Noah Snavely, Simon Lynen, João Carreira, Viorica Pătrăucean
TL;DR
Whether VLMs can form geographic spatial representations from passive visual data remains an open question, with earlier work focused on smaller-scale environments or simulated settings. KilometerVision tests city-scale understanding in real-world videos through landmark, route, and map tasks; its findings indicate that current models rely on visual recognition and text-matching rather than geometric spatial reasoning.
Problem
It remains unclear whether VLMs trained mainly on passive internet-scale data can infer valid geographic spatial representations from real-world videos.
Source: §1 · p. 2Method
Results
Current VLMs predominantly rely on 2D visual recognition and semantic text-matching rather than true path integration or geometric mental maps.
Source: §1 · p. 3, §5.3 · p. 14Takeaways & Limitations
The findings place current VLMs at the landmark stage, with path integration and allocentric survey knowledge not yet mature even when GPS, maps, or camera poses are provided.
Source: §6 · p. 15
Abstract
from arXiv · showhide
We push the frontier of large-scale spatial intelligence in Vision-Language Models (VLMs) and introduce the first benchmark that probes geographical layout understanding from real-world videos, spanning up to 1km distances. Inspired by the cognitive science literature, we evaluate models against the hierarchical stages of human spatial awareness: anchoring via landmarks, connecting them through routes, and integrating these into global mental maps. Extensive experiments reveal a fundamental divergence in how current AI models process spatial information. Instead of utilising true path integration or forming geometric survey knowledge, we find that VLMs rely almost entirely on 2D visual recognition and text-matching to bypass complex spatial reasoning. The benchmark is publicly available at https://perception-test-challenge.github.io/kilometervision.html.
1 Introduction
The paper asks whether VLMs trained mainly on passive internet-scale data can infer geographic representations from real-world walking-tour videos. KilometerVision evaluates city-scale spatial understanding through landmarks, routes, and survey knowledge, and the experiments find that models largely substitute visual recognition and text matching for spatial reasoning.
1 Introduction
- Prior work studied VLM spatial representations in small-scale settings or navigation in simulated and factory-like environments, while this work probes city-scale walking tours.The question is whether VLMs can infer valid geographic representations from hour-long real-world videos.Source: §1 · p. 2
- KilometerVision evaluates landmarks, egocentric routes, and allocentric survey knowledge in multiple-choice video QA, following a cognitive-science paradigm.The tasks assess landmark identification and localization, routes connecting locations, and maps integrating multiple routes.Source: §1 · p. 2, Figure 1 · p. 2
- The benchmark contains 1,000 five-way video questions over 235 YouTube videos and evaluates five state-of-the-art VLMs.Questions target visual spatial capabilities rather than semantic knowledge about places.Source: §1 · p. 2
- Current VLMs predominantly bypass complex spatial reasoning through 2D visual recognition and semantic text-matching rather than path integration or geometric mental maps.The authors report this pattern across the evaluated models.Source: §1 · p. 3
2 Related work
Prior work spans video-language evaluation, spatial-capability benchmarks, and geographic reasoning with text, images, or interactive agents. KilometerVision complements these efforts with real-world walking-tour videos to probe landmark, route, and mental-map capabilities together.
2 Related work
- Video-language research includes image- or video-encoder approaches and dense-caption representations, evaluated on benchmarks such as NextQA and EgoSchema.Other benchmarks extend beyond action classification and localization to synthetic, egocentric, curated real-world, long-form, and broad multimodal settings.Source: §2 · p. 3
- Spatial-capability studies use language interfaces and cognitive-science-inspired tasks, while traditional spatial evaluation includes depth, pose, and 3D reconstruction.The cited work asks whether internet-scale training can yield a valid world model from passive observations.Source: §2 · p. 3
- Geographic-scale research includes text-only LLM tasks, image-based tasks, and interactive VLM-agent navigation, whereas KilometerVision uses real-world walking-tour videos.It is presented as the first benchmark to comprehensively probe landmark recognition, routes, and city mental maps from videos alone.Source: §2 · pp. 3–4
3 Video dataset
The dataset comprises high-resolution, hour-long walking tours selected for geographic diversity. The authors ground videos on maps with an automated positioning pipeline, enabling scalable route annotation and validation against human traces.
3.1 Video source
- The source set combines nine Walking Tours dataset videos with 226 additional YouTube tours, totaling 235 videos and about 288 hours across cities and countries.Selection prioritized high-resolution, hour-long videos with diverse city coverage.Source: §3.1 · p. 4
3.2 Grounding videos on the map
- Map grounding produces frame-level coordinates and camera poses, exposing route structure such as loop closures and supporting heuristic, minimally annotated task labels.The pipeline uses Google VPS with human-provided start and end locations; forward and reverse passes are confidence-weighted and merged before a final run.Source: §3.2 · pp. 4–5
- Grounding quality was checked against human-drawn paths using Dynamic Time Warping, with about 10% of the data human-annotated.The authors report approximately five minutes of human location marking per video versus about six hours to draw a full one-hour route manually.Source: §3.2 · p. 5
- Unlike a related approach requiring ARIA glasses and newly collected walks, this system grounds already-filmed videos from any camera and location.Source: §3.2 · p. 5
4 Benchmark tasks
KilometerVision organizes city-scale spatial evaluation around the landmark–route–map hierarchy, using multiple-choice video QA to probe these constructs in walking tours up to about 1 km.
4 Benchmark tasks
- The benchmark uses multiple-choice video QA to probe landmark, route, and map representations, with clips up to 10 minutes (about 1 km) and access to intermediate thinking traces.The tasks reflect stages of spatial awareness from recognizing landmarks to integrating routes into a mental map.Source: §4 · pp. 5–6
4.1 Landmarks
- Landmark questions pair a clip with a candidate frame and ask whether it appears and its straight-line distance from the final frame; absent landmarks provide a negative option.Candidate frames are filtered for VPS confidence above .98 and deduplicated with non-maximal suppression.Source: §4.1 · pp. 6–7
4.2 Routes
- Compass questions give the camera’s starting orientation and ask for its ending orientation, probing simplified path integration across the clip.The task uses eight compass directions, with ground-truth orientations derived from VPS predictions after outlier removal.Source: §4.2 · p. 8
- Loop-closure questions ask whether the tour endpoint was visited earlier and when, including negative cases where it was not revisited.Candidate loops are first identified within a 4 m radius, then manually checked to remove possible VPS-outlier false positives.Source: §4.2 · pp. 8–9
- Route-summary questions ask which set of navigation actions matches the video, using API-generated directions selected to fit the VPS trace.Negative options include partially overlapping segments and reversed routes; fitting a polyline to VPS locations produced noisier, redundant instructions.Source: §4.2 · p. 9
4.3 Survey Knowledge (Maps)
- Map-trace questions ask models to select the route shown on one of five map images, with a harder version removing text labels and landmark icons.Correct and negative route options use the route-summary generation pipeline.Source: §4.3 · p. 10, Figure 6 · p. 10
- The Euclidean-distance task asks for the line-of-sight distance between tour endpoints after viewing the walking route, with incorrect options kept at least 100 m from the ground truth.Ground-truth endpoint positions come from VPS predictions.Source: §4.3 · pp. 10–11
5 Experiments
Across the experiments, GPT-5 leads the tested VLMs but remains below humans on most tasks; the discussion attributes models’ spatial performance largely to visual recognition and semantic matching rather than geometric reasoning.
5.1 Human Baseline
- Humans score 71.3% overall, while the blank-frame Gemini baseline is at chance; human errors concentrate in loop closure, where two VLMs outperform them.The human baseline covers 337 questions, with 10 answers per question.Source: §5.1 · p. 11, §5 · p. 11, Figure 7 · p. 11
5.2 Results and Ablations
- GPT-5 achieves the strongest overall performance, excelling at loop closure and route summary but remaining significantly below humans on all other tasks, especially map tasks.PLM-8B and Qwen2.5-VL-72B perform on par with the blind baseline.Source: Figure 7 · p. 11, §5.2 · p. 12
- 32→600 frames leave Gemini 2.5 Flash near 38% accuracy, while GPT-5 improves from 45% to 57%; lower resolution mainly degrades Claude Opus, except on chance-level map tasks.Gemini and GPT-5 were evaluated at their maximum temporal capacity; the resolution ablation compared 400-pixel with 200-pixel frames.Source: Figure 8 · p. 12, §5.2 · p. 12
- Camera pose and VPS information do not significantly help map traces; labeled maps help route summaries, while unlabeled maps hurt, and thinking ablations show no clear trend.Text route summaries help map recognition only when maps have text labels; removing video can improve results when labeled maps or route summaries are provided alone.Source: §5.2 · p. 13
5.3 Discussion
- VLMs recognize landmarks but poorly ground their distance or loop timing, and fail at physical tracking of turns, relying instead on semantic cues rather than path integration.The authors note low frame rate could also contribute to difficulties estimating rotation angles.Source: §5.3 · pp. 13–14
- On map tasks, VLMs show only an illusion of survey knowledge, defaulting to OCR and street-name matching rather than matching geometric trajectories to map topology.Preliminary attempts to have models locate landmarks on a grid were not satisfactory.Source: §5.3 · p. 14
- Two VLMs exceed human performance only on loop closure, using brute-force frame matching rather than route or map formation; misleading repeated cues can trigger false positives.Gemini 2.5 Flash mistakes a distant statue for the one seen at the end of the video in the illustrated trace.Source: §5.3 · p. 14, Figure 10 · p. 15
6 Conclusion
KilometerVision evaluates city-scale geographical understanding in real-world videos against the human landmark-route-map paradigm. The authors find that current VLMs rely on semantic matching and visual recognition, but fail at path integration and allocentric survey knowledge.
6 Conclusion
- KilometerVision is introduced as the first benchmark comprehensively evaluating city-scale geographical understanding in VLMs using real-world videos.Source: §6 · p. 15
- Current VLMs operate at the landmark stage, simulating spatial awareness with semantic matching and visual recognition but failing at path integration and allocentric survey knowledge.Source: §6 · p. 15
- With GPS, maps, or camera poses provided in context, current state-of-the-art models cannot leverage that privileged information, suggesting these capabilities are not yet mature.Source: §6 · p. 15
Supplementary material
The supplementary material extends the paper’s benchmark description with implementation details, VPS annotation validation, and additional analysis of model results.
Supplementary material
- The benchmark probes landmark recognition, compass, loop closure, route summary, and map trace recognition from real-world hour-long walking-tour videos.The annotation pipeline grounds videos using Google’s VPS and derives labels with route information and heuristics, minimizing human labeling.Source: Supplementary material · p. 21
- The supplement provides further detail on the pipeline for grounding videos on maps and the metric used to validate VPS annotations against human annotations.Source: Supplementary material · p. 21
- It also adds qualitative and quantitative analyses of model results, extending the main paper’s experiments.Source: Supplementary material · p. 21
A1 VPS pipeline details
The VPS pipeline seeds location estimates from human-identified video endpoints, then tracks locations and camera pose across each video. Conservative filtering prioritizes coordinate precision, retaining only a subset of video data.
A1.1 VPS initialisation
- Human-identified start and end locations seed VPS queries on video frames sampled at 4 FPS until confidence exceeds 0.9.The resulting high-confidence locations guide the full-video run at 1 FPS.Source: A1.1 VPS initialisation · p. 21
A1.2 Running VPS on the entire video
- The full-video pass updates estimates when confidence exceeds 0.9 and resets the seed after at most 39 seconds without an update.Forward and backward runs are combined using confidence-weighted coordinate averages, followed by a final per-frame VPS pass.Source: A1.2 Running VPS on the entire video · p. 22
A1.3 Filtering
- Filtering treats accepted and rejected coordinates as a binary classification, choosing thresholds for 100% specificity while minimizing false negatives.It combines confidence and neighboring-point distance thresholds with a 30-second window requiring at least 7 points to pass loose thresholds.Source: A1.3 Filtering · p. 23
- At 0 false positive rate, filtering retains 40% of video data (40% true positive rate) on the validation set.The selected setting corresponds to 100% specificity.Source: A1.3 Filtering · p. 23
A2 DTW-based distance metric between VPS and ground truth
The paper measures trajectory discrepancies by aligning VPS positions with human-drawn paths using a variant of dynamic time warping. The variant handles unequal sampling and gaps in the VPS trajectory.
A2 DTW-based distance metric between VPS and ground truth
- The distance metric sums distances between VPS and ground-truth positions paired by dynamic time warping to minimize total discrepancy.Ground-truth map segments are subsampled so successive positions are no more than 1 meter apart.Source: A2 DTW-based distance metric between… · p. 24
- The DTW variant keeps only the closest ground-truth match when a VPS point aligns with multiple points, avoiding artificially inflated distances from trajectory gaps.VPS positions are sampled at 1 FPS, while filtered-out points can leave gaps.Source: A2 DTW-based distance metric between… · p. 24
- The same metric compares human-collected trajectories with each other and with Routes API trajectories.Source: A2 DTW-based distance metric between… · p. 25
A3 Additional results and analysis
The easier non-overlapping route and map-trace options improve Gemini 2.5 Flash's performance on text-supported tasks, but not map tracing without text. Qualitative examples show models recognizing landmarks and street names while making errors in movement and spatial estimates.
A3.1 Map trace with non-overlapping paths
- Gemini 2.5 Flash performs significantly better on easier Route summary and Map trace with text questions, but still struggles with map trace without text.The authors suggest the gain primarily reflects landmark recognition rather than path construction.Source: A3.1 Map trace with non-overlapping… · p. 25, Figure 13 · p. 26
A3.2 Qualitative results
- In the compass example, Gemini misjudges turn angles and answers incorrectly, while Claude makes a turn-angle mistake but still answers correctly.Claude's response also contains an incorrect middle direction and rotation estimate.Source: A3.2 Qualitative results · p. 25, Figure 14 · p. 27
- With a text-labelled map trace, Gemini identifies streets and relative directions but misreads movement, while Claude matches street names and answers correctly.The illustrated Gemini response mistakes the direction of travel on Hlavné námestie.Source: A3.2 Qualitative results · p. 26, Figure 15 · p. 28
- In a landmark-distance example, Gemini relies on familiar landmarks and incorrect memorized coordinates, while Claude underestimates the distance.The example's correct distance is 178m.Source: A3.2 Qualitative results · p. 26, Figure 16 · p. 29
- The qualitative examples were hand-picked to highlight interesting failure modes.Source: A3.2 Qualitative results · p. 26