Source-linked AI summary
Trajectory-aware Cross-view Geo-localization with Sequential Observations
Tianyi Gao, Jiayu Lin, Danielle Beaulieu, Nathan Jacobs
TL;DR
Cross-view geo-localization needs sequential visual and linguistic cues because video-only methods overlook route descriptions useful when visual inputs are unavailable. This paper introduces SeqGeo-VL, TrajLoc, and TrajMod, achieving state-of-the-art performance on video- and text-based geo-localization.
Problem
Video-based cross-view geo-localization overlooks route descriptions, limiting use when visual inputs are unavailable or humans communicate through abstract language.
Method
The paper builds SeqGeo-VL with 38,863 aligned video–text–satellite triplets and proposes TrajLoc with TrajMod to jointly model queries and trajectory geometry.
Results
TrajLoc achieves state-of-the-art performance on cross-view geo-localization from both videos and route descriptions.
Takeaways & Limitations
Jointly modeling sequential visual and linguistic observations, with trajectory geometry, supports cross-view spatial reasoning across both query modalities.
Abstract
from arXiv · showhide
Cross-view geo-localization matches ground-level observations against geo-tagged satellite imagery. Recent methods show that sequential queries such as video clips yield richer spatiotemporal cues than single images, yet they overlook a complementary sequential modality: route descriptions -- which capture the same trajectory at a higher level of abstraction and are often the only input available (e.g., a user directing an autonomous vehicle to a pickup point). To bridge this gap, we introduce SeqGeo-VL, a dataset of $\sim$39K video-text-satellite triplets, and TrajLoc, a unified framework capable of processing both video clips and route descriptions. By leveraging both dense visual and abstract linguistic semantics, TrajLoc enables these modalities to mutually reinforce cross-view matching. We further propose TrajMod, a lightweight module that conditions query embeddings on trajectory geometry, yielding spatially-aware representations. Experiments show that TrajLoc achieves substantial gains over state-of-the-art methods on both video and text geo-localization. The project page is available at https://humblegamer.github.io/trajloc/.
1 Introduction
Existing cross-view geo-localization methods use sequential visual observations but overlook route descriptions and the synergy between textual and visual trajectory cues. The paper introduces SeqGeo-VL, TrajLoc, and TrajMod to support video and route-description localization with spatially grounded representations.
- Motivation: Noisy or degraded GPS in crowded urban environments makes precise localization from egocentric visual observations or natural-language descriptions necessary for safe navigation and cooperation.Tall buildings and vegetation frequently impair GPS signals, rendering global localization unreliable.
- Motivation: Sequential observations improve retrieval robustness through motion continuity, changing viewpoints, and route-level semantics, but video-to-image methods overlook textual narratives and visual-input-limited applications.The omitted textual modality limits use when visual inputs are unavailable or during human-robot interaction.
- Problem: Existing datasets lack route descriptions and video–text–satellite triplets, leaving the synergy between route descriptions and trajectory videos unexplored.This gap restricts flexibility in cross-view geo-localization.
- Contributions: 38,863 aligned video–text–satellite triplets comprise SeqGeo-VL, created with a progressive VLM-LLM annotation pipeline and human verification for video- and text-based geo-localization.The pipeline combines trajectory-shape priors, frame-level VLM labeling, LLM summarization, and sample-based human verification.
- Contributions: TrajLoc unifies video and route-description queries using separate modality encoders, a shared satellite encoder, and two-stage curriculum learning for cross-modal co-training.The curriculum addresses alignment difficulties that can hinder optimization of the shared satellite image encoder.
- Contributions: TrajMod injects trajectory geometry into vision-language embeddings, producing spatially grounded representations with state-of-the-art performance on cross-view video and text geo-localization.The motivation is that general CLIP and MLLM embeddings often fail to capture spatial layout.
2 Related Work
Prior cross-view geo-localization work has expanded from single images to sequential video retrieval, while natural-language localization remains limited and spatial understanding remains difficult. SeqGeo-VL addresses these gaps by enriching sequential data with trajectory-level text descriptions and targeting weaker metadata assumptions than topology-based approaches.
- Sequential Video Geo-localization: Sequential cross-view geo-localization datasets such as GaMa, SeqGeo, and CVLNet primarily support video queries, leaving trajectory-level textual descriptions underexplored.The proposed benchmark builds upon SeqGeo and uses a progressive annotation pipeline to add trajectory-level text for multimodal sequential observations.
- Sequential Video Geo-localization: Recent video-query methods improve robustness to viewpoint and appearance changes through specialized temporal aggregation and fusion mechanisms.Examples include GaMa’s 3D-CNN, SeqGeo’s Temporal Feature Aggregation Module, GARet’s transformer adapter, and FlexGeo’s inter-frame-similarity guidance.
- Language-Based Geo-localization: Natural-language cross-view geo-localization has limited prior work because text-to-satellite retrieval requires identifying an exact geolocation from visually grounded descriptions.CrossText2Loc aligns text and satellite-image embeddings with contrastive learning on CLIP and introduces Expanded Positional Embedding, while its dataset focuses on single isolated descriptions.
- Spatial Representation Learning: CLIP and even advanced multimodal models struggle with spatial relationships, relative directions, spatial layouts, and mapping egocentric video to allocentric positions and camera trajectories.SpatialRGPT and SpatialCLIP incorporate depth with RGB imagery to model properties such as directionality, adjacency, and perspective transformation.
- Spatial Representation Learning: UGE integrates street-view imagery, road networks, and POI data as graph-based spatial topology, whereas the proposed approach requires neither POI data nor road networks.The contrast is framed as operating under weaker assumptions about available metadata while retaining the goal of outdoor geolocalization.
3 Problem Formulation and Benchmark
The paper formulates cross-view geo-localization over waypoint trajectories with either video or route-description observations, optionally using trajectory information to retrieve a matching satellite image. It introduces SeqGeo-VL to address the lack of sequential route-description benchmarks through multimodal triplets and hierarchical annotation.
- 3.1 Task Definition: A trajectory is represented by waypoints with relative coordinates and headings, while sequential observations are provided as either detailed video or abstract route description.The observation sequence is denoted O_1:N ∈ {V, T}.
- 3.1 Task Definition: The task retrieves the satellite image maximizing similarity between the query encoder output and each satellite-image embedding.The query encoder processes video or text, and the satellite encoder embeds reference imagery.
- 3.1 Task Definition: Trajectory information enables trajectory-aware cross-view localization but remains optional when trajectory data are unavailable.The agent’s location is derived from the geo-tag associated with the retrieved satellite image.
- 3.2 Benchmark Dataset: Existing datasets mostly use single-image queries, while some support sequential video or single-text queries but not sequential route descriptions.This gap motivates extending SeqGeo into a multimodal benchmark for sequential observations.
- 3.2 Benchmark Dataset: A four-stage hierarchical pipeline generates trajectory-level descriptions from frame semantics, motion primitives, and structural summarization.Qwen3-VL-8B produces detailed frame captions, motion metadata adds actions such as turning left, and Qwen3-30B performs structural summarization.
- 3.2 Benchmark Dataset: SeqGeo-VL contains 38,863 triplets of videos, route descriptions, and geo-tagged satellite images at zoom level 20.The dataset follows the original SeqGeo protocol with an 80:20 training-testing split.
4 Approach
TrajLoc is a unified three-encoder framework for video- and route-description-to-satellite geo-localization, trained with a two-stage curriculum. TrajMod further conditions query embeddings on waypoint and global trajectory geometry to produce spatially-aware representations.
- Unified architecture: TrajLoc uses three CLIP-initialized encoders for video, text, and satellite imagery, enabling unified cross-view retrieval across both query modalities.The satellite encoder provides the shared retrieval reference for video and text.
- Text encoder: Long route descriptions average ∼129 tokens, so TrajLoc extends CLIP’s default 77-token context window by linearly interpolating positional embeddings.The interpolation preserves pretrained positional semantics while enabling trajectory-level text processing.
- Trajectory geometry embedding: Trajectory geometry combines waypoint heading angles and OD bearing relative to True North, forming z_traj for spatial conditioning.Waypoint headings capture local turning patterns, while OD bearing supplies stable global orientation.
- Spatially-grounded modulation: TrajMod uses modality-specific MLPs to predict scale and shift parameters from z_traj, modulating video and text embeddings into spatially-aware representations.The modulation follows a FiLM-inspired transformation of each query embedding.
- Two-stage curriculum: The two-stage curriculum first aligns video and satellite embeddings, then freezes the video encoder while aligning text and regularizing the satellite encoder against drift.This curriculum addresses the larger modality gap between abstract route descriptions and dense satellite imagery.
- TrajMod training: After freezing TrajLoc’s encoders, training optimizes TrajMod with video–satellite and text–satellite contrastive alignment, while a text–video term regularizes shared geometry-grounded factors.Both modalities are conditioned on the same trajectory geometry embedding.
5 Experiments
TrajLoc outperforms prior methods on both video and text cross-view geo-localization, with especially strong gains from trajectory-conditioned spatial reasoning. Ablations show that longer sequences, sequential relational context, location priors, co-training, and explicit geometric modulation improve retrieval.
- Video geo-localization: 12.09% R@1: TrajLoc outperforms SeqGeo† on video geo-localization using the same CLIP ViT-L/14 backbone and a lower computational budget.Upgrading SeqGeo from VGG16 to CLIP ViT-L/14 raises its R@1 from 1.80% to 8.14%, while TrajLoc reaches 12.09%.
- Text geo-localization: 2.52% R@1 and 45.48% R@1%: TrajLoc achieves the best text geo-localization performance, exceeding CrossText2Loc by 2.6× in R@1 and EVA2-CLIP by 1.7× in R@1%.Scaling CLIP-style backbones or increasing input resolution provides only modest gains, indicating spatial reasoning remains the bottleneck.
- Ablations: R@1, R@5, and R@10 generally improve as video frames increase from 1 to 6, confirming that longer temporal context provides richer localization cues.The evaluation compares varying frame counts with and without TrajMod.
- Ablations: Restricting the candidate gallery using a location prior substantially improves absolute retrieval accuracy by narrowing the search space.The experiment simulates users or robots knowing a rough geographic district when GPS is obscured.
- Ablations: With TrajMod, co-training raises video R@1 from 9.69% to 12.09% and text R@1 from 1.90% to 2.52%, whereas co-training alone yields only marginal gains.Without TrajMod, video R@1 changes from 6.87% to 7.27% and text R@1 from 0.80% to 0.98%.
- Ablations: 22.29% R@1: full SeqGeo-VL outperforms sequential-view objects at 17.41% and single-view descriptions at 10.63%, showing relational spatiotemporal context is essential.Sequential-view objects use only 26.3 words but retain chronological order, while the single-view variant describes only the final waypoint.
6 Conclusion
The paper extends route-level cross-view geo-localization to natural-language route descriptions and introduces a benchmark, unified framework, and geometry-aware feature module. TrajLoc achieves state-of-the-art performance for both video and route-description queries.
- SeqGeo-VL introduces a multimodal benchmark of video-text-satellite triplets for route-level cross-view geo-localization.
- TrajLoc provides a unified approach for processing video queries and natural-language route descriptions.
- TrajMod is a lightweight module that incorporates explicit trajectory geometry for feature modulation.
- TrajLoc achieves state-of-the-art performance on cross-view geo-localization from both videos and route descriptions.