Source-linked AI summary
SkyDrive: Learning to Drive in a New City from Aerial Traffic Monitoring
Weijiang Xiong, Lan Feng, Alexandre Alahi, Nikolas Geroliminis
TL;DR
Autonomous-driving models often degrade when transferred zero-shot to cities with different regulations, layouts, and behaviors, while collecting local vehicle-based data is resource-intensive. SkyDrive converts aerial traffic monitoring into supervision for trajectory planning and motion prediction, producing large-scale real-world scenarios. Experiments show pronounced cross-city gaps, with substantial planner improvements from about 30 minutes of monitoring per location.
Problem
Zero-shot autonomous-driving models can degrade in new cities because geographic and behavioral domain shifts limit reliable transfer, while local vehicle-based data collection is resource-intensive.
Method
SkyDrive converts geo-referenced aerial traffic tracks into virtual-ego driving scenes for trajectory planning and vectorized tracks for motion prediction.
Results
Experiments across trajectory planning and motion prediction reveal pronounced cross-city domain gaps, while limited aerial supervision substantially improves target-domain performance.
Takeaways & Limitations
Aerial traffic monitoring provides an efficient and scalable source of real driving supervision for adapting autonomous-driving systems to new cities.
Abstract
from arXiv · showhide
Autonomous driving has made remarkable progress through imitation learning with massive human demonstration data. However, a trained planner often degrades severely when applied to a new environment zero-shot, because of domain shifts in traffic regulations, road layout and driving behaviors. Therefore, adapting a trajectory planner to a new city typically requires resource-demanding local data collection with a vehicle sensor suite. In this work, we show that driving behavior can be learned from a scalable and efficient alternative. We introduce \emph{SkyDrive}, a framework that utilizes drone-based traffic monitoring to provide efficient supervision for autonomous driving agents in a new environment. While vehicle-based data collection logs the ego and its surroundings, an aerial platform naturally observes many road users simultaneously over an extended field of view. As a result, every vehicle can be a data source with grounded driving behavior, effectively scaling up the amount of supervision. Based on 137 hours of aerial traffic monitoring footage, we extract 650K driving samples and construct a benchmark for trajectory planners and motion predictors. Zero-shot experiments with multiple models reveal significant cross-city domain gaps, but many of them can be alleviated by limited supervision from the sky, e.g., 30 minutes of monitoring per location. Our findings show that aerial traffic monitoring is an efficient and scalable data source for adapting autonomous driving systems in new cities. Data and code will be made publicly available.
Introduction
SkyDrive addresses severe zero-shot degradation in new cities by converting aerial traffic monitoring into scalable driving supervision. Its benchmark uses large-scale real-world scenarios to evaluate adaptation for trajectory planning and motion prediction.
- Motivation: Zero-shot transfer can fail because new cities differ in landscape, road layout, traffic regulations, and driving behavior.These shifts affect object detection, motion prediction, and trajectory planning.
- Motivation: Collecting and annotating target-city driving logs with instrumented vehicles requires substantial resources, especially for long-tailed scenarios.Surrounding agents can provide planning supervision, but unclear intentions, fragmented trajectories, and perception noise complicate vehicle-based data.
- SkyDrive: SkyDrive selects vehicles from geo-referenced aerial tracks as virtual egos and creates ego-centric scenes for trajectory planning and vectorized tracks for motion prediction.The resulting supervision is grounded in real driving behavior and supports diverse tasks.
- SkyDrive: Aerial monitoring yields a scalable pipeline for localized supervision, with approximately 650K real scenarios from 137.2 hours over 20 complex intersections.The benchmark evaluates both trajectory planning and motion prediction under zero-shot transfer and aerial-supervision adaptation.
Related Work
Related work spans modular and end-to-end autonomous-driving models, multimodal trajectory generation, simulation and rendering, semantic scene representations, and drone-based traffic datasets. SkyDrive builds on aerial monitoring while repurposing such data for trajectory planning and motion prediction.
- Motion prediction: Modular motion-prediction systems encode agents, driving intentions, maps, traffic lights, and ego history, but cross-domain gaps remain significant.AutoBot, MTR, and Wayformer represent different approaches to forecasting heterogeneous traffic futures.
- End-to-end planning: Modern end-to-end systems learn trajectory planning directly from sensor observations, using camera, LiDAR, BEV, vectorized, or flow-based representations.Examples include TransFuser, UniAD, VAD, FlowDrive, SparseDrive, and DrivoR.
- Multimodal planning: Multimodal planners generate and score alternative futures because real-world driving involves considerable uncertainty.Prior methods use planning vocabularies, probability distributions, learned scores, diffusion, or goal-conditioned flows.
- Simulation and representation: Traffic simulators and neural rendering provide diverse scenarios and sensor observations, while semantic representations reduce reliance on photorealistic appearance.These approaches range from CARLA and MetaDrive to reconstructed 3D scenes, segmentation masks, and rendered semantic views.
- Aerial traffic data: Drone datasets exploit aerial mobility and low occlusion to record trajectories, signals, joint vehicle behavior, and vulnerable road users.SkyDrive repurposes Songdo Traffic’s geo-referenced coordinates and lane annotations into the SongdoDrive dataset for planning and prediction.
SkyDrive: Learn to Drive from the Sky
SkyDrive converts aerial traffic monitoring into city-focused supervision for trajectory planning and motion prediction. Its SongdoDrive benchmark combines geo-referenced vehicle tracks, semantic views, map information, and evaluation of accuracy, regulation compliance, and safety.
- SkyDrive: Learn to Drive from the Sky: SkyDrive centers aerially tracked agents as virtual egos to convert drone observations into localized driving supervision.Scenes are rasterized into ego-centric multi-view semantic images for planning and exported as vectorized tracks for motion prediction.
- The SongdoDrive Dataset: SongdoDrive repurposes the Songdo Traffic drone dataset into driving scenes for trajectory planning and motion prediction.The source contains geo-referenced vehicle coordinates and annotated lane bounding boxes.
- The SongdoDrive Dataset: The dataset covers 20 complex urban intersections with derived lane directions, stop lines, drivable areas, and lane connectors.These map elements are derived from trajectories and lane bounding boxes.
- The SongdoDrive Dataset: 650K scenarios come from 137.2 hours of drone monitoring logs after trajectory filtering and subsequent data-quality control.Approximately 368K car trajectories remain after near-static filtering, while about 234K segments are rejected later.
- Trajectory Planning Task: Trajectory planning uses 2 seconds of history to predict the next 4 seconds, with semantic images, ego history, and a high-level direction as inputs.Inputs and ground truth are sampled at 2Hz, producing four input frames and eight predicted steps.
- Trajectory Planning Task: Evaluation measures displacement accuracy, drivable-area compliance, collision risk, and motion-prediction quality using ADE, FDE, NCT, TTC, minADE, minFDE, BrierFDE, and Miss Rate.Trajectory types and Kalman Difficulty provide finer-grained analysis beyond aggregate results.
Experiments
Experiments evaluate trajectory planning and motion prediction under zero-shot transfer, in-domain supervision, varying data amounts, and cross-intersection generalization. Results show substantial domain-shift degradation, strong benefits from aerial supervision, and differing model performance across scenarios and tasks.
- Trajectory Planning Results: Both DrivoR and RAP perform unsatisfactorily when applied zero-shot outside their training domain.The reported high ADE, FDE, and TTC infraction rates indicate deviations from reasonable human choices and frequent risks.
- Trajectory Planning Results: 1.12% off-road rate is achieved by RAP in zero-shot evaluation, reflecting better preservation of regulatory adherence than DrivoR.
- Trajectory Planning Results: 66.4% and 57.4% reductions in overall FDE are reported for RAP and DrivoR, respectively, after in-domain training.TTC infraction rates also decrease from 25.56% to 5.89% for RAP and from 38.1% to 10.82% for DrivoR.
- Data Scaling and Cross-Intersection Performance: 1% data supervision substantially improves all four evaluated metrics, while RAP continues improving with more data and DrivoR improves more slowly.The scaling subsets correspond to 1.0, 5.8, 11.7, 23.3, and 58.3 hours of monitoring for 1%, 5%, 10%, 20%, and 50% data.
- Data Scaling and Cross-Intersection Performance: Cross-intersection performance gaps are generally not significant, although DrivoR shows a relatively large NCT gap because its cross-intersection model struggles more with road boundaries.Standard-split models are generally preferable because they use data from all locations.
- Motion Prediction Results: Wayformer performs best across all motion-prediction metrics among models trained on SongdoDrive, while AutoBot is a close second.Zero-shot MTR* has 108% higher BrierFDE and 120% higher minADE than trained MTR.
- Motion Prediction Results: 108% higher BrierFDE and 120% higher minADE are observed for zero-shot MTR* compared with trained MTR, confirming severe cross-domain degradation.BrierFDE errors increase from easy to hard cases, and trained MTR improves 75.8% on hard cases compared with zero-shot inference.
- Motion Prediction Results: Wayformer retains the best BrierFDE across easy, medium, and hard difficulty levels, with a clearer advantage on hard cases.For trajectory types, straight motion is easier while turnings are more challenging; MTR* has a 341% relative error increase for stationary trajectories versus trained MTR.
Conclusions
SkyDrive uses drone-based traffic monitoring to adapt autonomous driving models to new cities. Experiments show that limited aerial supervision can improve planners, while semantic-layout understanding remains important.
- SkyDrive turns drone-based traffic monitoring data into a supervision source for autonomous driving.
- Aerial monitoring efficiently provides diverse, high-quality trajectories from many vehicles within the same operation time.
- Zero-shot evaluation quantified the generalization gap for trajectory planners and motion predictors in unseen cities.
- Confusion between stopping and moving emerged as a common failure mode under cross-city transfer.
- ∼30 minutes of traffic monitoring per location significantly improved trajectory planners in experiments with RAP and DrivoR.
- Training the image encoder to understand semantic layout was important for trajectory-planning performance.