Source-linked AI summary
ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries
Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, Hang Zhao
TL;DR
Existing autonomous-driving pipelines pass limited hand-picked features from perception to prediction, while upstream errors can accumulate. ViP3D instead uses sparse 3D agent queries to detect, track, and predict directly from multi-view videos in a fully differentiable streaming pipeline. On nuScenes, it outperforms traditional and recent end-to-end baselines, with long-term performance remaining sensitive to the detection backbone.
Problem
Separated perception and prediction modules exchange limited hand-picked features, while perception errors can propagate into trajectory prediction.
Method
ViP3D uses 3D agent queries as a shared interface for end-to-end detection, tracking, and trajectory prediction from raw visual inputs.
Results
ViP3D outperforms traditional pipelines and recent end-to-end methods on the visual trajectory prediction task.
Takeaways & Limitations
The query-based design preserves rich visual information while explicitly representing agents across detection, tracking, and prediction.
Takeaways & Limitations
Long-term inference is sensitive to the detection backbone, and adapting ViP3D to different detectors remains necessary.
Abstract
from arXiv · showhide
Perception and prediction are two separate modules in the existing autonomous driving systems. They interact with each other via hand-picked features such as agent bounding boxes and trajectories. Due to this separation, prediction, as a downstream module, only receives limited information from the perception module. To make matters worse, errors from the perception modules can propagate and accumulate, adversely affecting the prediction results. In this work, we propose ViP3D, a query-based visual trajectory prediction pipeline that exploits rich information from raw videos to directly predict future trajectories of agents in a scene. ViP3D employs sparse agent queries to detect, track, and predict throughout the pipeline, making it the first fully differentiable vision-based trajectory prediction approach. Instead of using historical feature maps and trajectories, useful information from previous timestamps is encoded in agent queries, which makes ViP3D a concise streaming prediction method. Furthermore, extensive experimental results on the nuScenes dataset show the strong vision-based prediction performance of ViP3D over traditional pipelines and previous end-to-end models.
1. Introduction
Existing autonomous-driving pipelines separate perception from prediction through hand-picked interfaces, limiting visual information and allowing upstream errors to affect trajectory prediction. ViP3D addresses this with a query-centric, end-to-end vision-based pipeline using raw visual features and 3D agent queries.
- Motivation: Separate perception and prediction modules communicate through hand-picked features such as trajectories, agent types, and sizes.This interface omits fine-grained cues such as brake lights, tail lights, head pose, and body pose.
- Motivation: Historical trajectories from upstream perception contain errors that can accumulate and reduce prediction performance.
- Related limitations: LiDAR-based end-to-end methods cannot exploit abundant fine-grained camera information and use non-differentiable decoding or association operations.
- ViP3D: ViP3D uses 3D agent queries throughout an end-to-end pipeline to predict future trajectories from multi-view videos and high-definition maps.
- Contributions: ViP3D is presented as the first fully differentiable vision-based approach for future agent trajectory prediction.
- Contributions: ViP3D uses rich visual features from raw images, explicitly models detection, tracking, and prediction, and outperforms traditional and recent end-to-end baselines.
2. Related Work
Prior work studies camera-based 3D detection, tracking, trajectory prediction, and jointly optimized perception-prediction systems through varied representations and modeling strategies.
- 3D Detection: Camera-based 3D detection methods range from monocular detectors and depth-lifting pipelines to sparse query-based multi-view detection.
- 3D Tracking: Most 3D tracking methods detect objects and then associate detections with existing tracklets using geometric, probabilistic, appearance, or motion cues.
- Trajectory Prediction: Trajectory prediction research encodes historical trajectories and map geometry with graph neural networks and Transformers, while variety loss supports multiple plausible futures.
- Trajectory Prediction: Other trajectory predictors model uncertainty with latent variables or first predict goals such as endpoints, lanes, or intentions.
- End-to-End Perception and Prediction: End-to-end perception and prediction methods jointly optimize tasks using LiDAR or visual inputs, including detection, tracking, localization, occupancy forecasting, and trajectory prediction.
3. Method
ViP3D uses a query-centric design in which 3D agent queries carry agent information across time from visual detection and tracking into map-conditioned trajectory prediction.
- Pipeline: 3D agent queries serve as the main thread across time, updating from multi-view image features into tracked agent queries.
- Pipeline: Tracked agent queries can contain visual characteristics and motion dynamics before entering the query-based prediction module.
- Pipeline: The pipeline is organized as query-based detection and tracking followed by trajectory prediction using tracked queries and map information.
3.1. Query-based Detection and Tracking
ViP3D updates sparse agent queries with multi-view visual features, supervises their temporal identities, and stores historical query states in a fixed-size memory bank for efficient temporal modeling.
- Query Feature Update: Query-based detection and tracking extracts surrounding-camera features and updates agent queries across time to model agent motion dynamics.
- Query Feature Update: Each agent query corresponds to at most one agent and is updated by projecting 3D reference points into multi-view images and applying cross attention to extracted features.
- Query Feature Update: Cross attention uses query, key, and value projections of agent queries and image features, followed by a feed-forward network.
- Query Supervision: Per-timestep supervision preserves query-agent identity across frames by retaining assignments for matched queries and reinitializing queries when agents disappear.
- Query Supervision: Unmatched queries are assigned to newly appearing agents through bipartite matching based on predicted centers and matching costs involving class and box terms.
- Query Memory Bank: A query memory bank stores historical states in a fixed-size FIFO queue, applying attention between each current query and its stored states for efficiency.
3.2. Query-based Prediction
Query-based prediction uses tracked agent queries instead of a separate agent encoder, combines them with vectorized HD-map features, and decodes multiple possible future trajectories per agent.
- Query-based Prediction: Tracked agent queries replace the typical agent encoder output, so the prediction module consists of a map encoder and trajectory decoder.The queries produced by query-based detection and tracking serve as tracked agent representations.
- Query-based Prediction: The map encoder represents HD semantic maps containing lane types, road boundaries, traffic signs, and topological information as map features.Vectorized map encoding is used to produce the map features.
- Query-based Prediction: Cross attention enables interactions between map features and agent queries before trajectory prediction.The map features M interact with agent queries through cross attention.
- Query-based Prediction: The trajectory decoder takes agent queries as input and produces K possible future trajectories for each agent.ViP3D supports regression-based, goal-based, and heatmap-based decoding strategies.
3.3. Loss
ViP3D is trained end-to-end with losses for query classification, query coordinate regression, and trajectory decoding.
- Loss: The total training loss combines query classification, query coordinate regression, and trajectory decoding losses.The objective is L = Lcls + Lcoord + Ltraj.
4. Experiments
ViP3D is evaluated on nuScenes against multi-stage and end-to-end baselines using trajectory, end-to-end accuracy, input, decoder, view, and qualitative analyses. It outperforms the compared baselines, while long-term performance remains sensitive to the detection backbone.
- Evaluation metric: EPA complements minADE, minFDE, and miss rate by penalizing false-positive agents in end-to-end visual trajectory prediction.EPA matches predicted and ground-truth agents within τEPA = 2.0m, then accounts for matched agents whose minFDE is at most τEPA.
- Experimental settings: ViP3D is trained and evaluated on nuScenes, predicting all agents simultaneously with allocentric coordinates used for the reported comparisons.The dataset contains 1,000 scenes from Boston and Singapore, with multi-camera imagery and annotations at 2Hz.
- Main results: ViP3D outperforms the traditional perception-prediction pipeline and PnPNet-vision on all reported metrics under a fair detector and trajectory-decoder setup.The comparison uses DETR3D as detector and regression-based decoding for all methods.
- Ablation study: Agent queries improve over agent trajectories alone, while full end-to-end learning further improves performance over both trajectory-based input variants.The ablation attributes the gains to finer visual information in queries and reduced error accumulation in the multi-stage pipeline.
- Trajectory decoding methods: ViP3D remains superior to the traditional pipeline with goal-based and heatmap-based trajectory decoders, showing compatibility with multiple decoding methods.The compared decoders are TNT and HOME.
- Analysis and limitations: Allocentric prediction performs better than egocentric prediction, whereas PETRv2 improves short-term inference but fails in long-term inference beyond 10 seconds.The detector analysis identifies long-term inference as sensitive to the detection backbone and regards adaptation to different detectors as a limitation.
5. Conclusion
ViP3D predicts agent trajectories from multi-view videos with a fully differentiable pipeline that explicitly models agent instances. Its query-based design exploits raw visual information while supporting interpretability and debugging.
- ViP3D is a fully differentiable approach for predicting future agent trajectories from multi-view videos.
- 3D agent queries explicitly model agent instances, making the pipeline interpretable and debuggable.
- The qualitative comparison displays ground-truth, ViP3D, and traditional-pipeline trajectories, with only the highest-probability prediction drawn per agent.
A. Implementation Details
ViP3D is implemented as a query-based system combining visual backbone processing, query-based detection and tracking, and high-definition map encoding.
- Models are trained on nuScenes for 24 epochs with ADAM, batch size 8, and learning-rate decay at epochs 20 and 23.The initial learning rate is 2e−4; hidden sizes are 256 for detection and tracking and 128 for trajectory prediction.
- Query-based detection and tracking uses ResNet50 with a DETR3D detection head containing six feature-refinement and multi-head-attention layers.Separate branches predict agent center coordinates and sizes, and agent types.
- ViP3D encodes HD maps by converting each lane into an ordered sequence of vectors containing segment endpoints and lane attributes.
B. Trajectory Decoding
ViP3D decodes each agent query into multiple possible future trajectories using regression-based, goal-based, or heatmap-based methods.
- ViP3D supports regression-based, goal-based, and heatmap-based trajectory decoding methods.
- Regression-based: The regression decoder uses a two-layer MLP to map agent queries directly to multiple future trajectories.During training, it selects the trajectory closest to the ground truth and applies smooth ℓ1 regression loss.
- Goal-based: The goal-based decoder encodes randomly generated candidate goals, predicts their probabilities and offsets, and completes trajectories from them.
- Heatmap-based: The heatmap-based decoder densely samples goals at 1m spacing, selects K goals with NMS, and completes them into K trajectories.
C. Qualitative Results
Qualitative results show ViP3D predicting accurate trajectories on nuScenes, while failure cases reveal difficulty detecting distant or heavily occluded agents.
- ViP3D produces accurate future trajectories in the qualitative nuScenes validation examples.The visualizations show ground-truth and predicted trajectories for tracked agents.
- ViP3D struggles to detect agents that are far from the ego vehicle or substantially occluded.Examples include a distant, partially obscured vehicle and a pedestrian mostly hidden by a billboard.