Source-linked AI summary
CAR-Net: Clairvoyant Attentive Recurrent Network
Amir Sadeghian, Ferdinand Legros, Maxime Voisin, Ricky Vesel, Alexandre Alahi, Silvio Savarese
TL;DR
Path prediction must account for scene context, which can contain sparse, distant cues that past motion alone misses. CAR-Net combines past trajectories and raw top-view images with single- and multi-source attention. It outperforms previous state-of-the-art approaches on SDD and uses track elements to predict optimal paths on F1 tracks.
Problem
Method
CAR-Net combines an agent’s past trajectory with raw top-view scene images and single- plus multi-source attention to select relevant image regions.
Source: §1 · p. 2, §3.3 · pp. 5–6Results
CAR-Net outperforms previous state-of-the-art approaches on SDD by a large margin and uses track elements to predict optimal paths on F1 tracks.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
Attention visualizations identify scene elements associated with predicted paths, such as a curve ahead of the vehicle.
Source: §4.5 · p. 14
Abstract
from arXiv · showhide
We present an interpretable framework for path prediction that leverages dependencies between agents' behaviors and their spatial navigation environment. We exploit two sources of information: the past motion trajectory of the agent of interest and a wide top-view image of the navigation scene. We propose a Clairvoyant Attentive Recurrent Network (CAR-Net) that learns where to look in a large image of the scene when solving the path prediction task. Our method can attend to any area, or combination of areas, within the raw image (e.g., road intersections) when predicting the trajectory of the agent. This allows us to visualize fine-grained semantic elements of navigation scenes that influence the prediction of trajectories. To study the impact of space on agents' trajectories, we build a new dataset made of top-view images of hundreds of scenes (Formula One racing tracks) where agents' behaviors are heavily influenced by known areas in the images (e.g., upcoming turns). CAR-Net successfully attends to these salient regions. Additionally, CAR-Net reaches state-of-the-art accuracy on the standard trajectory forecasting benchmark, Stanford Drone Dataset (SDD). Finally, we show CAR-Net's ability to generalize to unseen scenes.
1 Introduction
CAR-Net predicts trajectories from agents’ past motion and top-view scene images, using visual attention to identify relevant scene regions. The paper introduces an F1-track dataset to study agent–space relationships and reports improved forecasting on SDD.
1 Introduction
- Scene-context forecasting must model agent–space interactions even though relevant cues may be sparse, small, and far from the agent.The task uses the agent’s past coordinates and a large top-view image to forecast its future trajectory.Source: §1 · pp. 1–2
- Earlier raw-image trajectory prediction methods did not reveal which visual information the model used to predict future paths.Some prior approaches instead relied on scene semantics collected in advance.Source: §1 · p. 2
- CAR-Net combines single-source and multi-source visual attention to predict trajectories from raw top-view images and past motion.It can attend to one region or a collection of image regions.Source: §1 · p. 2
- On SDD, CAR-Net outperforms previous approaches; on the new F1-track dataset, it uses track elements to predict vehicles’ optimal paths.The F1 dataset contains more than two hundred real-world tracks.Source: §1 · p. 3
- The F1-track dataset isolates scene effects by using static scenes, where vehicle paths are governed by track geometry rather than other agents.The authors compute optimal vehicle paths from track curvatures.Source: §1 · p. 3
2 Related Work
Related work covers trajectory forecasting from agent motion, methods that incorporate static scene context, and visual attention models. The paper positions CAR-Net as addressing agent–space interactions while making scene cues used in predictions more interpretable.
Trajectory forecasting
- Trajectory models address agent–agent interactions, but the cited approaches do not account for agent–space interactions in complex environments.The discussion contrasts Kalman filters, regression, Gaussian processes, social-force models, and recurrent models with CAR-Net’s use of scene context.Source: §2 · p. 3
Static scene context
Visual attention
- Visual attention methods include differentiable soft attention over image features and recurrent attention over sequences of image crops.The cited hard-attention approach requires reinforcement learning, while soft attention is differentiable.Source: §2 · p. 4
3 CAR-Net
CAR-Net predicts an agent’s future path from its past trajectory and a top-view scene image, using attention to select relevant scene context. It combines attention to localized and scattered cues with recurrent trajectory generation.
3.1 Overall Architecture
- The model predicts future positions recurrently from the agent’s past trajectory and a top-view image, conditioning each LSTM step on scene context and prior state and position.The attention module produces a context vector representing salient image areas for each time step.Source: §3.1 · pp. 4–5
3.2 Feature extractor module
- The feature extractor uses ImageNet-pretrained VGGnet-19 fine-tuned for scene segmentation to identify image features such as roads, obstacles, and sidewalks.It uses the output of the fifth convolutional layer, with each feature vector corresponding to an image region.Source: §3.2 · p. 5
3.3 Visual attention module
- CAR-Net concatenates single-source and multi-source context vectors to combine information from local and scattered visual cues.The paper describes the two attention mechanisms as complementary rather than interchangeable.Source: §3.3 · pp. 5–6
- CAR-Net outperforms both its single-source and multi-source attention mechanisms.The combined module uses both mechanisms to cover localized and scattered visual cues.Source: §3.3 · p. 6
- Multi-source attention weights spatial areas across the scene, while single-source attention uses a Gaussian grid to select one local image region.The single-source grid’s stride controls the size of the attended area.Source: §3.3 · pp. 6–7
- For multi-source attention, the attention weights depend on the previous hidden state, not on image features, which the authors say reduces computation without affecting performance.The paper motivates this choice by noting that future scene images are unavailable for path prediction.Source: §3.3 · pp. 6–7
- The attention layer maps the LSTM’s previous hidden state to attention parameters that form a Gaussian grid over a local image region.The model predicts the grid center, stride, and Gaussian variance at each time step.Source: §3.3 · p. 7
3.4 Implementation details
- The LSTM and attention module are trained from scratch with Adam for 100 epochs; at test time, each predicted position is fed back as the next input.Training instead feeds ground-truth positions into the LSTM.Source: §3.4 · pp. 7–8
4 Experiments
Experiments compare CAR-Net with prior methods, ablate its attention mechanisms, examine interpretability, and test generality and robustness across datasets.
4.1 Data
- The study evaluates real-world F1 tracks and synthetic Car-Racing; F1’s static setting isolates spatial-information contributions from other-agent interactions, with 250 tracks and more than 100K trajectories.F1 trajectories illustrate optimal driving patterns, and its test tracks are totally unseen locations.Source: §4.1 · pp. 8–9
4.2 Evaluation Metrics and Baselines
- Models are evaluated by mean trajectory L2 (ML2) and final-position L2 (FL2), against linear, social, trajectory-only, image-only, attention-only, and DESIRE baselines.The attention-only baselines use either multi-source or single-source attention; DESIRE results use its top-1 sample.Source: §4.2 · pp. 9–10
4.3 Ablation Study
- Across Car-Racing and F1, attention models outperform whole-image LSTM, and CAR-Net beats single- and multi-source variants by combining their complementary cues.The authors attribute the whole-image model's limited gain over trajectory-only LSTM to complex track feature maps.Source: Table 1 · p. 10, §4.3 · p. 10
4.4 Trajectory Forecasting Benchmark
- CAR-Net achieves the lowest SDD ML2 and FL2 errors, outperforming state-of-the-art methods when predicting 12 future positions from 8 past positions.DESIRE's reported value was interpolated from a 4-second prediction task to 4.8 seconds.Source: Table 2 · p. 11, §4.4 · p. 11
- CAR-Net generalizes to unseen locations, with similar SDD mean L2 errors on seen and unseen test locations: 23.87 and 26.93 pixels, respectively.Nine of 17 SDD test locations are unseen; all F1 test tracks are unseen locations.Source: §4.4 · p. 11
- On semantically simple SDD scenes CAR-Net and T-LSTM perform similarly, while CAR-Net performs much better on complex scenes.The authors interpret this as evidence that CAR-Net leverages scene context when scene semantics are likely to influence trajectories.Source: Table 3 · p. 12, §4.4 · p. 12
4.5 Qualitative analysis
- Redirecting attention toward a road orientation shifts predicted paths toward it, while off-road history perturbations recover onto the road in Car-Racing and lightly perturbed F1.The F1 experiment uses a smaller perturbation than Car-Racing because F1 is more challenging.Source: Figure 6 · p. 13, §4.5 · p. 14
- On racing tracks, CAR-Net uses near-car single-source attention and farther multi-source attention to identify upcoming curves; on SDD, multi-source attention covers scattered regions.On SDD, the single-source grid also reaches farther ahead than on racing tracks; attended regions grow over time, possibly reflecting prediction uncertainty.Source: §4.5 · p. 13
5 Conclusions
CAR-Net combines past trajectories and top-view scene images with visual attention to predict trajectories. It outperforms prior state-of-the-art approaches on SDD and uses attention to identify relevant scene semantics.
5 Conclusions
- CAR-Net outperforms previous state-of-the-art approaches by a large margin on the SDD trajectory forecasting benchmark.Source: §5 · p. 14
- CAR-Net processes past trajectory positions and top-view navigation-scene images using an attention mechanism that leverages multiple types of visual attention.Source: §5 · p. 14
- The authors introduce top-view images of hundreds of F1 race tracks, where vehicle dynamics are largely governed by specific image regions such as upcoming curves.Source: §5 · p. 14
- Visualized attention shows that CAR-Net leverages relevant scene semantic features when predicting trajectories.Source: §5 · p. 14