Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding

• Predictions should cover multiple plausible goal-directed futures while obeying physical constraints and remaining in drivable areas.Source: §1 · p. 2
• Because intentions, social interactions, and environmental constraints are unobserved, forecasting must infer actions from visible environmental features and simulate futures.Source: §1 · p. 2
• The model treats scene context and neighboring-agent interactions as complementary multimodal cues for diverse, admissible forecasting.Source: §1 · p. 2
• Most public datasets provide only one annotated future, while annotation-based displacement metrics do not faithfully assess diverse, admissible predictions.Source: §1 · p. 2
• The proposed system combines cross-agent self-attention, map-based visual attention, constrained flow decoding, and symmetric cross-entropy training.Source: §1 · p. 3
• Its drivable-area-based annotation-free distribution approximation and DAO metric assess prediction diversity under admissibility without requiring multiple future annotations.Source: §1 · p. 3

• Prior systems fuse LiDAR, map, or RGB cues and sometimes combine agent interactions; this work argues for explicitly modeling both agent-to-scene and agent-to-agent relations.Source: §2 · p. 4
• GAN and VAE approaches encourage diverse predictions, but often focus on candidate generation rather than diversity across distributional modes.Source: §2 · p. 4
• Both diversity and admissibility remain difficult to evaluate, motivating this paper’s admissibility-focused task metric.Source: §2 · p. 4

• An agent is an on-road dynamic object represented by 2D positions over time; t=0 denotes present, t≤0 past, and t>0 prediction.Source: §3 · p. 4
• The observation combines agents’ past trajectories with a bird’s-eye-view scene representation whose channels encode features such as drivable areas, positions, and distances.Source: §3 · pp. 4–5
• Diversity means covering modes of the future-trajectory distribution, with multiple samples treated as independent hypotheses for the same observation.Source: §3 · p. 5
• Admissible predictions should lie within the support of the true future-trajectory distribution rather than regions that are physically or statistically implausible.Source: §3 · p. 5

4.1 Model Architecture

• The encoder models interactions among agents, while the decoder generates future positions autoregressively using local and global scene context.Past trajectories are encoded and fused through cross-agent self-attention; the decoder also attends to scene context at each step.Source: §4.1 · pp. 5–6
• The model maps Gaussian noise z_t sampled from N(0, I) through σ_t z_t + μ_t to a trajectory, constraining μ_t to improve the prediction’s physical admissibility.At α = 1, the constraint is equivalent to Verlet integration and gives a perfect constant-velocity prior, which empirically overfits datasets; perturbing with α prevents overfitting, so α = 0.5 is chosen.Source: §4.1 · p. 7

4.2 Learning

• Symmetric cross-entropy trains the predicted distribution to cover plausible future modes while penalizing samples that are less likely under the real distribution.The objective combines forward and reverse cross-entropies and is optimized using ground-truth trajectories and scene context.Source: §4.2 · p. 8

4.3 The Drivable-area Map and Approximating p

• The drivable-area map approximates the real trajectory distribution by assigning equal probability to drivable locations and decreasing probability to non-drivable locations with distance.A distance transform over the map is converted into a probability distribution with softmax; the map also supplies scene input and supports the diversity and admissibility metrics.Source: §4.3 · p. 8

5 Experimental Setup

• The experiments target interaction modeling, diverse and admissible predictions, robustness to increasing agent counts, and generalization across domains.Source: §5 · p. 9

5.1 Dataset

• NuScenes provides longer scenarios but about 25K episodes, while Argoverse provides about 320K independent five-second episodes.Both datasets include drivable-area maps; the experiments use them for complementary validation across domains.Source: §5.1 · p. 9
• NuScenes sequences are cropped to five seconds—two seconds observed and three predicted—and Kalman smoothing reduces trajectory noise and imputes missing points.Source: §5.1 · p. 9

5.2 Baseline Models

• Deterministic baselines include LSTM, convolutional social pooling, and MATF-D; stochastic baselines include DESIRE, MATF-GAN, and R2P2-MA.Deterministic models use matched capacity for a fair comparison.Source: §5.2 · p. 9
• The model ablation progresses from CAM-NF to local and global scene features, then agent-to-scene attention in the final ATTGLOBAL-CAM-NF model.Source: §5.2 · p. 10

5.3 Metrics

• Precision uses ADE and FDE errors, while RF—the ratio of average to minimum FDE—characterizes prediction spread and distinguishes multimodal coverage from perturbation.The paper reports MINADE and MINFDE for precision and RF, DAO, and DAC for diversity and admissibility.Source: §5.3 · pp. 10–11
• DAO measures predicted occupancy within the drivable area, while DAC counts off-road hypotheses; together with RF, they distinguish diversity from admissibility failures.DAO is normalized by 10,000 in reported results; the hypothesis count is held equal across models, with k=12 by default.Source: §5.3 · pp. 10–11, Figure 6 · p. 11

Experimental settings

• The experiments assess precision, diversity, and admissibility using MINADE, MINFDE, RF, DAC, and DAO; Table 5 identifies its baseline and ablation partitions.The supplied table caption defines the abbreviations as MINADE (A), MINFDE (B), RF (C), DAO (D), and DAC (E).Source: §6 · p. 11, Table 5 · p. 14

Effectiveness of cross-agent interaction module

• CAM outperforms CSP and MATF-D on MINADE and MINFDE, favoring explicit cross-agent attention over convolutional interaction modeling.The authors infer that cross-agent interactions can reveal road structure and permissible regions from surrounding trajectories.Source: §6 · p. 12, Table 1 · p. 12

Effectiveness of agent-to-scene interaction module

• On NUSCENES, the stochastic models achieve the best precision, diversity, and admissibility; OURS has the highest RF and DAC with DAO comparable to R2P2-MA.R2P2-MA has the highest DAO, while MATF-GAN’s small RF indicates unimodal predictions.Source: Table 2 · p. 12, §6 · p. 13

Generalizability across datasets

• On ARGOVERSE, ATTGLOBAL-CAM-NF outperforms or matches baselines; with 6 hypotheses, its MINFDE is 0.915, compared with 1.399 for MFP3.Source: §6 · p. 13

Effectiveness of new ˜p

• The drivable-area approximation of p improves most reported metrics over MSE training, especially MINERROR, DAO, and RF, while DAC remains comparable.The authors attribute the change to using drivable-area information to estimate additional trajectories beyond the most probable one.Source: §6 · p. 13, Table 3 · p. 13

Complexity from number of agents

• Performance rises as more surrounding agents are included, and the proposed model has the greatest relative MINFDE improvement from 1 to 10 agents.The authors report this pattern even though many nearby agents move little.Source: §6 · p. 13, Table 4 · p. 13

7 Conclusion

• The proposed attention-based agent-to-agent and agent-to-scene interactions improve prediction precision, diversity, and admissibility.The conclusion also presents a richer approximation for flow-based training, holistic diversity metrics, and nuScenes trajectory-extraction code.Source: §7 · p. 14

A.1 nuScenes Trajectory Extraction Process

• For nuScenes, bounding-box tracks are associated by agent ID, then Kalman-smoothed with a constant-velocity model to filter noise and impute occluded or missing points.Expectation-Maximization initializes the filter states and transition and emission covariance parameters.Source: §A.1 · p. 17
• Each nuScenes episode uses 4 past and 6 prediction frames; snippets are normalized at the present ego-pose and paired with 224 × 224 drivable-area masks covering 112 × 112 m.Up to 30 samples are extracted per episode.Source: §A.1 · p. 17

A.2 Argoverse Motion Forecasting Data

• Argoverse provides about 320,000 episodes of 50 frames and map access; episodes are down-sampled to 10 frames and normalized at the present ego-agent, with a 224 × 224 road mask covering 112 × 112 m.Because the episodes are preprocessed for forecasting, association, smoothing, and imputation are unnecessary.Source: §A.2 · p. 17
• The association figure shows bounding boxes across frames, associated and concatenated sequences, then final trajectory snippets, with colors distinguishing agents.Source: Figure 8 · p. 18

A.3 CARLA Trajectory Extraction Process and Experiments

• CARLA trajectories, bird’s-eye-view maps, lane center-lines, and LiDAR are synthesized; four depth sensors supply depth images used to construct point clouds.The CARLA physics engine provides limited surface-contour points through its built-in ray-tracing LiDAR sensor.Source: §A.3 · p. 18
• The experiments use bird’s-eye-view maps to build distance transforms and binary road masks marking drivable areas.Source: §A.3 · p. 18
• Estimated camera calibration transforms perspective segmentation to bird’s-eye view; maps are synchronized with past and future trajectories for five agents per snapshot.The authors say the extended CARLA dataset and its extraction code will be released.Source: §A.3 · p. 18
• The visualization distinguishes ground-truth points in red, past trajectory points in green, and future trajectory points in blue over a bird’s-eye view of drivable and non-drivable areas.Source: Figure 9 · p. 19

Performance Analysis

• In CARLA, ATTGLOBAL-CAM-NF outperforms baselines by about 15% on MINADE and MINFDE; CAM-NF performs well on DAO diversity.The authors report that the full model better mimics the simulator autopilot’s trajectory distribution.Source: Performance Analysis · p. 19, Table 6 · p. 19

A.4 Analysis of α

• α = 0.5 gives the largest RF and DAC, competitive DAO, and the best MINERRORS precision among the tested settings.The authors report that values below 1.0 tend to perform better than the constant-velocity assumption α = 1.0.Source: §A.4 · p. 20

B.1 The Drivable-area Map and Approximating p

• A distance transform of the binary road mask produces ˜p, assigning greatest probability to drivable pixels and decreasing probability with distance from drivable areas.The transformed map is normalized and passed through softmax; it also supplies the basis for scene context Φ.Source: Figure 10 · p. 20, §B.1 · p. 21, Figure 11 · p. 21

B.2 Generating scene context input Φ

• Scene context Φ adds pixel indices and distances from the map center to the distance-transformed map, giving the ConvNet access to spatial position.The augmented input has dimensions 224 × 224 × 3 and is normalized using training-set statistics.Source: §B.2 · p. 22

C Additional Results Visualization

• Across nuScenes, Argoverse, and CARLA, the visualized predictions are diverse and admissible in most cases, including rapid left or right turns.The figures show results on nuScenes, Argoverse, and CARLA, respectively.Source: Figure 13 · p. 22, Figure 14 · p. 23, Figure 15 · p. 23
• Predictions of explicit left or right turns are challenging when the past trajectory proceeds only forward; encoded lane information is proposed as a possible addition.The examples cited are in nuScenes and CARLA.Source: C Additional Results Visualization · p. 24

D.1 Training Setup

• Training uses Adam with an initial learning rate of 1.0E-04, batch size 64 for most models, and early stopping when overfitting is observed.ATTGLOBAL-CAM-NF uses batch size 4; training runs for at most 100 epochs.Source: §D.1 · p. 24

D.2 Hyperparameters

• The experiments predict 3 seconds of future trajectories from 2 seconds of past records, sampled as 2D positions every 0.5 seconds.The setting is used across nuScenes, Argoverse, and CARLA.Source: §D.2 · p. 24

D.3 Network Details

• The encoder uses a linear transform and single-layer LSTM for each observed trajectory, then models agent-to-agent interactions with an attention architecture.The observations are represented by position differences for a translation-robust encoding.Source: §D.3 · pp. 24–26
• The decoder feeds prior outputs back autoregressively through a GRU and uses visual attention to interact with scene context.The appendix identifies Table 8 as listing network hyperparameters.Source: Table 8 · p. 25, §D.3 · pp. 26–27