Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data

Sergio Casas; Cole Gulino; Renjie Liao; Raquel Urtasun

Source-linked AI summary

Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data

Sergio Casas, Cole Gulino, Renjie Liao, Raquel Urtasun

arXiv:1910.08233v1Oct 18, 2019cs.CVcs.LGcs.ROeess.SP

TL;DR

The paper addresses relational behavior forecasting and joint detection in autonomous-driving sensor data, where existing approaches do not explicitly model interactions between agents. It proposes SPAGNN, a spatially aware probabilistic GNN with differentiable message passing over detected actors. SPAGNN achieves significant improvements over the state of the art across detection, motion forecasting, and interaction metrics on ATG4D and NUSCENES.

Problem
Safe autonomous driving requires detecting vehicles and forecasting their long-term behavior, but existing approaches do not explicitly model interactions between agents.
Method
SPAGNN uses a CNN for actor detection and initial states, then iteratively updates actor states through spatially transformed probabilistic messages in a graph neural network.
Results
SPAGNN achieves significant improvements over the state of the art across detection, motion forecasting, and interaction metrics on ATG4D and NUSCENES.
Takeaways & Limitations
Joint spatially aware relational modeling provides socially coherent probabilistic trajectory estimates for autonomous-driving scenes.

Abstract

from arXiv · show

In this paper, we tackle the problem of relational behavior forecasting from sensor data. Towards this goal, we propose a novel spatially-aware graph neural network (SpAGNN) that models the interactions between agents in the scene. Specifically, we exploit a convolutional neural network to detect the actors and compute their initial states. A graph neural network then iteratively updates the actor states via a message passing process. Inspired by Gaussian belief propagation, we design the messages to be spatially-transformed parameters of the output distributions from neighboring agents. Our model is fully differentiable, thus enabling end-to-end training. Importantly, our probabilistic predictions can model uncertainty at the trajectory level. We demonstrate the effectiveness of our approach by achieving significant improvements over the state-of-the-art on two real-world self-driving datasets: ATG4D and nuScenes.

I. INTRODUCTION

Relational behavior forecasting is central to safe autonomous driving because human maneuvers depend strongly on interactions with neighboring drivers. SPAGNN addresses this gap with a differentiable probabilistic model that jointly handles detection and forecasting, improving results across two datasets.

Motivation: Human drivers perform diverse maneuvers shaped by interactions with neighboring drivers, making relational behavior forecasting important for autonomous vehicles.The paper identifies detecting vehicles and forecasting their long-term behavior in realistic environments as core requirements for safe motion planning.
System overview: SPAGNN takes LiDAR point clouds and HD maps as inputs and outputs detections together with socially coherent probabilistic motion forecasts.The figure distinguishes predicted detections and forecasts from gray ground-truth boxes and future waypoints.
Gap: Existing approaches using detected actors or joint perception and forecasting do not explicitly model social interactions between agents.The limitation applies to methods that use CNNs for trajectories and to joint detection-and-forecasting systems.
Approach: SPAGNN uses graph neural networks to capture vehicle interactions in a probabilistic model, while full differentiability enables joint optimization of detection and behavior forecasting.The joint formulation is intended to mitigate propagation of early detection errors.
Results: SPAGNN shows significant improvements over the state of the art across detection, motion forecasting, and interaction metrics on ATG4D and NUSCENES.ATG4D contains over a million frames, and NUSCENES provides a second real-world autonomous-driving evaluation setting.

II. RELATED WORK

Prior work covers point-cloud detection, motion forecasting, interaction modeling, and joint perception-prediction systems. However, existing joint vehicle forecasting approaches do not explicitly model multi-agent interactions, motivating SPAGNN.

Object Detection from Point Clouds: Point-cloud object detectors use voxel, front-view, or direct point-cloud representations, each with stated computational, geometric, or sparsity trade-offs.Voxel methods can be redundant for sparse point clouds, while front-view projections lose original metric-space information and complicate object-size variation.
Motion Forecasting: Motion forecasting methods include recurrent variational, flow-based, insertion-area, and lane-graph raster approaches.DESIRE and R2P2 require expensive sampling to cover possible outcomes, limiting their suitability for time-critical applications.
Interaction Modeling: Interaction modeling has used game theory, social pooling, and graph neural networks to represent relational behavior.The related work positions GNNs as effective tools for modeling interplay among components or agents.
Joint Perception and Behavior Forecasting: FaF and IntentNet jointly perform perception and future behavior prediction, with later extensions adding intentions or cost maps.These systems exploit LiDAR or HD-map information but do not explicitly model multi-agent interactions.

III. OBJECT DETECTION FROM LIDAR AND HD MAPS

The object-detection stage combines voxelized LiDAR and semantically encoded HD maps in a two-stream CNN built on PIXOR. It produces candidate detections through confidence and bounding-box prediction followed by NMS.

Input parametrization: The model voxelizes LiDAR using HD-map ground height and stacks ego-motion-compensated past sweeps to incorporate motion information.The voxelization reduces variance in vehicle Z coordinates and supports learning height priors.
Input parametrization: The HD-map raster encodes roads, lanes, intersections, crossings, signs, and traffic lights in 17 separate binary semantic channels.Separate channels avoid predefining orderings and ease CNN learning.
Object detection network: The detector extends PIXOR into LiDAR and HD-map streams, processes them with condensed backbones, and fuses their features with a header convolutional network.The LiDAR stream uses the condensed backbone, while the HD-map stream halves the number of filters for efficiency.
Inference pipeline: Inference comprises an object-detection stage followed by a relational behavior-forecasting stage.The detection stage is shown in the top row of the inference diagram.
Object detection network: Two convolutional layers predict confidence scores and bounding boxes at anchor locations, which are reduced to final candidates using nonmaximum suppression.The modifications are designed to produce a detector that is both high-performing and fast.

IV. RELATIONAL BEHAVIOR FORECASTING

The relational forecasting stage predicts future vehicle states with a probabilistic graph model over detected actors. It draws on Gaussian belief propagation while using a spatially aware GNN to handle interactions and variable scene cardinality.

Relational formulation: Each actor state contains 2D future waypoints and heading angles, and the model predicts future states probabilistically from LiDAR and HD-map scene input.The number of actors varies by scene, and the relational model supports any cardinality.
Relational formulation: SPAGNN uses a fully connected directed graph with bidirectional relationships so the model can represent asymmetric interactions such as vehicle following.The graph is practical because scenes typically contain fewer than a hundred vehicles.
Model design: The method is inspired by Gaussian Markov random fields and Gaussian belief propagation, but introduces a graph-neural formulation for the forecasting task.The Gaussian formulation provides design inspiration rather than a direct model of all states and updates.

A. Gaussian MRFs and Gaussian Belief Propagation

The paper formulates future actor states with a Gaussian Markov random field and applies Gaussian belief propagation to compute marginal distributions through iterative message passing.

Assumption: The formulation assumes future states are conditionally independent across future time steps, leaving temporal dependency modeling for future work.This assumption is made conditioned on the observed input and detection output.
Gaussian MRF: Future states are modeled as a multivariate Gaussian conditioned on the observed input and detection output.The joint distribution uses a concatenated state vector s with model parameters A and b.
Gaussian MRF: The Gaussian MRF decomposes the joint probability according to the interaction graph into unary and pairwise potentials.The potential parameters depend on the input and can be designed for the application.
Gaussian belief propagation: Gaussian belief propagation computes each actor’s marginal distribution by iteratively updating message means and precision matrices.Messages from node i to node j are represented by their mean µij and precision matrix Pij.
Gaussian belief propagation: After message passing converges, the model computes each node’s marginal mean and precision, yielding p(si|Ω) = N(si|µi, P_i^-1).The neighborhood excludes the destination node when forming intermediate message updates.

B. Spatially-Aware Graph Neural Networks (SPAGNN)

SPAGNN generalizes graph-based message passing for relational trajectory prediction by combining actor-specific states, spatial coordinate transformations, and probabilistic output distributions.

Model formulation: The Gaussian MRF inspiration is limited because heading angles have bounded support and non-Gaussian belief-propagation integrals are generally intractable.SPAGNN uses the inspiration without retaining those Gaussian MRF limitations as exact inference requirements.
Model formulation: SPAGNN treats each actor as a graph node and uses neural message passing to update hidden and output states over the interaction graph.The model is motivated by the correspondence between Gaussian belief propagation and GNN message updates.
Node state: Initial hidden states are extracted from rotated RoI features, while a 2-layer MLP independently predicts each actor’s initial output state.A down-sampling convolutional network and max pooling reduce each RoI feature map before state construction.
Probabilistic state: The output state stores statistics of waypoint Gaussian distributions and angle Von Mises distributions at each propagation step.This representation supports probabilistic predictions for both spatial position and heading angle.
Message passing: For each directed edge, a shared message function transforms a neighbor’s state into the receiving actor’s local coordinate system before computing the message.The transformation uses the detected bounding boxes of the source and receiving actors, making spatial relationships explicit.
State update: Incoming messages are aggregated with an ordering-invariant feature-wise max operator, and a GRU-based update produces the next node state.The update combines the aggregated message with the node’s previous state through learned functions.
Inference: The message-passing procedure is unrolled for K steps, with the final actor predictions taken from the output states at step K.K is treated as a hyperparameter, and the neural updates generalize the Gaussian belief-propagation form without guaranteeing exact convergence.

V. END-TO-END LEARNING

The full detection and relational prediction model is trained jointly end-to-end by back-propagation using a multitask objective that combines detection and probabilistic forecasting losses.

Objective: Joint training combines binary cross-entropy classification, bounding-box regression, and negative log-likelihood trajectory prediction losses.Hard negative mining is applied to the classification loss.
Objective: The trajectory negative log-likelihood includes separate terms for 2D Gaussian waypoints and Von Mises angles.The Von Mises term uses the modified Bessel function of order 0.
Optimization: Back-propagation through time passes gradients through the GNN message-passing process to the detection backbone.This connects detection learning with relational trajectory prediction during end-to-end optimization.

VI. EXPERIMENTAL EVALUATION

The evaluation compares SPAGNN with detection, forecasting, and interaction baselines on ATG4D and NUSCENES, using detection, collision, trajectory, and heading metrics. Results show improved detection and interaction understanding, strong long-term forecasting gains, and benefits from spatially aware graph designs.

Datasets and metrics: Experiments evaluate ATG4D and NUSCENES across detection, motion forecasting, and social interaction metrics.ATG4D contains over 1 million frames from 5,500 scenarios, while NUSCENES contains 1,000 snippets collected in Boston and Singapore.
Comparison against the state of the art: SPAGNN substantially improves collision rate and centroid error on both datasets, while achieving the lowest heading error on ATG4D and results on par with the best NUSCENES baseline.Comparisons use 80% recall at IoU 0.5 for ATG4D and 60% recall for NUSCENES.
Ablation study: Rotated RoI Align outperforms simple feature indexing in the non-interaction ablation.Its pooled features cover a region spanning 25 meters, providing more surrounding context than indexing a single anchor location.
Graph neural network architectures: Adding spatial awareness improves the graph model: relative bounding boxes outperform global coordinates, and trajectory-distribution messages recover the full SPAGNN design.Models become more certain, with lower entropy, as spatial-awareness mechanisms are added.
Qualitative results: In a crowded ATG4D example, all baselines predict a collision between two vehicles, illustrating the challenge of modeling agent-agent interactions.The visualization encodes uncertain future trajectories with bivariate Gaussian distributions, while IntentNet and NMP provide waypoint trajectories without uncertainty.

VII. CONCLUSION

The paper addresses joint detection and relational behavior forecasting with a spatially aware graph neural network that produces socially coherent probabilistic trajectories. It reports significant improvements over the state of the art on ATG4D and NUSCENES, while identifying future extensions to additional outcomes, sensors, and agent types.

Conclusion: The paper combines detection and relational behavior forecasting in a single model.Its spatially aware graph neural network produces socially coherent, probabilistic future trajectories.
Conclusion: Future work includes generating multiple scene outcomes, incorporating images and radar, and reasoning about pedestrians and cyclists.These extensions are stated as planned directions beyond the current vehicle-focused setup.
Conclusion: The full inference algorithm is intended to run every 0.1 seconds when a new LiDAR sweep is gathered.The implementation is fully vectorized because inference time is critical for onboard autonomy.
Conclusion: The model predicts all future time steps feed-forward with an MLP rather than using a recurrent trajectory decoder.The authors report no gain from explicitly modeling future time dependencies recurrently and adopt the simpler formulation.

B. Implementation details

The implementation combines LiDAR and map processing for detection with per-actor feature extraction and iterative graph-based relational forecasting. Training also addresses detection–ground-truth mismatch and compares against tracking-based, end-to-end adapted, and joint perception-prediction baselines.

Detection network: SPAGNN processes LiDAR and high-definition maps through separate CNN streams built on a condensed PIXOR-style detector.The LiDAR and map backbones use residual blocks, while their multi-resolution features feed a header network.
Detection network: The detector retains high recall by accepting cars observed in either the current sweep or two different previous sweeps.Final detections are selected from the top 200 anchors using NMS with an IoU threshold of 0.1.
Per-actor feature extraction: Per-actor features are extracted with rotated ROI alignment over a heading-aligned 41 by 25 meter region and reduced to initial hidden and output states.A three-layer CNN, spatial max pooling, and a two-layer MLP produce the actor states.
Relational behavior forecasting: Relational forecasting uses K = 3 graph propagation steps with shared edge, aggregation, update, and output functions to iteratively refine probabilistic trajectories.The edge function is a three-layer MLP operating on actor hidden states, while message aggregation and state updates follow the graph structure.
Scheduled sampling: Scheduled sampling mitigates the mismatch between ground-truth and detected bounding boxes during end-to-end training.The model detects vehicles before forecasting their motion for the future 3 seconds, with state updates depending on neighboring messages.
Baselines: Evaluation compares SPAGNN with tracking-based, end-to-end adapted, and joint perception-and-prediction baselines.Tracking-based baselines use reconstructed trajectories, whereas adapted baselines share the detector and feature extractor with alternative interaction operators and decoders.

D. Additional qualitative results

Qualitative results show SPAGNN producing detections and probabilistic motion forecasts across varied ATG4D scenes and generalizing to nuScenes. The examples also expose multimodal forecasting failures and less reliable detection with lower-beam LiDAR.

ATG4D: ATG4D examples cover varied actor densities, lane-graph topologies, interactions, and high-level actions while showing accurate detections and motion forecasts.Examples include illegal U-turns and cars pulling out into non-mapped driveways.
Failure modes: In multimodal situations, SPAGNN can predict a plausible trajectory while the ground truth follows another mode.These examples appear in the last row of the qualitative results as a main failure mode.
nuScenes: nuScenes qualitative results indicate dataset generalization, but detection is less reliable than in ATG4D with its 32-beam rather than 64-beam LiDAR.The lower-beam sensor is identified as the main reason for the reduced detection reliability.
Visualization: The visualizations encode forecasts as one-standard-deviation bivariate-Gaussian ellipses colored from blue at 0 seconds to pink at 3 seconds.Blue boxes denote detections, gray boxes and waypoints denote ground truth, and dashed gray boxes indicate occlusion.