PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Contextual Perception and Prediction with Interaction Transformer

Lingyun Luke Li, Bin Yang, Ming Liang, Wenyuan Zeng, Mengye Ren, Sean Segal, Raquel Urtasun

arXiv:2008.05927v1cs.CVcs.RO

TL;DR

Serial forecasting pipelines can produce inaccurate trajectories, while raster-based forecasting makes real-time inference difficult. This paper combines multi-sensor detection with recurrent Transformer-style interaction reasoning, outperforming the state of the art in detection, trajectory accuracy, and collision rate on ATG4D and nuScenes.

  • Problem

    Traditional serial pipelines often yield inaccurate trajectories, and raster-based forecasting makes real-time inference difficult while limiting recovery from detection errors.

    Source: §I · p. 1

  • Method

    The model performs end-to-end multi-sensor detection and recurrent motion forecasting, using Transformer-inspired reasoning about actors’ relative spatial relationships.

    Source: §I · p. 1, 2) Our Interaction Transformer · p. 4

  • Results

    The model outperforms the state of the art in detection, trajectory accuracy, and collision rate on ATG4D and nuScenes.

    Source: §I · p. 2

  • Takeaways & Limitations

    The approach improves social compliance between predicted trajectories, with fewer collisions between predicted actors.

    Source: §I · p. 2

Abstract

from arXiv · show

In this paper, we tackle the problem of detecting objects in 3D and forecasting their future motion in the context of self-driving. Towards this goal, we design a novel approach that explicitly takes into account the interactions between actors. To capture their spatial-temporal dependencies, we propose a recurrent neural network with a novel Transformer architecture, which we call the Interaction Transformer. Importantly, our model can be trained end-to-end, and runs in real-time. We validate our approach on two challenging real-world datasets: ATG4D and nuScenes. We show that our approach can outperform the state-of-the-art on both datasets. In particular, we significantly improve the social compliance between the estimated future trajectories, resulting in far fewer collisions between the predicted actors.

I. INTRODUCTION

The paper targets inaccurate, hard-to-run motion forecasting and the missing actor interactions in prior end-to-end systems. It presents an end-to-end, recurrent interaction model and reports gains in detection, trajectory accuracy, and collision rate.

I. INTRODUCTION

  • Traditional serial pipelines use filters, dynamical models, or lane goals to estimate future motion, but their limited inputs often yield inaccurate trajectories.Objects are detected and associated across frames before future positions and velocities are estimated.Source: §I · p. 1

II. RELATED WORK

  • Raster-based forecasting approaches do not share computation with detection, making real-time inference difficult and leaving their predictions unable to recover information lost during detection.These approaches use bird’s-eye-view rasters of maps and past and current detections.Source: §I · p. 1
  • Prior end-to-end models share computation but predict actors independently, while several interaction methods rely on past ground-truth trajectories unavailable to self-driving systems.The paper therefore addresses joint detection and forecasting from raw sensor data.Source: §I · p. 1

I. INTRODUCTION

  • The model combines a multi-sensor detector with recurrent, Transformer-inspired actor interaction reasoning, and reports improvements over the state of the art in detection, trajectory accuracy, and collision rate.It is evaluated on ATG4D and nuScenes and runs at 10 FPS on a 1080Ti GPU.Source: §I · pp. 1–2

1) 3D Object Detection:

The related work covers 3D detection, trajectory prediction, joint perception and prediction, and interaction modeling. The paper places its approach in the gap between end-to-end systems that omit actor interactions and interaction methods that use ground-truth trajectories.

1) 3D Object Detection:

  • Multi-sensor 3D detectors and HD-map priors are reported to outperform single-sensor approaches.Detection methods use voxel, bird’s-eye-view, range-view, and point-wise representations.Source: 1) 3D Object Detection · p. 2

2) Trajectory Prediction:

  • Data-driven trajectory predictors use LSTMs or rasterized vehicle states and maps, but raster-based methods do not share computation with perception.Dynamic models can be too simple for complex longer-horizon prediction.Source: 2) Trajectory Prediction · p. 2

3) Joint Perception and Prediction:

  • Existing end-to-end systems train perception and forecasting from raw sensor data with shared features, but do not explicitly model actor interactions.Some also estimate semantic intentions or jointly estimate motion planning.Source: 3) Joint Perception and Prediction · p. 2

4) Interaction Modeling:

  • Interaction modeling includes game theory, graph networks, feature aggregation, and attention, while this work combines spatial and temporal dependencies in a recurrent Transformer-like module.The paper also proposes architecture modifications for this task.Source: 4) Interaction Modeling · p. 2

III. INTERACTION TRANSFORMER

  • The proposed end-to-end model takes raw sensor data, fuses LiDAR, images, and HD maps for detection, then predicts trajectories with a recurrent interactive module.The module processes interactions among actors.Source: §III · p. 2

A. Input and Output Parameterizations

The model represents LiDAR and map inputs in bird’s-eye view, uses oriented boxes for detections and future trajectories, and fuses camera and BEV features for downstream tasks.

1) Input Representation:

  • LiDAR points are voxelized into a BEV occupancy grid, with multiple ego-motion-compensated sweeps and road and lane masks concatenated to preserve motion and map information.Ground height is subtracted before voxelization to remove variation caused by ground slope.Source: 1) Input Representation · p. 2

2) Output Representation:

  • The output space is BEV, with detections represented as oriented boxes and trajectories as sequences of future boxes whose sizes remain constant under a rigid-object assumption.The missing box height can be recovered from the HD-map ground prior.Source: 2) Output Representation · p. 2

1) Backbone Network Architecture:

  • The two-stream backbone fuses LiDAR-and-map BEV features with camera features, back-projecting image features into BEV and interpolating them where LiDAR points are absent.The fused feature map supplies multi-sensor features to detection and prediction.Source: 1) Backbone Network Architecture · pp. 2–3

2) Dense BEV Object Detection:

The section describes anchor-free dense BEV detection and a recurrent forecasting model that reasons jointly over actors and time. It also reviews Transformer attention as the basis for the interaction module.

2) Dense BEV Object Detection:

  • Dense BEV detection uses 1 × 1 convolutions to predict confidence, box parameters, and orientation, followed by oriented nonmaximum suppression.The orientation output combines sin 2θ and cos 2θ regression with a classification step to distinguish orientation direction.Source: 2) Dense BEV Object Detection · p. 3

C. Recurrent Interactive Motion Forecasting

  • The recurrent forecasting design jointly models actor interactions and iteratively infers trajectories because actor behavior and each time step’s output depend on other actors and previous outputs.The Interaction Transformer adapts the Transformer to motion forecasting.Source: C. Recurrent Interactive Motion… · p. 3

1) A Review on the Transformer:

  • Transformer attention computes compatibility between queries and keys, uses normalized weights to aggregate values, and applies nonlinear transformations with shortcut connections.The input is a sequence of feature vectors.Source: 1) A Review on the Transformer · pp. 3–4

2) Our Interaction Transformer:

The Interaction Transformer represents actors with BEV and spatial features, then uses pairwise relative geometry to incorporate neighboring actors’ context. Its attention uses sigmoid weights to allow the amount of interaction to vary by target.

2) Our Interaction Transformer:

  • Queries represent actors whose motion is forecast, while keys and values encode contextual information from neighboring actors.Actor representations combine features extracted from the BEV feature map with center location, size, and orientation.Source: 2) Our Interaction Transformer · p. 4
  • Transforming other actors into each target’s local coordinate frame makes interaction learning translation- and rotation-invariant.The frame is centered on the target actor and aligned with its orientation; relative locations and orientations are encoded in R.Source: 2) Our Interaction Transformer · p. 4
  • Pair-specific relative location embeddings are combined with actor features to compute keys and values for each query.Because relative embeddings cannot be fused into a single actor feature, keys and values are three-dimensional matrices.Source: 2) Our Interaction Transformer · p. 4
  • Sigmoid attention replaces softmax because interaction strength can vary: attention may sum low when actors do not interact and high when they interact strongly.Softmax would force the attention scores for other actors to sum to 1 for each target.Source: 2) Our Interaction Transformer · p. 4

3) Recurrent Temporal Prediction:

Motion prediction proceeds autoregressively, with one recurrent step for each predicted time step. Each step uses updated actor features and relative positions to produce the next waypoints.

3) Recurrent Temporal Prediction:

  • The recurrent model predicts motion autoregressively, using as many recurrent steps as there are predicted time steps.At each step, the Interaction Transformer outputs features that feed a two-layer MLP to compute the next waypoints.Source: 3) Recurrent Temporal Prediction · p. 4
  • At time step t, relative positions R(t) come from the previous step’s waypoints; R(0) is computed from detection boxes.The initial actor features are bilinearly interpolated BEV features at the detection-box centers.Source: 3) Recurrent Temporal Prediction · p. 4
  • The output features at each step become the input actor features for the next recurrent step.The Interaction Transformer receives F_in(t) and R(t), outputs F_out(t), and uses F_out(t) as F_in(t+1).Source: 3) Recurrent Temporal Prediction · p. 4

4) Per-Time-Step refinement:

The model refines each time step’s waypoint proposal using updated actor features and spatial relations. Proposal and refinement outputs are computed in parallel for efficiency.

4) Per-Time-Step refinement:

  • At each time step, the Interaction Transformer first predicts a waypoint proposal.The proposal step uses the prior output features.Source: 4) Per-Time-Step refinement · p. 4
  • The updated feature and relative embedding computed from the proposal are fed back into the Interaction Transformer to produce refined waypoints.The refined output encodes more up-to-date spatial relations between actors than the proposal.Source: 4) Per-Time-Step refinement · p. 4
  • Proposal and refined waypoints are computed in parallel in one Interaction Transformer forward pass, using two different MLPs.Both outputs use the parameterization (dx, dy, sin 2θ, cos 2θ, clsθ), with dx and dy relative to the detection-box center.Source: 4) Per-Time-Step refinement · p. 4

D. Learning

Learning is end-to-end, combining detection and motion-forecasting losses. The training procedure also uses hard-negative mining and detection-associated ground truth for forecasting supervision.

D. Learning

  • The fully differentiable model is trained end-to-end by minimizing a weighted sum of detection and prediction losses.The detection loss combines object classification, box regression, and orientation-flip classification; the motion loss combines trajectory and heading regression, orientation-flip classification, and collision loss.Source: D. Learning · p. 4
  • The collision loss penalizes colliding vehicle pairs with negative ℓ2 distance, pushing colliding vehicles apart.This term is part of the motion-forecasting loss.Source: D. Learning · p. 4
  • Online hard-negative mining keeps the most difficult negative detection samples and ignores easy negatives.Classification uses positive and negative samples, while regression uses positive samples only.Source: D. Learning · pp. 4–5
  • For each target actor, the query and keys produce attention weights that aggregate neighboring actors’ values into a contextual feature.A residual block then produces the updated contextual feature for that actor.Source: D. Learning · p. 5
  • Forecasting loss is averaged over detections assigned to ground-truth boxes by maximum oriented IoU, retaining only the best-matching detection when assignments overlap.This prevents the loss from being dominated by ground-truth actors that are not detected.Source: D. Learning · p. 5

IV. EXPERIMENTAL EVALUATION

The evaluation uses ATG4D and nuScenes, with detection, trajectory-accuracy, and collision-rate metrics, under dataset-specific training and evaluation settings.

1) Datasets:

  • ATG4D evaluation uses 5,000 training and 500 evaluation snippets, plus an approximately 800-snippet interactive subset; nuScenes contains 850 scenes.The ATG4D interactive subset focuses on blocking interactions; nuScenes actor positions are labeled at 2 Hz.Source: 1) Datasets · p. 5
  • AP measures detection, ADE and FDE measure trajectory error, and TCR measures future-trajectory collisions using a >0.05 IoU threshold.ADE, FDE, and TCR are evaluated at fixed 70% and 90% recall rates; FDE emphasizes long-term prediction.Source: 2) Evaluation Metrics · p. 5

3) Implementation Details:

  • The model predicts three seconds ahead in six 0.5-second steps, aggregates 0.5 seconds of LiDAR history, and uses dataset-specific evaluation ranges.ATG4D uses five past sweeps and a 100 m range; nuScenes uses ten sweeps and a 50 m range, without images or maps.Source: 3) Implementation Details · pp. 5–6

4) Comparison with State-of-the-Art:

The proposed approach outperforms prior methods on detection, trajectory prediction, and interaction metrics across ATG4D and nuScenes. Its interaction module lowers collision rates, and qualitative examples show vehicles adjusting trajectories around other actors.

4) Comparison with State-of-the-Art:

  • Against NeuralMP on ATG4D, AP gains are 4.8%/6.0% at 0.5/0.7 IoU, with 17.9%/19.3% relative ADE/FDE reductions and 80.8% TCR reduction at 90% recall.Adding collision loss further reduces TCR by 56.3% relative.Source: 4) Comparison with State-of-the-Art · p. 6
  • In qualitative examples, highlighted vehicles in the proposed model react to other actors and adjust their future trajectories to avoid collision.The figure contrasts the interaction model with a baseline lacking interaction.Source: Figure 3 · p. 7
  • Against ConvSocialPool, the method improves ADE/FDE/TCR by 3.8%/3.7%/59.4% on ATG4D and L2@1s/L2@3s/TCR by 4.2%/5.0%/62.7% on nuScenes.The reported comparisons use 90% detection recall on ATG4D and 80% on nuScenes; the nuScenes TCR uses IoU 0.1.Source: 4) Comparison with State-of-the-Art · p. 7
  • The detection model achieves 81.4% mAP on nuScenes car detection, which the authors describe as on par with the state of the art.The nuScenes model uses LiDAR without maps in the SpAGNN comparison.Source: 4) Comparison with State-of-the-Art · p. 7

5) Ablation Studies:

Ablations show that the full model’s architecture improves forecasting over the backbone, while collision loss is most effective with the Interaction Transformer. Qualitative examples illustrate collision-avoiding adjustments, and the conclusion identifies additional actor types and joint motion planning as future directions.

5) Ablation Studies:

  • 6.8%/7.6%/83.1% relative reduction on ADE/FDE/TCR on ATG4D-interact and 4.9%/7.0%/54.6% relative gain on ADE/FDE/TCR on nuScenes versus the backbone network.These gains follow the progression from a two-stage architecture through the Interaction Transformer, recurrent prediction, and additional refinement.Source: 5) Ablation Studies · p. 7
  • Collision loss alone slightly reduces TCR but worsens ATG4D-interact ADE and FDE by 2.0% and 1.3%; with the recurrent Interaction Transformer, TCR falls 55.7% and 86.6% across the two datasets while ADE and FDE are almost unaffected.The authors report that collision loss works better when the network understands spatial relations between actors.Source: 5) Ablation Studies · p. 7
  • Adding relative-location features, reasoning in each actor’s coordinate system at each prediction step, and replacing softmax with sigmoid each significantly improves prediction metrics on both datasets.The softmax attention constraint that scores across actors sum to 1 is removed by the sigmoid modification.Source: 5) Ablation Studies · p. 7

6) Qualitative Results:

  • In the first four scenarios, the proposed model adjusts future motion after sensing a front vehicle, while the baseline’s past-state-like prediction leads to a collision; in the last, it turns more sharply around a parked vehicle.Figure 3 compares detections and estimated trajectories with the interaction actors highlighted in red; neither method uses collision loss.Source: 6) Qualitative Results · pp. 7–8

V. CONCLUSION

  • The conclusion reports improvements over the state of the art on ATG4D and nuScenes and proposes investigating bicyclists and pedestrians, as well as joint motion planning, as future work.Joint motion planning is proposed in a similar spirit to NeuralMP.Source: §V · p. 8

Related papers

Loading 2008.05927v1…