PaperlayerView exact revision on arXiv

Source-linked AI summary

On Adversarial Robustness of Trajectory Prediction for Autonomous Vehicles

Qingzhao Zhang, Shengtuo Hu, Jiachen Sun, Qi Alfred Chen, Z. Morley Mao

arXiv:2201.05057v3cs.CVcs.CR

TL;DR

Trajectory prediction is critical to safe AV planning, but its adversarial robustness and worst-case safety have been insufficiently studied. The paper develops physically constrained attacks on normal vehicle trajectories and evaluates them across models and datasets. Attacks increase prediction error by more than 150%, while mitigation methods reduce error under attack, although detection and smoothing introduce false positives and normal-case degradation.

  • Problem

    Trajectory prediction’s adversarial robustness and whether worst-case predictions support safe planning have received little study, despite its importance to AV driving safety.

  • Method

    The paper develops white-box and black-box attacks that perturb normal trajectories under physical constraints, and explores data augmentation and trajectory smoothing as mitigations.

  • Results

    Across 10 model–dataset combinations, adversarial perturbations increase ADE/FDE by 167%/150%, while mitigation methods reduce prediction error under attacks by 28%.

  • Takeaways & Limitations

    Adversarial trajectories can cause inaccurate predictions and unsafe AV behavior, supporting evaluation of worst-case prediction accuracy and robustness mitigation.

  • Takeaways & Limitations

    Mitigation can produce false positives: test-time smoothing reduces adversarial-case error by 12% but raises normal-case error by 6%.

Abstract

from arXiv · show

Trajectory prediction is a critical component for autonomous vehicles (AVs) to perform safe planning and navigation. However, few studies have analyzed the adversarial robustness of trajectory prediction or investigated whether the worst-case prediction can still lead to safe planning. To bridge this gap, we study the adversarial robustness of trajectory prediction models by proposing a new adversarial attack that perturbs normal vehicle trajectories to maximize the prediction error. Our experiments on three models and three datasets show that the adversarial prediction increases the prediction error by more than 150%. Our case studies show that if an adversary drives a vehicle close to the target AV following the adversarial trajectory, the AV may make an inaccurate prediction and even make unsafe driving decisions. We also explore possible mitigation techniques via data augmentation and trajectory smoothing. The implementation is open source at https://github.com/zqzqz/AdvTrajectoryPrediction.

1. Introduction

Trajectory prediction is essential to safe AV planning, yet its adversarial robustness and safety consequences have received little study. The paper introduces physically constrained attacks, evaluates their effects, and explores mitigation.

  • Trajectory prediction forecasts nearby objects’ future paths and directly informs AV planning and driving behavior.
  • Few studies assess trajectory prediction security or robustness against adversarial examples, despite attackers potentially influencing predictions by controlling nearby vehicles.
  • The proposed white-box and black-box attacks add minor perturbations to normal trajectories to maximize prediction error while enforcing physical constraints for naturalness.
  • 150%: adversarial perturbations increase prediction error by around 150% across 10 model–dataset combinations.
  • 62.2%: attacks cause prediction deviation exceeding half a lane width, potentially altering AV navigation decisions.
  • Data augmentation and trajectory smoothing reduce prediction error under adversarial attacks, while case studies demonstrate possible hard brakes and collisions.

2. Background and Related Work

Trajectory prediction models use past observations and auxiliary information to forecast road-agent motion within AV perception pipelines. This work connects such prediction algorithms to adversarial robustness and real-world safety impact.

  • Trajectory prediction models forecast future spatial coordinates of vehicles and pedestrians from recent observed states, sometimes using headings, interactions, dynamics, or semantic maps.
  • Within AVs, trajectory prediction belongs to perception and supplies important inputs to planning, making accurate predictions critical for safe driving.
  • Figure 1 depicts an attack scenario that spoofs lane-changing behavior in trajectory prediction.
  • Prior prediction algorithms did not consider safety impact in real-world applications; this work bridges adversarial robustness with AV systems.

3. Problem Formulation

The paper formulates repeated trajectory prediction over observable road agents, models an adversary controlling a nearby vehicle’s crafted trajectory, and defines metrics for prediction error and directional attack impact.

  • 3.1. Trajectory Prediction: 3.1. Trajectory Prediction: The task repeatedly consumes current and historical states of observable vehicles and pedestrians to predict their future trajectories.
  • 3.1. Trajectory Prediction: 3.1. Trajectory Prediction: An object trajectory is a sequence of states over consecutive time frames, including spatial coordinates and optional features.
  • 3.2. Attack Model: 3.2. Attack Model: The adversary drives one other vehicle along a crafted trajectory that the AV observes and uses for iterative prediction.
  • 3.2. Attack Model: 3.2. Attack Model: The controlled vehicle can appear to change lanes while actually driving straight, potentially inducing braking and rear-end collision hazards.
  • 3.2. Attack Model: 3.2. Attack Model: The attacker may require model parameters or APIs, predicts surrounding-object trajectories, then computes and follows the adversarial trajectory.
  • 3.2. Attack Model: 3.2. Attack Model: Adversarial trajectories must obey bounded physical properties and represent normal rather than ruthless driving behavior.
  • 3.3. Evaluation Metrics: 3.3. Evaluation Metrics: ADE and FDE measure prediction error, while directional deviation metrics capture targeted lateral and longitudinal attack effects.
  • 3.3. Evaluation Metrics: 3.3. Evaluation Metrics: Half the lane width, about 1.85 meters in the used datasets, is treated as a threshold likely to produce different AV decisions.

4. Adversarial Example Generation

The paper generates adversarial trajectories by perturbing observed histories while enforcing physical-feasibility and location-deviation constraints, then maximizes prediction error over selected future frames. It supports both gradient-based white-box and inference-only black-box optimization.

  • Attack design: Adversarial trajectories are created by adding perturbations to history trajectories, with single-frame attacks controlling one prediction and multi-frame attacks maximizing error across L_P predictions.Multi-frame scenarios perturb the first L_I + L_P − 1 frames so predictions from L_I through L_I + L_P − 1 are controlled.
  • Attack design: The attack objective is the negation of average prediction error across the considered future time frames.The objective uses predicted and future trajectories and one of six metrics.
  • Constraint design: Perturbations must preserve physically feasible motion and avoid dangerous driving behaviors through explicit hard constraints.The constraints are enforced by scaling perturbations until the resulting trajectory satisfies all requirements.
  • Constraint design: Physical-property constraints keep velocity, acceleration, and acceleration derivatives within dataset mean ± 3 standard deviations.The stated range covers 99.9% of the dataset under a normal-distribution assumption while excluding outliers.
  • Constraint design: A default 1-meter location-deviation bound limits lane-changing behavior while preserving the original driving pattern and controlling attack stealth.The bound is compared with approximately 3.7-meter urban lanes and 1.7-meter vehicle widths.
  • Optimization: White-box attacks use Projected Gradient Descent, whereas black-box attacks use Particle Swarm Optimization requiring only model-inference API access.In PSO, particles represent perturbation candidates evaluated by the attack objective within the hard-constraint search space.

5. Mitigation Mechanisms

The paper mitigates adversarial effects by injecting constrained perturbations into training data and smoothing trajectories during training and testing. These methods target unstable acceleration and velocity patterns associated with adversarial trajectories.

  • Data augmentation: Data augmentation adds randomly constrained perturbations to selected training trajectories to expose the model to adversarial patterns.The perturbations satisfy the hard constraints defined for adversarial trajectory generation.
  • Train-time trajectory smoothing: Trajectory smoothing is applied to both training and testing data to partially remove unstable velocity or acceleration patterns.The experiments use a simple linear convolution-based smoother.
  • Motivation: Adversarial trajectories frequently change acceleration, a pattern described as rare in normal trajectories.This observed distributional difference motivates both mitigation mechanisms.

6. Experiments

Experiments show that constrained adversarial perturbations substantially degrade trajectory prediction across models and datasets, while robustness varies with scenario, representation, attack setting, and mitigation strategy.

  • General results: Across models and datasets, single-frame white-box attacks increase ADE/FDE by 167%/150%.Lateral/longitude deviation reaches 2.03/3.84 meters on average.
  • Different scenarios: High-acceleration scenarios have higher prediction error, while worst-case error appears independent of traffic density.The models particularly struggle with turning and stopping scenarios, where traffic-rule knowledge may be needed.
  • Different models: Adding map information reduces Trajectron++ ADE by 22% on original trajectories and 31% on perturbed trajectories.Map features reduce reliance on perturbed trajectory features and make predictions less sensitive to attack inputs.
  • Different models: Normal-trajectory accuracy does not necessarily predict adversarial robustness: FQA outperforms Trajectron++ under perturbation on Apolloscape and NGSIM.Trajectron++ has better unperturbed accuracy across all three datasets but is not best under attack.
  • Transferability: 90.25% of transferred adversarial trajectories increase prediction error relative to non-attack cases, although transferability depends strongly on the target model.Transfer to FQA is easier than transfer to Trajectron++, which uses more trajectory features.
  • Mitigation results: Mitigation is model-dependent: train-time augmentation helps Trajectron++, smoothing helps FQA, and combining them reduces attacked-case error by 26% while increasing normal-case error by 11%.Test-time detection plus smoothing reduces adversarial-case error by 12% while increasing normal-case error by 6%.

7. Conclusion

The paper analyzes adversarial robustness in trajectory prediction for autonomous vehicles and finds that realistic perturbations can induce dangerous driving behavior. It motivates worst-case evaluation and proposes mitigation methods, including map information and driving-rule semantics.

  • The study presents the first analysis of adversarial robustness for trajectory prediction in autonomous vehicles.
  • Prediction models are generally vulnerable to adversarial perturbations, which may cause dangerous behaviors such as hard braking.
  • The paper argues for evaluating worst-case prediction accuracy under hard scenarios or adversarial examples.
  • The proposed mitigation methods include data augmentation and trajectory smoothing.
  • The paper suggests leveraging map information and driving-rule semantics to guide trajectory prediction.

A. Implementation Details

The implementation supports adversarial-attack experiments under physical constraints and includes both white-box and black-box optimization procedures. Perturbed trajectories are constrained using five physical properties selected according to dataset distributions.

  • The implementation is an open-source framework for testing adversarial attacks and mitigation methods on trajectory prediction algorithms.
  • Perturbed trajectories are restricted using scalar velocity, longitudinal and lateral acceleration, and derivatives of both accelerations.
  • Physical-property bounds are selected separately for Apolloscape, NGSIM, and nuScenes according to each dataset’s data distribution.
  • White-box and black-box attacks are iteration-based and are tuned by monitoring whether objective loss decreases and stabilizes.
  • The white-box attack uses Adam with learning rate 0.01 and 100 iterations, while the black-box attack uses 10 particles and 100 iterations.

A.4. Implementation of Mitigation

The mitigation implementation augments training data with constrained perturbations and smooths trajectories using neighboring positions. Detection combines normal and perturbed trajectories to identify cases for selective smoothing, while hard scenarios illustrate remaining prediction challenges.

  • Data augmentation randomly perturbs 50% of training trajectories under hard constraints and doubles the maximum training iterations.
  • Trajectory smoothing uses a linear convolution kernel that averages three trajectory positions to produce the middle smoothed position.
  • Detection training combines normal trajectories with randomly perturbed trajectories to fit an RBF-kernel SVM and select thresholds.
  • Hard-scenario visualizations compare ground-truth trajectories with model predictions when vehicles brake and stop at a stop sign or turn right at an intersection.

C. More Case Studies

Additional case studies show that adversarial trajectories can hide real lane changes or destabilize predicted longitudinal behavior. These examples broaden the attack impact beyond spoofing a fake lane change.

  • C. More Case Studies: The appendix adds case studies examining attack impacts under different attack goals.
  • C. More Case Studies: The case studies complement the hard-scenario example of a right turn at an intersection.
  • C. More Case Studies: A rightward deviation can hide an actual lane change, delaying the AV’s recognition by 2.5 seconds and potentially causing hard braking or a rear-end collision.The adversarial trajectory remains smooth and natural, making mitigation ineffective in this case.
  • C. More Case Studies: A rearward deviation can destabilize predicted velocity even when the other vehicle remains on its route.The case involves an other vehicle traveling about 10 meters ahead of the AV.

D. Realistic Attack Setting

The realistic attack setting uses predicted trajectories for other road users because the adversary lacks their ground truth. Across 75 Apolloscape scenarios, this setting retains nearly the effectiveness of attacks using ground-truth trajectories.

  • Because the adversary lacks other vehicles’ and pedestrians’ ground truth, it includes predicted trajectories when generating the adversarial trajectory.
  • Figure 13 presents reproduction of a prediction attack on Baidu Apollo 6.0.
  • The evaluation selects 75 Apolloscape scenarios and attacks the GRIP++ model using predicted trajectories for all other objects between 3–6 seconds.The adversary knows trajectories during the first 3 seconds, then predicts other objects before generating its own trajectory.
  • 6.82/10.41/2.61/2.38/3.92/5.53 meters: prediction error across six metrics for the predicted-trajectory attack.
  • 95%: the predicted-trajectory attack’s effectiveness relative to attacks using ground-truth object trajectories.

E. Attack Real-world AV System

The attack transfers to Baidu Apollo 6.0, where an adversarial trajectory spoofs a lane change and leads the Apollo AV to brake in simulation.

  • The attack is effective against Baidu Apollo 6.0, which uses an LSTM predictor with 2 seconds of trajectory history at 10 Hz.
  • A spoofed lane change in the adversarial trajectory results in braking by the right Apollo AV in the LGSVL simulator.
Loading 2201.05057v3…