PaperlayerView exact revision on arXiv

Source-linked AI summary

PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving

Thibault Buhet, Emilie Wirbel, Andrei Bursuc, Xavier Perrotton

arXiv:2003.08744v3cs.CVcs.AIcs.RO

TL;DR

Urban driving requires anticipating other road users despite multiple plausible actions and uncertain futures. PLOP uses onboard camera and Lidar inputs in a probabilistic, polynomial trajectory-prediction framework for ego vehicles and neighbors. The authors report improved open-loop state-of-the-art performance on nuScenes and evaluate closed-loop driving in simulation.

  • Problem

    Urban driving requires safe anticipation of other road users, but scenes can be uncertain and allow multiple plausible future paths for each agent.

    Source: §1 · pp. 1–2

  • Method

    PLOP uses onboard camera and Lidar data with a probabilistic Mixture Density Network and polynomial constraints to predict multiple trajectories for ego vehicles and neighbors.

    Source: §1 · p. 2

  • Results

    PLOP improves open-loop state-of-the-art performance on nuScenes and is evaluated in simulated closed-loop driving.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The authors present the simulated closed-loop evaluation as a step toward real-vehicle online application.

    Source: §6 · p. 8

    The architecture does not yet include object detections and still needs work on other road users and weather-related domain shift.

    Source: §6 · p. 8

Abstract

from arXiv · show

To navigate safely in urban environments, an autonomous vehicle (ego vehicle) must understand and anticipate its surroundings, in particular the behavior and intents of other road users (neighbors). Most of the times, multiple decision choices are acceptable for all road users (e.g., turn right or left, or different ways of avoiding an obstacle), leading to a highly uncertain and multi-modal decision space. We focus here on predicting multiple feasible future trajectories for both ego vehicle and neighbors through a probabilistic framework. We rely on a conditional imitation learning algorithm, conditioned by a navigation command for the ego vehicle (e.g., "turn right"). Our model processes ego vehicle front-facing camera images and bird-eye view grid, computed from Lidar point clouds, with detections of past and present objects, in order to generate multiple trajectories for both ego vehicle and its neighbors. Our approach is computationally efficient and relies only on on-board sensors. We evaluate our method offline on the publicly available dataset nuScenes, achieving state-of-the-art performance, investigate the impact of our architecture choices on online simulated experiments and show preliminary insights for real vehicle control

1 Introduction

Urban driving requires an ego vehicle to anticipate other road users amid difficult scenes and multiple plausible actions. PLOP predicts multiple trajectories for the ego and neighbors from onboard sensing, using a probabilistic polynomial formulation.

1 Introduction

  • Urban driving is challenging because road topology, unusual circumstances, and uncontrolled road users can complicate safe interaction.The ego vehicle also follows a destination or high-level goal such as lane keeping or turning.Source: §1 · p. 1
  • Trajectory prediction estimates future positions of all scene agents over a fixed period and must represent diverse paths and uncertainty.The paper describes these challenges in settings where traffic can have multiple possible outcomes.Source: §1 · p. 2
  • PLOP predicts multiple plausible trajectories for the ego vehicle and neighbors using Mixture Density Networks with polynomial constraints.The approach uses front-camera and Lidar data, plus nearby-vehicle detection tracks; auxiliary semantic segmentation can further improve prediction accuracy by injecting scene-layout information.Source: §1 · p. 2
  • PLOP uses onboard sensors without requiring an HD map, and its polynomial trajectory formulation is intended to produce coherent, smooth predictions while mitigating MDN training instability and mode collapse.The paper also reports a comparison showing improvement over PRECOG on nuScenes, and studies closed-loop performance in simulation with preliminary insights for real vehicle control.Source: §1 · p. 2

2 Related Work

Related work spans behavioral cloning, interaction modeling, and probabilistic approaches to multiple futures, with different assumptions about scene inputs and prediction targets. PLOP positions itself as an onboard-sensor, end-to-end imitation-learning approach that predicts trajectory coefficients in one step without anchors or an RNN decoder.

2 Related Work

  • Behavioral cloning trains on expert samples, but datasets offering multiple possible outcomes are scarce, apart from a recent synthetic pedestrian dataset.Many existing approaches assume tracked past positions are known, often by accumulating past object detections.Source: §2 · p. 2
  • Pedestrian prediction often models agent interactions with graph structures, while this paper focuses on vehicle trajectory prediction.Examples cited include Trajectron and Social-BiGAT.Source: §2 · p. 2
  • Vehicle prediction distinguishes the controllable ego trajectory from neighbor trajectories, whose goals are unavailable and whose behavior cannot be controlled.Offline benchmarks such as Argoverse and nuScenes use global metrics that do not distinguish critical from non-critical scenarios.Source: §2 · p. 3
  • Related vehicle-prediction approaches vary in scene representation, from high-level maps and dynamic context to trajectory anchors or Gaussian modes constrained by maneuvers.PLOP instead also uses low-level camera and Lidar data.Source: §2 · p. 3
  • Existing vehicle-prediction work includes probabilistic methods that generate multiple futures for ego vehicles, neighbors, or both, including PRECOG's modeled agent interactions and goal-conditioned ego prediction.Other approaches generate and classify candidate trajectories, retrieve futures from stored similar trajectories, or combine expert policies.Source: §2 · p. 3
  • PLOP uses onboard raw sensors, can be trained end-to-end from imitation learning, and predicts polynomial trajectory coefficients in one step without anchors or an RNN decoder.The paper contrasts this with approaches relying on high-precision maps and GPS positioning; it says polynomial coefficients avoid dependence on anchors whose quality can vary across datasets.Source: §2 · p. 3

3 Network Structure

PLOP uses a multi-input, multi-output network to predict probabilistic future trajectories for the ego vehicle and its neighbors. It combines past trajectories, camera imagery, a Lidar-derived bird’s-eye view, and an ego navigation command.

3.1 Inputs: Past Trajectories, Camera and Lidar

  • The BEV combines vehicle detections and Lidar points in 1 m cells, while the model also uses 2 s of past trajectories, a frontal RGB camera, and an ego navigation command.The BEV is accumulated over 20 frames; commands are follow, left, straight, or right.Source: Figure 2 · p. 3, §3.1 · pp. 3–4

3.2 Network Architecture

  • The encoder synthesizes input information and the predictor processes it; BEV implicitly encodes vehicle interactions and supports an arbitrary neighbor count.Training fixes the maximum number of neighbors at 10 for implementation constraints.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Network Outputs and Training

  • For each vehicle, the network predicts K probabilistic trajectories, using shared mixture weights and degree-4 time polynomials to constrain trajectory means.Ego predictions are conditioned on the navigation command; forecasts span 4 seconds, sampled at 10 Hz.Source: §3.3 · pp. 4–5
  • An auxiliary segmentation module predicts seven scene classes to encourage image features to encode road position, traffic rules, and vulnerable road users.The datasets lack both segmentation and tracking annotations, so training mixes samples for either objective.Source: §3.3 · p. 5
  • Training minimizes negative log-likelihood over sampled ground-truth ego and neighbor positions, with the lateral-coordinate loss weighted by α=3.The auxiliary segmentation task uses cross-entropy loss.Source: §3.3 · p. 5

4 Experiments on Offline Data

The offline evaluation compares trajectory prediction using best-of-K displacement metrics, then assesses PLOP against published methods and examines metric interpretation and mixture size. PLOP’s relative performance improves as more agents are considered, while adding mixture components helps with diminishing returns.

4.1 Metrics

  • minMSD emphasizes high errors, whereas minADE is neutral to error magnitude; both metrics select the best predicted trajectory regardless of its confidence.This avoids penalizing valid alternative futures that differ from the recorded path.Source: §4.1 · p. 6

4.2 Comparison to State of the Art

  • The nuScenes trajectory-prediction evaluation uses 850 scenes, while A2D2 frames supply auxiliary semantic-segmentation training data; results are evaluated only on nuScenes.The segmentation task is intended to inject scene-layout awareness, rather than optimize segmentation accuracy.Source: §4.2 · p. 6
  • With 5 or more agents, PLOP outperforms the compared approaches by a large margin on minMSD, while it slightly outperforms them for up to 4 agents.For ego-only prediction, PRECOG significantly outperforms PLOP; PLOP performs as well as ESP despite ESP having the same goal information as PRECOG.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Finer analysis on metrics and architecture

  • Low minADE paired with high minMSD tends to indicate many correct trajectories alongside a few very incorrect ones; the reverse pattern tends to indicate many moderately correct trajectories.Small minADE variations can produce larger minMSD variations because minMSD squares the error.Source: §4.3 · p. 7, Table 2 · p. 7
  • As the number of predicted trajectories K increases, metric performance improves, but the contribution diminishes for K ≥6.The architecture keeps K under 12 to limit output-layer parameters; results so far use K =12 for fair comparison with other methods.Source: §4.3 · p. 7

5 Closed Loop on Real Vehicle

The closed-loop evaluation uses an internal dataset and simulator, with preliminary real-vehicle testing. It reports online error measures and runtime, while offline minADE alone does not distinguish online behavior adequately.

5.1 Setup

  • The internal dataset covers busy urban driving and a test track, using GPS ground truth for ego trajectories and detector-tracker outputs for neighbors; the prediction horizon is 2s.The neighbor tracks are not manually annotated and therefore contain noise.Source: §5.1 · p. 7, Table 4 · p. 7

5.2 Evaluation

  • Closed-loop evaluation uses simulated lateral, high-speed, and low-speed error counts; offline minADE is not discriminating enough for online behavior.The simulator results cover 8km of busy urban driving and 5km of test track recordings; Figure 5 maps error locations, where braking can stack red dots.Source: §5.2 · pp. 7–8, Figure 5 · p. 8, Table 5 · p. 8

5.3 Runtime and Optimization

  • On the embedded platform, PLOP runs at 13–15 FPS with all outputs and 22–25 FPS when producing only ego and neighbor trajectories.The implementation uses TensorFlow and TF-TRT optimization.Source: §5.3 · p. 8

6 CONCLUSIONS

PLOP combines camera and Lidar inputs with reparameterized Mixture Density Networks and auxiliary semantic segmentation for multi-agent trajectory prediction. The paper identifies missing object detections, road-user coverage, and weather-related domain shift as directions for further work.

6 CONCLUSIONS

  • PLOP uses frontal camera and Lidar inputs to predict multiple trajectories with reparameterized Mixture Density Networks and auxiliary semantic segmentation.The paper reports improved open-loop state-of-the-art performance on nuScenes and a simulated closed-loop evaluation.Source: §6 · p. 8
  • Future work includes incorporating object detections, adding pedestrians and cyclists, and addressing generalization under weather-related domain shift.The authors frame these as prospective improvements, not demonstrated capabilities.Source: §6 · p. 8

Supplementary material: active testing

The supplementary material describes the test vehicles and recorded driving scenarios, then reports preliminary test-track successes alongside a weather-related domain-shift issue. Planned urban testing was delayed by the pandemic.

Test vehicle

  • Two Range Rover Evoque test vehicles use a frontal fisheye camera, 360-degree Lidar array, and high-precision GPS and IMU for past positions.The predicted ego trajectory is converted into steering and acceleration commands by targeting a point 1s ahead, with a dSpace microAutobox and Paravan actuator.Source: Test vehicle · p. 9

Dataset

  • The recorded dataset includes about 100km of complex urban driving and 43km on a closed urban test track with scenarios such as yields, orange lights, and roundabouts.Recordings were collected over one month during summer.Source: Dataset · p. 9

Ongoing active testing

  • Two days of test-track online testing had encouraging results, including lane changes, sharp turns, stop-and-go, and roundabout management, but weather change caused domain-shift issues.Urban online testing was delayed by the pandemic.Source: Ongoing active testing · p. 9

Detailed network architecture

Figure 8 details PLOP’s network architecture and highlights a minimum path for predicting the ego trajectory. The image encoder backbone is not fixed to VGG16.

Detailed network architecture

  • Figure 8 highlights the minimum ego-prediction path, which uses neither semantic segmentation nor neighbor-trajectory inputs.The architecture does not require a specific image encoder backbone; VGG16 could be replaced by ResNet.Source: Detailed network architecture · p. 10, Figure 8 · p. 10

Detailed results on nuScenes

The nuScenes analysis examines additional endpoint-error metrics, offline ablations, and a 6-second prediction horizon. PLOP improves on the compared methods in minFDE at 6 seconds, while results also show trade-offs across metrics and design choices.

Additional metrics

  • The study evaluates confidence-selected FDE and weightFDE alongside minFDE, which measures endpoint accuracy regardless of the intermediate path.Confidence-selected metrics use the highest-weight trajectory; weightFDE checks whether higher-weight components also have higher endpoint error, while comparison with confFDE contrasts other components with the most confident one.Source: Additional metrics · p. 11

Finer analysis on offline metrics

  • Figure 9 shows that minFDE is better by definition, while the neighbor gap between minFDE and confidence-based or weighted FDE persists to 7 m.The authors interpret the persistent gap as evidence of varied predicted neighbor behaviors; it is not observed for ego, which is conditioned on a navigation command.Source: Finer analysis on offline metrics · p. 11, Figure 9 · p. 12
  • Figure 10 shows FDE is significantly lower on the lateral than longitudinal axis, although lateral error is more critical for control and prediction.The axis distinction matters because the same-sized error can have different significance depending on whether it is longitudinal or lateral.Source: Finer analysis on offline metrics · p. 11, Additional metrics · p. 11, Figure 10 · p. 12

Ablation studies on offline metrics

  • Removing Lidar degrades all metrics, especially for neighbors; removing the semantic auxiliary loss worsens ego metrics and increases training instability.The authors attribute the Lidar result to loss of static-scene free-space information.Source: Ablation studies on offline metrics · p. 12
  • Predicting only one trajectory with K = 1 yields very poor results, while adding trajectory components improves minFDE with diminishing gains.The diminishing impact is steeper on the lateral axis, which the authors suggest may reflect greater lateral variability at intersections.Source: Ablation studies on offline metrics · pp. 12–13, Figure 11 · p. 13
  • Removing the camera performs less poorly than expected for ego planning, but the authors caution that testing may not penalize reactive behavior enough.The camera ablation removes useful scene information such as road markings, traffic-light states, and signs.Source: Ablation studies on offline metrics · p. 12

Longer time horizon

  • PLOP improves over cxx and MHA-JAM in minFDE and is competitive in miss rate, but has worse minADE at 6 seconds.The comparison uses the nuScenes prediction benchmark; a degree-4 polynomial fit has mean ADE around 0.6 m at 6 seconds versus around 0.2 m at 4 seconds.Source: Longer time horizon · pp. 13–14, Table 7 · p. 14

Additional results on online simulation

The online-simulation analysis details the real-vehicle image input and examines closed-loop failures. It reports lateral-error cases without semantic-segmentation supervision and speed-related failures in several settings.

Details on image input

  • Fisheye images span 195 degrees of view and are cylindrically reprojected to 150 degrees to compensate for distortion; effective detection range is around 20–30 m.Figure 12 also shows ground-plane projection lines at 1 m spacing.Source: Details on image input · p. 15, Figure 12 · p. 15
  • The fisheye camera enables generating deviations from expert trajectories for imitation-learning training, but its limited range motivates Lidar as a complement.Lidar deviations are generated by changing the point-cloud reference frame.Source: Details on image input · p. 15

Closer analysis of closed loop failure cases

Related papers

Loading 2003.08744v3…