Source-linked AI summary
Pedestrian Prediction by Planning using Deep Neural Networks
Eike Rehder, Florian Wirth, Martin Lauer, Christoph Stiller
TL;DR
Accurate pedestrian prediction is needed for safe autonomous-vehicle maneuvering, while extrapolating observed motion becomes infeasible at long lookahead times. The paper predicts destinations and plans toward them in a jointly trained neural network; its evaluations report destination and trajectory prediction outcomes, including obstacle-avoiding paths.
Problem
Observed-motion extrapolation becomes infeasible at long lookahead times because human motion is highly dynamic, despite the need for accurate prediction to support safe maneuvering.
Source: §1 · p. 1Method
A monolithic neural network infers destination distributions from observations, maps them to a discrete state grid, and plans over generated topology maps to predict positions over time.
Source: §1 · p. 1, §2 · p. 2, §4.3 · p. 5Results
The RMDN represents different motion patterns and its accuracy decreases sub-linearly over prediction time; full prediction certainty declines more slowly than the IMM’s, although its distribution is wider during the first second.
Source: §6.1 · p. 7, §6.2 · p. 7Takeaways & Limitations
The paper presents joint destination forecasting and planning-based prediction using image and map information, and reports predicted paths that can avoid obstacles.
Source: §7 · p. 8, §6.3 · p. 8The system relies on precomputed map features; training exclusively from obstacle and top-view maps is described as a possibility with more training data.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we predict pedestrians by emulating their own motion planning. From online observations, we infer a mixture density function for possible destinations. We use this result as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The entire system is modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.
1 Introduction
Accurate pedestrian prediction matters for collision avoidance, but extrapolating observed motion becomes infeasible at longer lookahead times. This work combines destination inference and planning-based prediction in one neural network.
1 Introduction
- Long-lookahead pedestrian prediction is difficult because human motion is highly dynamic, limiting extrapolation of observed dynamics.Accurate prediction is especially important for vulnerable road users because collisions are likely to be fatal.Source: §1 · p. 1
- Earlier planning-based methods could predict over long periods but required static observation and prelocated destinations, while dynamic-scene adaptations used relatively naive scene and destination models.Source: §1 · p. 1
- The paper proposes one neural network that predicts destinations from observations and plans toward them, with topology generated from destination maps and environment features.It proposes MDP and forward-backward planning architectures and joint training in a monolithic inverse-reinforcement-learning setup.Source: §1 · pp. 1–2, Figure 1 · p. 2
2 Prediction Network Architecture
The prediction network combines destination inference, topology generation, and planning. It outputs a position probability grid for every predicted time step.
2 Prediction Network Architecture
- The network has three stages: destination prediction from pedestrian images and positions, topology generation from destination maps and environment features, and planning on the topology.Source: §2 · p. 2
- A recurrent mixture density network predicts possible destinations as a probability distribution map from pedestrian images and positions.Source: §2 · p. 2
- The topology network produces a map for planning, which may represent a cost, reward, or state probability depending on the planning algorithm.The planning stage uses MDP or forward-backward methods and outputs one position-probability grid per predicted time step.Source: §2 · p. 2
3 Destination Prediction
Destination prediction represents possible destinations as a mixture distribution. A recurrent mixture density network estimates its parameters from pedestrian images and positions.
3 Destination Prediction
- Each mixture component represents a destination alternative, while its density captures uncertainty within that alternative.The model uses Gaussian-von-Mises distributions for position and heading, weighted by mixing coefficients.Source: §3 · p. 2
- The network estimates each component’s position mean and covariance, heading mean and concentration, and mixing weight.The Gaussian position and von-Mises heading distributions combine into the final mixture density.Source: §3 · pp. 2–3
- A recurrent mixture density network processes pedestrian tracks using both images and positions, with eight outputs per mixture component.The recurrent model uses an LSTM for time-series processing.Source: §3 · p. 3
4 Planning Network
The planning network turns destination predictions into trajectories by selecting actions over a discretized state space and propagating pedestrian-state probabilities. It combines an MDP planner or a forward-backward planner with a destination map and an environment-conditioned topology map.
4 Planning Network
- Motion planning selects actions across a discretized state space to reach a goal, then simulates motion under the resulting policy to produce predictions.The grid representation allows motion sequences to be treated as graphs and solved efficiently.Source: §4 · p. 3
4.1 Markov Decision Process Network
- The MDP network uses Value Iteration to find a policy that maximizes expected reward, with reward linked to reaching the destination.The policy selects actions per state; training the policy, transition models, and reward from observed trajectories is inverse reinforcement learning.Source: §4.1 · pp. 3–4, Figure 2 · p. 4
4.2 Forward-Backward Network
- The forward-backward network models suboptimal paths by propagating probabilities from the start toward the future and from the destination toward the past, then combining them.The timestep-wise product yields the predicted trajectory.Source: §4.2 · p. 4, Figure 3 · p. 5
4.3 Destination Mapping
- The continuous destination distribution is discretized into a three-dimensional grid of position and orientation, with each cell representing the likelihood of a destination state.The grid is formed by evaluating the RMDN over mesh grids of position and orientation values.Source: §4.3 · p. 5
4.4 Topology Network
- The topology map connects behavior to locations; an FCN predicts it from the RMDN destination and environmental features for use by the planner.For the MDP it represents state rewards; for the forward-backward method it governs action probabilities.Source: §4.4 · p. 5
5 Network Training Details
Training combines destination prediction and planning, with adjustments to the destination and transition components. For larger prediction horizons, the joint network uses losses from both components and a weighted regularization loss.
5.1 Destination Mixture Density Network
- The destination RMDN is trained on manually annotated pedestrian-track time series, using the minimum per-example loss within each batch and dropout on mixture coefficients to train all mixture components meaningfully.The likelihood loss is evaluated at ground-truth destinations; the minimum batch loss and coefficient dropout are the stated training adjustments.Source: §5.1 · pp. 5–6
5.2 Planning-Networks
- Planning parameters, including action transition filters, are trained jointly, and the planning loss is the negative log likelihood of predictions at ground-truth pedestrian positions.The transition filters are trained without requiring prior knowledge of the selected action.Source: §5.2 · p. 6
5.3 Joint Network
- For larger horizons, the joint network combines the RMDN and planning-network log-likelihood losses with weighted regularization because destination prediction otherwise fails to converge.For small horizons, prediction loss alone is sufficient.Source: §5.3 · p. 6
6 Experiments
Experiments on real-world pedestrian data evaluate destination and trajectory prediction, including comparisons with an IMM filter and between two planning architectures. The example prediction illustrates obstacle avoidance.
6 Experiments
- Environment features are represented in occupancy grids, with camera images segmented into obstacle, road, sidewalk, and top-view maps.Prediction performance is evaluated as probability in a roughly 0.1 m² circle centered on ground truth, with a constant position-constant velocity IMM as reference.Source: §6 · p. 6
6.1 Destination Prediction
- RMDN represents different motion patterns more accurately than IMM and a single-component RDN, with accuracy decreasing sub-linearly rather than linearly over prediction time.The comparison evaluates predicted probability around ground-truth destinations across prediction horizons.Source: §6.1 · pp. 6–7, Figure 5 · p. 7
6 Experiments
- The full network was tested on real-world data using three-second pedestrian tracks to predict three seconds ahead at each time step.The collected dataset contains roughly 400 manually annotated pedestrian tracks; the evaluation uses an independent test set.Source: §6 · p. 6, §6.2 · p. 7
6.2 Trajectory Prediction
- Prediction certainty declines more slowly than IMM overall, although spread is larger in the first second; MDP performs slightly better than Fwd-Bwd, with similar results.The paper attributes the slower decline to more certain destination predictions and corresponding planning, while fixed transition distributions limit motion uncertainty to the cell discretization.Source: §6.2 · p. 7
6.3 Example Prediction
- In the example, predicted action directions tend to avoid obstacles, and the maximum of the predicted path distribution avoids an obstacle.All destination-mixture components coincide at one location in this example.Source: §6.3 · p. 8
7 Conclusion
The work uses predicted destinations as goals for planning-based pedestrian prediction and, to the authors’ knowledge, is the first to solve destination forecasting jointly with planning from image and map information. It relies on precomputed map features, though the authors describe annotation-free map-based training as a possibility with more data.
7 Conclusion
- Planning toward predicted possible destinations can serve as a predictor in a monolithic neural network architecture.Source: §7 · p. 8
- To the authors’ knowledge, this is the first work to solve destination forecasting for planning-based prediction jointly using image and map information.Source: §7 · p. 8
- The system relies on precomputed map features, while annotation-free training from obstacle and top-view maps is presented as a possibility with more training data.Source: §7 · p. 8