Source-linked AI summary
ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planning
Yuxiao Chen, Boris Ivanovic, Marco Pavone
TL;DR
Trajectory prediction for autonomous systems must account for uncertain, multimodal behavior while supporting downstream planning with compatible joint predictions. ScePT predicts interacting agents’ futures using a policy-based approach with explicit scene-consistency regularization. Across real-world pedestrian and driving datasets, it reports state-of-the-art accuracy with significantly improved scene consistency.
Problem
Many trajectory predictors do not enforce scene consistency or generate joint predictions, despite downstream planners needing compatible multimodal futures across agents.
Source: §1 · p. 1, §1.1 · p. 2Method
ScePT predicts futures for cliques of interacting agents using a policy network that autoregressively rolls out closed-loop trajectories, with a collision penalty used as training regularization.
Source: §1.2 · p. 2, §5 · p. 8Results
On ETH, UCY, and nuScenes, ScePT achieves state-of-the-art prediction accuracy with significantly improved scene consistency.
Source: §5 · p. 8Takeaways & Limitations
ScePT demonstrates integration with a downstream contingency planner and generation of human-like behaviors.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Trajectory prediction is a critical functionality of autonomous systems that share environments with uncontrolled agents, one prominent example being self-driving vehicles. Currently, most prediction methods do not enforce scene consistency, i.e., there are a substantial amount of self-collisions between predicted trajectories of different agents in the scene. Moreover, many approaches generate individual trajectory predictions per agent instead of joint trajectory predictions of the whole scene, which makes downstream planning difficult. In this work, we present ScePT, a policy planning-based trajectory prediction model that generates accurate, scene-consistent trajectory predictions suitable for autonomous system motion planning. It explicitly enforces scene consistency and learns an agent interaction policy that can be used for conditional prediction. Experiments on multiple real-world pedestrians and autonomous vehicle datasets show that ScePT} matches current state-of-the-art prediction accuracy with significantly improved scene consistency. We also demonstrate ScePT's ability to work with a downstream contingency planner.
1. Introduction
ScePT targets trajectory predictions that are useful to downstream planning: multimodal futures should remain compatible across agents while keeping joint modes tractable. It predicts interacting agents jointly and reports improved scene consistency alongside strong accuracy on real-world datasets.
1. Introduction
- Because human driving behavior is uncertain and multimodal, accurate predictions of other road users are critical to autonomous-system safety.The paper describes self-driving cars sharing roads with vehicles, pedestrians, and cyclists.Source: §1 · p. 1
1.1. Desiderata
- Downstream planning calls for compatible joint predictions, a limited but representative set of multimodal futures, time consistency, and conditional prediction.Individually predicted modes can combine exponentially across agents, overwhelming a planner; sampling may also make predictions change between time steps.Source: §1.1 · pp. 1–2
1.2. Contributions
- ScePT predicts futures for cliques of interacting agents and uses a policy network to autoregressively roll out closed-loop joint trajectories.A tunable risk measure is used to improve output sample diversity.Source: §1.2 · p. 2
- ScePT reports significantly improved scene consistency and reduced dimensionality for scene-level multimodality on real-world pedestrian and driving datasets.The paper identifies these properties as relevant to simulation, downstream planning, and autonomous-vehicle performance verification.Source: §1.2 · p. 2
2. Related Work
Prior trajectory-prediction work spans hand-structured behavior models, data-driven sequence models, and generative approaches. Related work also addresses agent interactions, scene consistency, and output diversity, but the paper identifies policy-based planning with explicit scene-consistency enforcement as a gap.
2. Related Work
- Early behavior models encoded assumptions about decision-making, while data-driven methods and sequence models learned behavior from data and achieved strong results.The paper notes that early structured models could not scale to complicated scenarios despite extensive tuning.Source: §2 · p. 2
- GNNs represent agent interactions through scene-graph message passing, whereas CNNs rasterize scene information and preserve geometry with typically fixed computation complexity.The paper notes that GNN complexity scales at least linearly with agent count and geometric information may be lost without special structures.Source: §2 · p. 2
- CVAEs are common for multimodal prediction, but continuous latent spaces require sampling that can remove time consistency, while discrete spaces are less expressive and more prone to mode collapse.The paper also describes GANs as another generative-model choice.Source: §2 · p. 2
- Prior methods model interactions or couple decoding in several ways, but the paper identifies a gap in modeling agents as policy planners while explicitly enforcing scene consistency.Some cited approaches use pooling, group-level modes, message passing, fictitious play, or autoregressive decoding; searches for likely joint trajectories but predicts only one mode.Source: §2 · pp. 2–3
- Existing diversity-sampling techniques largely target continuous latent spaces, whereas ScePT introduces a risk-based loss modification for diverse outputs from a discrete latent space.The cited approaches include determinantal point processes [49], farthest-point sampling, and a GNN-based reparameterization trick.Source: §2 · p. 3
3. ScePT
ScePT predicts joint trajectories for interacting agents by partitioning scene graphs into cliques, encoding joint latent modes, and decoding them through a policy network. Its training combines an ELBO objective with collision regularization and CVaR-based diverse sampling.
3.1. Preprocessing
- ScePT partitions interacting agents into bounded-size cliques, reducing the dimensionality of the joint latent space that grows exponentially with clique size.It builds a spatiotemporal graph using agents’ closest future distance as an interaction proxy, then partitions the graph and connects every pair within each clique.Source: §3.1 · p. 3
3.2. Encoder
- The encoder uses LSTMs for agent and edge histories and models a joint discrete latent distribution whose cardinality grows exponentially with clique size.The paper reports that probability mass typically concentrates on fewer than 10 modes.Source: §3.2 · pp. 3–4
3.3. Decoder
- The decoder uses a policy network to autoregressively predict trajectories, combining reference trajectories with attention-pooled neighbor information to generate controls.Predicted states pass through agent dynamics and feed back into the next rollout step.Source: §3.3 · p. 4, Figure 3 · p. 4
3.4. Conditioning via policy learning
- Conditional predictions require no structural change: ScePT fixes conditioned agents’ trajectory rollouts and predicts the remaining agents in the clique.Factors involving conditioned nodes are removed from the Gibbs-distribution factor graph.Source: §3.4 · pp. 4–5
3.5. Training
- Training uses an ELBO objective with a Gaussian-noise 2-norm likelihood and adds a collision penalty to discourage incompatible predictions.Because the latent space grows exponentially with clique size, diversity sampling selects the highest-probability modes plus random modes from the remainder.Source: §3.5 · p. 5
3.6. Mode Collapse and Diverse Sampling
- Discrete CVAEs can collapse modes because weighted 2-norm errors favor similar predictions; ScePT uses CVaR to focus training on the best α-percentile losses.This does not force every mode to match ground truth; it focuses on already-close predictions to prevent mode collapse and maintain output diversity.Source: §3.6 · p. 5
4. Experiments
ScePT is evaluated on pedestrian and vehicle prediction, with near-state-of-the-art accuracy and substantially improved scene consistency. The experiments also test conditional prediction, downstream MPC planning, and design ablations.
4.1. Pedestrian Motion Prediction
- On pedestrian benchmarks, ScePT achieves the best or second-best performance on most datasets, with slightly worse FDE in UCY scenes while remaining on par with state of the art.The authors attribute the UCY trade-off to dense scenes being partitioned into small cliques, which leaves inter-clique interactions unmodeled.Source: §4.1 · p. 6
- Three to five sampled modes balance pedestrian prediction quality and runtime, and predictions are already very accurate after three ETH samples.The experiments use the highest-probability modes for Best-of-N ADE/FDE; when N exceeds the clique-mode count, only the available modes are used.Source: §4 · pp. 5–6, §4.1 · p. 6, Figure 5 · p. 6
4.2. Vehicle Motion Prediction
- ScePT’s nuScenes prediction accuracy is near state of the art, especially at later timesteps, and improves with only one or two extra modes.The experiments evaluate vehicles and pedestrians; the dataset contains 1,000 driving scenes, with up to 23 object classes.Source: §4.2 · p. 6, Table 3 · p. 7
- ScePT produces fewer vehicle-trajectory collisions than Trajectron++ [42], and collision cost regularization reduces its collision rate to virtually zero.The collision rate tends to decrease with prediction horizon; with regularization, prediction accuracy is reported as the same or better.Source: §4.2 · p. 6, Figure 6 · p. 6, §4.5 · p. 8
4.3. Conditioning and Counterfactual Analyses
- Conditional predictions capture reactions across clique members: braking by vehicles A and C leads B to brake and D to change lanes, while C accelerating triggers a broader chain reaction.The model also produces reasonable conditioned predictions when evaluated with clique sizes up to 8, although it was trained with a maximum size of 4.Source: §4.3 · pp. 6–7, Figure 7 · p. 7
4.4. Integration with a Downstream Planner
- ScePT’s multimodal predictions feed an MPC contingency planner that plans one ego trajectory per joint mode while requiring a shared first control input.For three modes, unoptimized prediction takes under 240 ms and MPC planning under 60 ms, both on a CPU.Source: §4.4 · p. 7, Figure 8 · p. 8
4.5. Ablation study
- Collision regularization improves scene consistency in out-of-distribution conditional scenarios without reducing prediction accuracy; the CVaR loss also outperforms the expectation baseline.Without collision regularization, errant conditioned behavior can produce collisions; the ablation reports accuracy as unchanged or better after adding the penalty.Source: §4.5 · pp. 7–8, Table 5 · p. 8
- Flow-distance clique partitioning performs better than Euclidean-distance partitioning, which still yields decent results; a maximum clique size of 4 performs best in the reported ablation.Small cliques can omit nearby neighbors, while large cliques expand the product latent space and complicate greedy sampling.Source: §4.5 · p. 8, Table 6 · p. 8, Table 7 · p. 8
5. Conclusion and Future Work
ScePT combines clique-level joint prediction with interaction policies and collision regularization, achieving strong accuracy and scene consistency while retaining limitations in scalability and inference time.
5. Conclusion and Future Work
- Across ETH, UCY, and nuScenes, ScePT achieves state-of-the-art prediction accuracy with significantly improved scene consistency.Source: §5 · p. 8
- ScePT uses a CVAE to generate multimodal joint trajectories, with clique-level mode probabilities and decoder-learned agent interaction policies.An explicit collision penalty regularizes closed-loop predictions during training.Source: §5 · p. 8
- The model demonstrates integration with a downstream contingency planner and generates human-like behaviors through conditioning.Source: §5 · p. 8
- Concurrent clique decoding loses interaction-graph sparsity, limits clique size, and ignores cross-clique interactions, compromising accuracy in crowded scenes such as UCY.The autoregressive policy network also increases computation time.Source: §5 · p. 8
A.1. Pre-encoding of Nodes and Edges
Node and edge features are pre-encoded through fully connected layers before use in the model.
A.1. Pre-encoding of Nodes and Edges
- Global state histories and futures are transformed into local frames centered at each node’s current position.Vehicle heading is represented by cos(ψ) and sin(ψ) to avoid the ±2π issue.Source: §A.1 · p. 11
- A shared fully connected pre-encoding layer processes raw state features for each node type whenever state information is used.Source: §A.1 · p. 11
- Each edge type has its own pre-encoding layer, which encodes relative position, relative velocity in the local frame, and agent sizes.Source: §A.1 · p. 11
A.2. Dynamic Models and Collision Checking
ScePT uses different dynamic models for vehicles and pedestrians, with bounded inputs intended to keep generated trajectories dynamically feasible.
A.2. Dynamic Models and Collision Checking
- Vehicles use a Dubin’s car model with longitudinal velocity and acceleration, heading angle, and yaw rate.Source: §A.2 · p. 11
- Pedestrians follow a double integrator model.Source: §A.2 · p. 11
- Input bounds are imposed on longitudinal acceleration, yaw rate, and pedestrian velocity so generated predictions remain dynamically feasible.The stated bounds are ˙v ∈ [−5m/s, 5m/s], ˙ψ ∈ [−1m/s2, 1m/s2], and vx, vy ∈ [−5m/s, 5m/s].Source: §A.2 · p. 11
A.3. Collision Check
Collision checking represents pedestrians as circles and vehicles as rectangles, with pairwise checks designed to support differentiable training regularization.
A.3. Collision Check
- Pedestrians are modeled as circles and vehicles as rectangles; pedestrian collisions are checked using Euclidean distance.Figure 9 depicts the vehicle–pedestrian collision check.Source: §A.3 · p. 11, Figure 9 · p. 11
- Vehicle–pedestrian collision checks use the vehicle’s local coordinate frame.Source: §A.3 · p. 11, Figure 9 · p. 11
- Vehicle-to-vehicle checks use the four corners of the two rectangles to calculate the collision function.Figure 10 depicts the two-vehicle collision check.Source: §A.3 · p. 11, Figure 10 · p. 11
- The collision functions are differentiable or piecewise differentiable, allowing their use as training regularization.Source: §A.3 · p. 11
A.4. Diversity Scheduling during Training
The training schedule uses α to balance encoder accuracy against trajectory diversity, starting small and increasing it over time. Above a threshold, gradients from prediction errors are retained only for the mode with the largest Q.
A.4. Diversity Scheduling during Training
- α balances encoder accuracy against diversity, beginning small so the decoder can learn diverse trajectory patterns without mode collapse.α is increased later to improve encoder prediction accuracy.Source: §A.4 · p. 11
- When α exceeds a threshold, prediction-error losses for modes other than the one with the largest Q are detached from the gradient graph to avoid mode collapse.This reduces mode collapse under small α while allowing encoder mode-probability prediction to keep improving.Source: §A.4 · p. 11
- The schedule first favors diversity, then raises α to improve encoder prediction accuracy.The passage presents this as a way to reduce mode collapse while continuing to improve mode-probability prediction.Source: §A.4 · p. 11
A.5. Diverse Sampling from Product Latent Space
Greedy sampling from the product latent space can lose diversity when clique modes with similar latent variables have similar probabilities. ScePT follows a diverse-sampling scheme that selects latent assignments one by one using a distance function.
A.5. Diverse Sampling from Product Latent Space
- Similar-probability clique modes that differ at only one latent-variable node can make greedy sampling lose diversity; the paper follows the diverse-sampling scheme in.The passage identifies this as a known issue in Markov random fields.Source: §A.5 · pp. 11–12
- The diverse-sampling procedure greedily picks latent assignments one by one.It uses the distance function to distinguish assignments.Source: §A.5 · p. 12
- The distance between two latent assignments counts the nodes whose latent variables differ.For example, the supplied cases give distances of 1 for [0, 1, 2] versus, and 2 for [1, 0, 0] versus [2, 1, 0].Source: §A.5 · p. 12