PaperlayerView exact revision on arXiv

Source-linked AI summary

Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion

Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, Or Litany

arXiv:2304.01893v1cs.CVcs.GRcs.LG

TL;DR

Pedestrian trajectory models must balance realistic motion with explicit user control, while physics-based animation controllers often require task-specific planning. The paper combines guided trajectory diffusion with a physics-based humanoid controller in a closed loop, and reports realistic, controllable animation across tasks, terrains, and characters; value guidance further improves trajectory suitability. Sampling remains a limitation for real-time use, and multiple guidance objectives can be difficult to balance.

  • Problem

    Learning-based pedestrian models can improve naturalness but provide limited explicit control, while physics-based animation controllers require task-specific planners for new tasks, terrains, and body shapes.

  • Method

    The method combines TRACE, a guided trajectory diffusion model, with PACER, a physics-based humanoid controller, in a closed-loop system using map, social, terrain, and value-function guidance.

  • Results

    The system generates realistic, controllable pedestrian motion and performs robustly across user objectives, tasks, terrains, and characters, while value guidance reduces failures and improves discriminator reward.

  • Takeaways & Limitations

    Guided diffusion can provide test-time control for realistic pedestrian trajectories while supporting their execution as full-body physics-based animation.

  • Takeaways & Limitations

    TRACE takes 1–3s to sample for a single character, and combining multiple guidance objectives can make the total guidance strength too large depending on the scene.

Abstract

from arXiv · show

We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet user-defined goals. We draw on recent advances in guided diffusion modeling to achieve test-time controllability of trajectories, which is normally only associated with rule-based systems. Our guided diffusion model allows users to constrain trajectories through target waypoints, speed, and specified social groups while accounting for the surrounding environment context. This trajectory diffusion model is integrated with a novel physics-based humanoid controller to form a closed-loop, full-body pedestrian animation system capable of placing large crowds in a simulated environment with varying terrains. We further propose utilizing the value function learned during RL training of the animation controller to guide diffusion to produce trajectories better suited for particular scenarios such as collision avoidance and traversing uneven terrain. Video results are available on the project page at https://nv-tlabs.github.io/trace-pace .

1. Introduction

The paper addresses the tension between controllability and naturalness in pedestrian trajectory generation, then combines guided diffusion with physics-based animation for controllable full-body simulation. TRACE provides test-time trajectory control, while PACER follows those trajectories across terrains, social settings, and body types.

  • Motivation: Users can specify objectives such as waypoints, social groups, or social distance for generated pedestrian trajectories.
  • Motivation: Rule-based pedestrian models are controllable but often unnatural, whereas learning-based models improve naturalness yet offer limited explicit control.Existing learned approaches commonly predict short-term trajectories in a single forward pass and restrict control to sampling from an output distribution or related mechanisms.
  • TRACE: TRACE is a data-driven trajectory diffusion model that uses locally queried map features and test-time guidance to generate realistic trajectories meeting user-defined objectives.Classifier-free sampling supports mixed annotations and trades sample diversity against controllability at test time.
  • PACER: PACER is a general-purpose physics-based controller designed for natural locomotion across diverse body types, terrains, and social settings.It is intended to convert high-level planned trajectories into full-body motion while handling transitions, obstacle avoidance, and uneven terrain.
  • Integrated system: The integrated system forms a closed loop in which TRACE plans and PACER animates, with frequent replanning according to simulation results.The system is presented as a controllable pedestrian animation system for simulated characters and crowds.
  • Contributions: The paper contributes a controllable trajectory diffusion model, a general-purpose pedestrian animation controller, and an integrated system for controllable simulated pedestrians.

2. Related Work

Related work spans physics- and planning-based forecasting, learned trajectory prediction, and systems that combine high-level planning with low-level animation. TRACE-PACER follows the two-stage paradigm while applying task information to both levels and linking them through closed-loop runtime integration.

  • Pedestrian trajectory prediction: Pedestrian forecasting methods range from physics and planning-based approaches to learned methods, with many applications motivated by autonomous vehicles or social robots.
  • Pedestrian trajectory prediction: Data-driven models produce human-like motion but are difficult to control explicitly, with some methods offering limited goal-based or objective-based control.Examples include selecting goal locations near a target or minimizing objectives such as collisions.
  • High-level planning and low-level control: Single-network systems can combine trajectory following with low-level body control, but physics-based humanoid controllers may require separate models for each task.The cited examples include kinematic waypoint following, gamepad-input control, and task-specific physics-based controllers.
  • High-level planning and low-level control: Two-stage systems separate high-level planning from low-level character control using planners such as A*, learned prediction, latent-space search, or hierarchical reinforcement learning.Prior systems achieve dynamic control across tasks but can require lengthy training.
  • TRACE-PACER: TRACE and PACER both consume task information, with TRACE using test-time guidance and PACER using map-conditioned path following.Although trained independently, they are integrated at runtime in a closed loop.
  • Diffusion models and guidance: Compared with related diffusion-guided trajectory methods, TRACE adds expressive map-feature conditioning, classifier-free sampling, clean-output guidance, and value-function coupling to animation.

3. Method

TRACE generates future pedestrian trajectories by conditioned diffusion and guides denoising at test time toward user objectives. PACER then executes these plans with physics-based humanoids, while value-function guidance couples trajectory generation to terrain and body-aware control.

  • TRACE: TRACE formulates pedestrian behavior as agent-centric future trajectory forecasting conditioned on ego history, neighboring motion, and semantic map context.The model generates a future state trajectory over a fixed horizon from these inputs.
  • TRACE: TRACE learns reverse diffusion denoising to generate plausible trajectories from progressively noised inputs, with each reverse step conditioned on the scene context.The forward process adds Gaussian noise, while the learned reverse process predicts clean trajectories and uses them to parameterize denoising.
  • TRACE: Classifier-free sampling combines conditional and unconditional predictions at test time, allowing users to adjust conditioning strength and trade off compliance against trajectory diversity.Conditioning is randomly dropped during training, and the sampling weight controls the effect of conditioning.
  • Controllability through Clean Guidance: Clean guidance perturbs the predicted clean trajectory during denoising so analytical or learned objectives can steer trajectories toward waypoints, obstacle avoidance, collision avoidance, and social-group goals.The method computes a guided mean from a guidance loss and applies gradients through the denoising model.
  • Physics-Based Pedestrian Animation: PACER is a physics-based, goal-conditioned humanoid controller that follows TRACE trajectories while accounting for simulated state, environment, body type, terrain, and social context.TRACE and PACER are trained independently but operate in a closed feedback loop with frequent replanning.
  • Controllable Pedestrian Animation System: The PACER value function guides TRACE denoising toward trajectories that are easier for the controller to follow on the current terrain and around surrounding agents.Because the value function reflects body pose, terrain, and nearby agents, it provides tighter two-way coupling between planning and animation.

4. Experiments

The experiments evaluate TRACE on synthetic and real-world data, then test its closed-loop integration with PACER across controllable animation settings. Results show successful guidance, out-of-distribution flexibility, and improved simulated animation with value-function guidance.

  • Experimental setup: TRACE is evaluated on synthetic and real-world pedestrian data, including open-loop 5s prediction and closed-loop 10s rollout with replanning at 1Hz.The real-world evaluation uses ETH/UCY and nuScenes, while the synthetic evaluation uses ORCA-Maps and ORCA-Interact.
  • Synthetic data: TRACE full diffusion guidance improves objective satisfaction over VAE latent optimization and selective sampling while maintaining strong realism on ORCA-Maps.The evaluation compares TRACE with a VAE baseline and TRACE-Filter, which selects the best unguided sample.
  • Real-world data: Mixed training and classifier-free sampling with w<0 are important for achieving out-of-distribution objectives on nuScenes.Using w=−0.5 downweights semantic-map conditioning and leverages more diverse trajectories learned from ETH/UCY.
  • Real-world data: TRACE supports noisy waypoint guidance, social-group cohesion, and more diverse unconditional samples when trained on mixed ETH/UCY and nuScenes data.Figure 5 highlights these flexibility benefits in nuScenes scenes.
  • Controllable animation: The closed-loop TRACE-PACER system follows waypoints and avoids collisions across flat, obstacle-filled, rough, sloped, and stair terrains.The animation evaluation uses 600 10-second rollouts across 30 characters with varied bodies and environmental settings.
  • Controllable animation: Value-function guidance makes trajectories easier to follow, reduces failures, improves discriminator reward, and lowers waypoint-guidance error.These gains are reported across settings involving varying terrains, obstacles, and other agents.

5. Discussion

The paper combines controllable trajectory diffusion with a physics-based humanoid controller in an end-to-end animation system. It identifies real-time diffusion sampling and low-level full-body diffusion control as directions for future work.

  • Contributions: The proposed system combines a controllable trajectory diffusion model, a robust physics-based humanoid controller, and an end-to-end animation pipeline.The discussion presents this combination as a step toward controlling high-level behavior in learned pedestrian models.
  • Future work: TRACE currently takes 1–3s to sample for a single character, motivating future work on real-time diffusion sampling.The paper points to diffusion-model distillation as a potential way to improve sampling efficiency.
  • Future work: Extending diffusion models from high-level motion controllability to low-level full-body character control is identified as another future direction.This direction concerns control beyond the trajectory-level interface developed in the paper.

Appendices

The appendices provide implementation details for TRACE and PACER, including trajectory inputs, map representations, dynamics-aware denoising, and diffusion sampling. They also point readers to expanded experimental details and video results.

  • Appendix contents: Appendix A covers TRACE, while Appendix B covers PACER; Appendix C adds experiment details and Appendix D adds supplementary results.Appendix E discusses limitations and future work in more detail.
  • TRACE inputs: The model uses local pedestrian history and neighbor states containing position, heading, speed, bounding-box dimensions, and visibility indicators.Invisible people are zeroed before being provided to the diffusion model.
  • TRACE inputs: TRACE represents the local environment with a 224×224 bird’s-eye semantic map at 12 px/m, showing 14m ahead, left, and right and about 4.6m behind.Map channels are binary semantic layers, such as walkable area and obstacles in ORCA.
  • Denoising: Dynamics-aware denoising converts noisy action sequences into state trajectories, predicts a clean action trajectory, and applies the resulting denoising update.TRACE uses a unicycle dynamics model to encourage smooth, followable predictions.
  • Denoising: TRACE predicts the clean trajectory τ^0 during diffusion, uses it to compute each step’s Gaussian mean, and mixes conditional and unconditional outputs for classifier-free sampling.The appendix specifies a cosine variance schedule and explains how the predicted clean trajectory determines the next denoising step.

A.1.2 Architecture

The architecture conditions trajectory denoising on histories and locally queried map features, then applies test-time guidance losses to steer single- or multi-agent futures toward user objectives.

  • Denoising network: TRACE processes denoising steps with a positional step embedding and a 1D U-Net over trajectory sequences.The U-Net uses temporal convolutions, residual connections, and temporal downsampling and upsampling.
  • Scene conditioning: A ResNet-18 encoder and 2D U-Net decoder produce a 56 × 56 × 32 learned map-feature grid for spatial conditioning.Map features are queried locally at current noisy trajectory positions through bilinear interpolation.
  • Trajectory conditioning: Ego and neighbor trajectory histories are encoded separately, with neighbor features max-pooled into a single interaction representation.The histories are flattened and processed by MLPs before being combined with other conditioning features.
  • Conditioning strategy: Classifier-free conditioning drops map and neighbor inputs independently during training, enabling test-time trade-offs between controllability and sample diversity.Each conditioning source is dropped with 10% probability, while ego history is retained.
  • Test-time guidance: TRACE supports waypoint, obstacle-avoidance, agent-avoidance, social-distance, and learned-value guidance objectives during denoising.Multi-agent objectives denoise pedestrians jointly and filter complete scene samples using aggregate guidance loss.
  • Waypoint guidance: Global waypoint guidance uses preferred speed and urgency to make progress toward distant targets before switching to local waypoint control when the target enters the planning horizon.The global specific-time objective is designed for closed-loop operation and supports eventual arrival at a desired time.

B.1. Implementation Details

PACER is a goal-conditioned physics-based humanoid controller that follows TRACE trajectories using terrain-aware observations, reinforcement learning, and motion-style objectives across varied bodies and terrains.

  • Inputs and state: PACER observes a 64 × 64 × 3 local height-and-velocity map covering a 4m × 4m egocentric area, along with humanoid state and trajectory inputs.The trajectory input has shape 10 × 2, while the map encodes terrain height and planar velocity.
  • Policy architecture: The policy separates task-feature processing from action prediction, producing a 256-dimensional latent task vector before mapping humanoid state to actions.The task processor is convolutional, and the action network is an MLP.
  • Control output: The policy outputs Gaussian actions corresponding to PD targets for the 23 actuated joints of the SMPL body.The value function shares the policy architecture, supporting its use for trajectory guidance.
  • Learning objectives: Training combines trajectory-following, style, and energy terms, while a directly optimized symmetry loss addresses asymmetric low-speed gaits.The authors report that asymmetric motion can manifest as limping during training.
  • Terrain and termination: Training uses synthetic terrains including obstacles, discrete terrain, stairs, uneven terrain, and slopes, with early termination after collision forces above 50 N.The termination rule also functions as fall detection, except at ankles and feet contacting the ground.
  • Training data and scale: The controller trains on roughly 200 AMASS locomotion sequences and 476 body shapes, with 2048 humanoids sampled for parallel training.Reference motions include walking, turning, and stair traversal at varied speeds.
  • Datasets: TRACE data are resampled to 10 Hz, while ORCA-Maps and ORCA-Interact each contain 1000 ten-second scenes with distinct obstacle and interaction settings.ORCA-Maps includes maps and obstacles; ORCA-Interact contains more pedestrians but no obstacles or map annotations.

C.2. Guidance Metrics

The guidance metrics quantify how closely generated trajectories satisfy waypoint, collision-avoidance, social-group, and multi-objective constraints.

  • Collision objectives: Obstacle collision rate measures the average fraction of rollout time that an agent disk overlaps an obstacle.The disk representation matches the ORCA simulator and its collision-free ground-truth representation.
  • Collision objectives: Agent collision rate measures the average fraction of agents involved in agent-agent collisions within each scene rollout.This metric also uses the simulator’s disk representation.
  • Waypoint objectives: Waypoint guidance error is the target distance at a specified timestep or the minimum trajectory-to-goal distance when timing is unconstrained.The metric is reported in meters for waypoint objectives.
  • Social-group objectives: Social-group error is the mean absolute difference between the specified social distance and each pedestrian’s distance to the closest group neighbor.The measure is averaged across pedestrians in the group.
  • Multi-objective evaluation: For waypoint-plus-avoidance guidance, the reported guidance error is waypoint error because collision rates are reported separately.The multi-objective metric therefore isolates waypoint accuracy in that table.

C.3. Other Metrics

Additional metrics assess trajectory realism and contextualize a conditional-VAE baseline through optimization cost and sample diversity.

  • Realism metrics: Realism EMD compares generated and ground-truth histograms of velocity, longitudinal acceleration, and lateral acceleration over the test set.The histograms are normalized before earth mover’s distance is computed.
  • Realism metrics: Realism Mean reports average longitudinal and lateral acceleration in m/s2 as a proxy for motion smoothness.The metric is computed on generated trajectories and reflects the tendency toward smooth pedestrian motion.
  • VAE baseline: The conditional VAE baseline controls trajectories through latent-space optimization using a guidance loss and a prior term.Optimization uses Adam with 100 iterations for one experiment and 200 for more difficult out-of-distribution objectives.
  • VAE baseline: With sufficient compute, the VAE can often meet objectives faithfully, but it may require more iterations than diffusion and converge toward less diverse samples.The prior term drives optimized trajectories toward the mean when optimization runs for a long time.

C.5. Additional Experiment Details

The supplementary experiments describe evaluation setups for crowd simulation, real-world data, and controllable animation, alongside additional qualitative results.

  • Augmenting Crowd Simulation: Crowd-simulation evaluations average three no-guidance runs because random samples are selected without filtering.TRACE uses w = 0.0, drawing and guiding 20 samples per pedestrian before filtering.
  • Real-world Data Evaluation: Real-world evaluation draws 10 samples before filtering and uses global waypoint guidance with urgency u = 0.7 and preferred speed vpref = 1.25m/s.Perturbed waypoint targets add Gaussian noise with standard deviation 2m.
  • Controllable Pedestrian Animation: Controllable animation evaluates waypoints placed 7–12 meters ahead and up to 5 meters sideways, targeted 9 seconds into the future.Agents begin standing with uniform random root velocity in [1.2, 2.0]m/s.
  • Additional Results: Additional experiments include 20-sample trajectory visualizations, supplementary videos, and setup details omitted from the main paper.The qualitative videos cover both TRACE and PACER.

D.2. TRACE– No Guidance Ablation Study

The ablations examine TRACE’s no-guidance behavior, classifier-free sampling, PACER design choices, and planning runtime. They show trade-offs among collision avoidance, waypoint adherence, motion realism, and computational cost.

  • Evaluation Metrics: ADE and FDE evaluate trajectory accuracy against ground-truth samples over the rollout horizon and at its endpoint.The evaluation also considers collision avoidance, realism, and accuracy in open-loop 5s rollouts.
  • TRACE Architecture: TRACE’s feature-grid map conditioning achieves the lowest obstacle collision rate while retaining competitive accuracy and realism.Local map cues help model subtle obstacle interactions, although agent collisions remain challenging across models.
  • Classifier-Free Sampling: For w > 1, classifier-free guidance becomes too strong and makes trajectory samples nearly deterministic despite small quantitative differences.Increasing w has a considerable qualitative effect on sampled trajectories.
  • Classifier-Free Sampling: Smaller w improves waypoint-reaching error on nuScenes, while w < −0.5 causes mean accelerations to deviate from ground-truth motion.More extreme trajectories are produced to reach waypoints.
  • PACER Ablations: Agent-aware PACER is beneficial even when TRACE already models neighboring pedestrians, while body awareness mainly improves unguided terrain traversal.Body awareness leaves waypoint-guided performance essentially unchanged.
  • Runtime Analysis: TRACE planning takes ≤5 sec for ≤50 agents but becomes more costly with 100 agents, especially under agent-avoidance guidance.Collision avoidance is expensive because it requires pairwise comparisons among agents.
  • Runtime Analysis: Standalone PACER runs at approximately 30 fps for one humanoid and 25 fps for 100 humanoids.This measurement concerns PACER rather than the full TRACE planning pipeline.

E. Discussions and Limitations

The discussion identifies diffusion denoising speed and multi-objective guidance strength as practical limitations of TRACE.

  • TRACE Efficiency: Diffusion denoising limits real-time use, although diffusion distillation could potentially accelerate sampling.This is described as a known issue for diffusion models.
  • Multi-Objective Guidance: Combining guidance objectives can overstrengthen trajectory updates and push samples off-manifold, making weight balancing difficult.The authors mention dynamic clipping as a possible direction, but note that adapting it to trajectories is nontrivial.
Loading 2304.01893v1…