PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation

Siyuan Liu, Miao Li, Haibao Yu, Haohong Lin, Qing Zhou, Bingbing Nie, Ding Zhao

arXiv:2610.06171v1cs.RO

TL;DR

Rare vehicle–pedestrian conflicts require sensor-level scenarios for evaluating end-to-end driving, but existing trajectory-based and video-based approaches lack visual observations or controllability. ControlPed combines conflict trajectories, text-conditioned 3D motion, and photorealistic rendering using HazardPed. Across seven end-to-end driving models, mean HDScore drops from 88.8 to 47.4 on rendered pedestrian hazards.

  • Problem

    Rare, safety-critical vehicle–pedestrian interactions make sensor-level data for end-to-end driving evaluation scarce.

    Source: §I · p. 1

  • Method

    ControlPed combines conflict trajectory generation, text-conditioned motion diffusion, and photorealistic rendering, using HazardPed’s trajectories and hazardous human motions.

    Source: §I · pp. 1–2

  • Results

    Mean HDScore drops from 88.8 to 47.4 across seven end-to-end driving models evaluated under rendered pedestrian hazards.

    Source: C. End-to-End Safety Evaluation · p. 7

  • Takeaways & Limitations

    The photorealistic benchmark is intended to support third-party and regulatory assessment of end-to-end driving safety before real-world deployment.

    Source: §I · p. 2, §VI · p. 8

    ControlPed currently operates in an open-loop setup; extending it to closed-loop interactive simulation is left for future research.

    Source: §VI · p. 8

Abstract

from arXiv · show

Evaluating end-to-end autonomous driving under rare, safety-critical vehicle-pedestrian interactions requires photorealistic, sensor-level scenarios. However, trajectory-based scenario generators cannot synthesize raw visual observations, whereas video-based approaches lack controllability. To bridge this gap, we present ControlPed, a novel framework that combines trajectory-level conflict synthesis with 3D Gaussian Splatting (3DGS) to generate photorealistic, motion-controllable safety-critical scenarios. Built upon HazardPed, a dataset derived from 10,352 traffic videos comprising 422 conflict trajectories, HD maps, and 857 annotated 3D human motions, ControlPed first generates conflict trajectories, lifts them into 3D human motion sequences via text-conditioned motion diffusion, and finally renders multi-view sensor observations using animatable 3DGS avatars. Safety evaluation in 88 rendered photorealistic scenarios reveals that seven leading end-to-end driving models suffer a severe performance drop, with their mean HDScore plunging from 88.8 to 47.4, exposing major failure modes under dangerous pedestrian behaviors. The dataset and testing benchmarks will be released to facilitate safety assessment of vehicle-pedestrian interactions.

I. INTRODUCTION

ControlPed addresses the need for sensor-level evaluation of rare vehicle–pedestrian conflicts by combining controllable conflict generation with photorealistic rendering. It introduces the HazardPed dataset, a three-stage generator, and a benchmark for evaluating end-to-end driving safety.

I. INTRODUCTION

  • The framework decomposes scenario generation into macro-level trajectory control, micro-level joint dynamics, and sensor-level photorealistic rendering.ControlPed is trained on 422 real conflict trajectories and 857 hazardous human motion sequences.Source: §I · p. 1, Figure 1 · p. 1
  • HazardPed combines real conflict trajectories, HD maps, and human motions to address the scarcity of risky interaction data.The dataset provides a prior for controllable pedestrian scenario generation.Source: §I · p. 2
  • ControlPed couples trajectory-level conflict generation with text-conditioned motion diffusion to combine sensor-level photorealism with fine-grained pedestrian control.The framework lifts conflict trajectories into 3D human motion and composites animatable avatars into 3DGS scenes.Source: §I · pp. 1–2, Figure 1 · p. 1
  • The benchmark evaluates autonomous driving safety in photorealistic scenarios and is intended to support third-party and regulatory assessment before real-world deployment.The framework’s safety-test stage evaluates end-to-end driving algorithms on rendered multi-view scenarios.Source: Figure 1 · p. 1, §I · p. 2

A. Safety-Critical Scenario Generation

The section describes gaps in controllable, sensor-level pedestrian scenario generation and the scarcity of safety-critical motion data. It then introduces HazardPed’s data and the stages used to construct it.

A. Safety-Critical Scenario Generation

  • Trajectory-based generators perturb behavior, [12] or use traffic priors, but 2D bird’s-eye-view representations strip away key pedestrian dynamics; video methods [5], add sensor realism yet lack fine-grained trajectory or human control.Asset insertion struggles to enforce precise terminal collision states, while 2D pose editing lacks metric grounding for real-world vehicle–pedestrian distances.Source: A. Safety-Critical Scenario Generation · p. 2

B. Human Motion Generation and Datasets

  • Standard motion benchmarks focus on daily activities, leaving safety-critical reactions such as abrupt dodging or freezing under vehicle threat scarce.Text-conditioned motion models [8], lack spatial grounding, and spatial-control techniques – remain bounded by their training data.Source: B. Human Motion Generation and Datasets · p. 2

C. Scenario Reconstruction and End-to-End Evaluation

  • Reconstruction-based closed-loop simulations combine realism with interactivity but lack critical pedestrian hazards, motivating controllable pedestrian insertion into photorealistic scenes.Open-loop tests on passive logs lack reactivity, while simulators exhibit a pronounced sim-to-real gap.Source: C. Scenario Reconstruction and… · p. 2

III. HAZARDPED DATASET

  • HazardPed integrates vehicle–pedestrian conflict trajectories, aligned HD maps, and paired text-conditioned human motion data.It is constructed by filtering dynamic interactions from web videos and open-source datasets.Source: §III · p. 2, Figure 2 · p. 2

A. Safety-critical scenario dataset

  • The dataset pipeline retained 3,497 valid clips from 10,352 traffic videos and yielded 422 scenarios with 566 vehicle–pedestrian conflict pairs.It uses pedestrian tracking, camera and pose estimation, trajectory extraction, world-coordinate mapping, and road-semantic extraction.Source: A. Safety-critical scenario dataset · pp. 2–3

B. Human Motion Extraction and Annotation

The authors annotate pedestrian motion clips with action categories and text descriptions, pairing them with 3D root and joint motion data. They release processed motion and map data, but not the raw source videos.

B. Human Motion Extraction and Annotation

  • After human verification, 1,473 motion clips were retained and annotated using BABEL’s vocabulary, generated descriptions, action categories, and frame-wise kinematic boundaries.Final review yielded 586 navigation clips and 271 hazardous reaction clips.Source: B. Human Motion Extraction and… · p. 3
  • Each sample pairs root translation and orientation, 21 body-joint rotations, an action category, and a natural-language description.To protect privacy, the release includes processed trajectories, HD maps, and motion data, excluding raw source videos.Source: B. Human Motion Extraction and… · p. 3

A. Hazardous Trajectory Generation

Hazardous trajectory generation synthesizes future pedestrian paths conditioned on scene context and a target direction, then uses a conflict controller to steer sampled trajectories toward vehicle interceptions. Direction augmentation and two-stage training address limited conflict data.

A. Hazardous Trajectory Generation

  • The generator models future trajectories from pedestrian history, surrounding vehicles and pedestrians, map semantics, scene origin, and a unit-vector goal heading.The task is defined in a pedestrian-centric frame.Source: A. Hazardous Trajectory Generation · p. 3
  • Conditional diffusion encodes heterogeneous context into a condition vector that modulates a 1D U-Net via FiLM, with denoising trained by mean squared error.During training, goal heading and map inputs are sometimes replaced with learnable null embeddings for classifier-free guidance.Source: Figure 3 · p. 3, A. Hazardous Trajectory Generation · p. 3
  • Direction augmentation progressively rotates future velocities while preserving each scenario’s history, agents, and map, expanding goal-heading coverage for paths toward oncoming vehicles.Pairing identical historical context with varied future directions supports synthesis when the goal heading points toward a vehicle.Source: A. Hazardous Trajectory Generation · p. 4
  • Two-stage training learns general pedestrian priors from 700 nuScenes scenes, then fine-tunes on augmented conflict data to cover goal headings toward vehicles.The first stage lacks conflict scenarios, which the second stage is intended to compensate for.Source: A. Hazardous Trajectory Generation · p. 4
  • The conflict controller selects interception timing and severity, then replans goal headings every 0.4 s and filters trajectories by speed, drivable area, and threat constraints.The selected frame determines conflict timing; the target gap sets severity from grazing contact to deeper collisions.Source: A. Hazardous Trajectory Generation · p. 4

B. Human Motion Generation

Human motion generation lifts a 2D trajectory into a full-body motion sequence, using the input path to anchor root position and guide orientation. Text and action conditions guide synthesis, which is trained with a composite motion-quality loss and extended to long sequences at inference.

B. Human Motion Generation

  • The motion state combines root translation, root orientation, and 21-joint rotations, guided by an action category, text prompt, and input trajectory.The state has 135 channels per frame.Source: B. Human Motion Generation · p. 4
  • The input trajectory occupies and clamps the root’s two horizontal channels, framing synthesis as inpainting the remaining 133 motion channels.During diffusion, the root horizontal channels are overwritten with the input trajectory at every step and excluded from the loss.Source: B. Human Motion Generation · p. 4
  • Root heading is derived by smoothing trajectory differentiation, so the network predicts residual orientation that is recomposed with the path heading.This focuses the network on body turns relative to the path rather than global direction.Source: B. Human Motion Generation · p. 4
  • The motion condition sums diffusion time, frozen CLIP text, and action-category embeddings, and adaLN-Zero modulates each transformer sublayer.The backbone is an eight-layer pre-LN transformer with hidden dimension 512 and 8 attention heads.Source: B. Human Motion Generation · p. 4
  • A composite loss combines rotation reconstruction, world-space joint position and velocity fidelity, contact foot-sliding suppression, and upweighted root-height and spine-orientation terms.The root-height and spine-orientation terms counter feature dilution.Source: B. Human Motion Generation · pp. 4–5
  • Inference uses classifier-free guidance and sliding windows conditioned on historical frames, then blends seams and adds a sampled hazardous reaction motion at the conflict point.Navigation text is assigned by trajectory curvature.Source: B. Human Motion Generation · p. 5

C. Pedestrian Injection and Rendering.

Generated pedestrian motions are transformed into world coordinates to drive animatable avatars in reconstructed scenes. Per-frame root-height adjustment addresses terrain-reference mismatch before multi-view rendering.

C. Pedestrian Injection and Rendering

  • Generated motions drive newly initialized pedestrian avatars after conversion from canonical local frames to world coordinates.Direct use of generated root heights can cause floating or ground-penetration artifacts because they lack a shared datum with reconstructed terrain.Source: C. Pedestrian Injection and Rendering · p. 5
  • The rendering pipeline aggregates and rasterizes background, vehicle, and dynamic-pedestrian 3D Gaussians into multi-view video streams.Per-frame root height is resolved using local ground elevation and the lowest posed-body point.Source: C. Pedestrian Injection and Rendering · p. 5

V. EXPERIMENTS

The experiments assess trajectory generation, 3D motion synthesis, and end-to-end model degradation in rendered scenes, with qualitative visualizations in Figures 5–7.

V. EXPERIMENTS

  • Evaluation spans trajectory generation, 3D motion synthesis, and end-to-end model degradation in rendered scenes, with qualitative results shown in Figures 5–7.Source: §V · p. 5

A. Experimental Setup.

The experimental setup evaluates trajectory generation, motion synthesis, and end-to-end driving on distinct scenario and motion sets. It specifies metrics for realism, criticality, diversity, motion quality, and driving performance, alongside comparison baselines.

A. Experimental Setup

  • Trajectory generation uses 92 nuScenes validation conflicts, motion generation uses 857 HazardPed clips across 34 categories, and end-to-end tests use 88 human-inspected rendered scenes.Source: A. Experimental Setup · p. 5
  • Trajectory metrics cover path and speed realism, collision and hard-braking criticality, and hazardous-scene coverage, geometric diversity, and risk-spectrum span.Hard braking is triggered above 3.0 m/s2 deceleration; hazardous trajectories are those with distance below 1.0 m.Source: A. Experimental Setup · pp. 5–6
  • Motion evaluation reports FID, MM-Dist, R@3, pelvis-height error for low-pose actions, heading-error deviation, and class-separability-ratio deviation.Source: A. Experimental Setup · p. 6
  • End-to-end evaluation reports NC, DAC, TTC, comfort, route completion, and composite HDScore, using an open-loop route-completion proxy that stops after a collision within 0.5 seconds.Source: A. Experimental Setup · p. 6
  • The setup compares three trajectory generators, five motion generators retrained on HazardPed, and seven end-to-end models evaluated open-loop with official pretrained weights.The end-to-end set spans modular dense and sparse representations, an implicit world model, a diffusion policy, and a vision-only baseline.Source: A. Experimental Setup · p. 6

B. Comparison with Baselines.

ControlPed combines trajectory realism and criticality with broad hazard coverage, while its motion model produces realistic, heading-aligned motions. Ablations show that goal-heading guidance, conflict fine-tuning, conditioning design, geometric losses, and category embeddings materially affect results.

Trajectory generation model

  • FD of 15.5 accompanies ControlPed’s highest HBR and high CR, enabling realistic near-miss events that force emergency braking without collision.Adv-BMT has the highest CR but severe realism degradation and zero Span; CCDiff and CounterScene retain realism with moderate criticality and narrower trajectory ranges.Source: B. Comparison with Baselines · p. 6, Table I · p. 6
  • ControlPed achieves high Cov and Span, while Adv-BMT scatters unconstrained and CCDiff and CounterScene stop at safety gaps of 8.9 m and 8.8 m.Qualitatively, ControlPed forms plausible hazard corridors reaching spatial contact.Source: B. Comparison with Baselines · p. 6, Figure 5 · p. 6

Ablation study for trajectory generation

  • Removing goal heading g drops CR by 85.5% and Span by 46.8%; omitting Stage-2 fine-tuning lowers HBR from 0.735 to 0.282 and worsens FD from 15.50 to 22.04.Removing both components yields the worst performance and fails to generate effective hazards.Source: B. Comparison with Baselines · p. 6, Table II · p. 7

Motion generation model

  • FID of 0.517 and HED of 0.27° indicate ControlPed’s distribution realism and heading alignment, while PriorMDM and MDM outperform on R@3 and MM-Dist.OmniControl matches ControlPed on MPHE and ∆CSR but has higher FID and HED; extreme ∆CSR captures mode collapse associated with stronger retrieval metrics.Source: B. Comparison with Baselines · p. 7, Table III · p. 7
  • ControlPed generates plausible, text-aligned motions, whereas baselines show heading misalignment, upright crouching, foot sliding, stationary turning poses, or motion jitter.Each Fig. 6 column compares ControlPed above a baseline under the same text prompt.Source: B. Comparison with Baselines · p. 7, Figure 6 · p. 7

Ablation study for motion generation

  • Replacing adaptive layer normalization worsens FID from 0.517 to 1.179 and MPHE from 6.81 to 9.08 cm, while removing category embeddings gives the worst FID and highest ∆CSR.Residual-yaw removal raises HED from 0.27° to 2.28°; omitting geometric losses spikes ∆CSR to +0.356.Source: B. Comparison with Baselines · p. 7, Table IV · p. 7

C. End-to-End Safety Evaluation

Across seven end-to-end driving models, rendered pedestrian hazards sharply reduce average HDScore, mainly through declines in route completion, no-collision performance, and time-to-collision. The rendered pedestrians maintain multi-view temporal consistency across surround cameras.

C. End-to-End Safety Evaluation

  • Average HDScore falls from 88.8 to 47.4 across seven models, driven primarily by declines in route completion, no collision, and time-to-collision.Drivable area compliance remains largely unaffected; collisions prematurely terminate route execution.Source: C. End-to-End Safety Evaluation · pp. 7–8, Table V · p. 8
  • Rendered hazardous pedestrians maintain robust multi-view temporal consistency across all surround cameras, providing sensor-level photorealism for perception and planning pipelines.Fig. 7 shows representative rendered keyframes.Source: C. End-to-End Safety Evaluation · p. 8, Figure 7 · p. 8

VI. CONCLUSIONS

ControlPed, built upon HazardPed, provides controllable and photorealistic scenario generation; the work presents it as a benchmark for pre-deployment safety validation.

VI. CONCLUSIONS

  • ControlPed combines controllable, photorealistic scenario generation with a benchmark intended to support pre-deployment safety validation.The current setup is open-loop; extending it to closed-loop interactive simulation is identified as future work.Source: §VI · p. 8

Related papers

Loading 2610.06171v1…