PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Physical Consistency and Plausibility in Generative Scenario Models for Autonomous Driving

Manasa Mariam Mammen, Zafer Kayatas, Stefan Wagner

arXiv:2610.01581v1cs.AI

TL;DR

Generative driving scenarios can appear realistic, but output-level evaluation does not establish that their representations are transparent, controllable, or physically executable. The paper introduces a five-layer evaluation protocol and demonstrates it on a VAE-based scenario generator. The protocol exposes representational and physical-validity behaviors that standard metrics alone miss.

  • Problem

    Output-level realism, reconstruction, and distributional checks do not establish whether generative scenario models learn meaningful, controllable structures or produce physically executable trajectories.

    Source: §1 · p. 2

  • Method

    The five-layer protocol evaluates latent kinematic alignment, structure beyond statistical baselines, decoder organization, latent controllability, and generated trajectories against physical constraints.

    Source: §1 · p. 2, §3 · p. 3, §3.5 · p. 5

  • Results

    Applied to baseline architectures trained on highway maneuver data, the protocol revealed partial kinematic alignment, nonlinear and scenario-dependent control responses, and physically invalid outputs missed by standard metrics.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The findings support evaluating generative driving models across structural, controllability, and physical dimensions to make hidden behaviors more visible for industrial adoption.

    Source: §6 · p. 9

Abstract

from arXiv · show

Generative AI models are increasingly used for scenario generation in autonomous driving. While they can generate realistic-looking scenarios, they often provide limited transparency into learned representations and consistency with real-world vehicle dynamics. This lack of formal assurance limits their use in safety-critical validation and certification workflows. To address this aspect, we introduce a layered evaluation protocol that complements existing methods by assessing models across five layers. The first four layers inspect internal representations and network layers through kinematic alignment, statistical baseline comparison, latent controllability, and activation analysis. The fifth layer evaluates model outputs against vehicle dynamics constraints such as lateral jerk thresholds. We demonstrate the protocol on a Variational Autoencoder (VAE)-based scenario generator. Although standard output-level metrics and visualizations suggest that the generated scenarios are realistic, our protocol provides deeper insight into the extent to which the model's latent space aligns with kinematic features and whether visually plausible trajectories satisfy vehicle-dynamics constraints. We further apply the protocol to additional generative models, demonstrating its applicability beyond the VAE architecture.

1 Introduction

Scenario-based testing offers structured, reproducible evaluation for autonomous driving, while AI generation can model complex and rare driving situations. Yet limited transparency and output-focused checks leave open whether generated scenarios have meaningful, controllable representations and satisfy vehicle dynamics.

Motivation

  • Scenario-based simulation tests Advanced Driving Systems in structured, reproducible situations involving traffic participants, environments, infrastructure, and dynamic behavior.Traditional approaches can be resource-intensive and rely on mathematical models that oversimplify real-world complexity.Source: §1 · p. 1
  • AI scenario generation can learn from large driving datasets to create varied scenarios and explore rare or extreme events for stress-testing.The paper frames these opportunities alongside a need for more systematic validation as autonomous systems grow more complex.Source: §1 · pp. 1–2
  • AI-generated scenarios may look realistic while remaining unsuitable for industrial testing because output-level checks do not establish meaningful structure, controllability, or physical executability.The paper identifies limited transparency as a key reason for constrained industrial transferability.Source: §1 · p. 2

Contribution

  • The paper introduces a five-layer evaluation protocol for generative driving scenarios, demonstrated on a VAE and applicable to other architectures with appropriate representation-layer adjustments.The layers cover latent alignment, statistical baselines, controllability, layer-wise maneuver organization, and physical executability.Source: §1 · p. 2

2 Evaluation Setup

The evaluation requires a generative model, its training dataset, and a kinematic parameter extractor. The demonstration uses a guided VAE trained on labeled real-world highway maneuvers.

Evaluation Setup

  • The protocol uses a generative model, its training dataset, and a kinematic parameter extractor to probe physically meaningful representations.The extracted parameter set is a representative baseline that can be updated, extended, or replaced for a particular user, dataset, or application.Source: §2 · pp. 2–3

Model

  • The demonstration model is a ten-dimensional VAE with a guided latent dimension for scenario type and nine remaining dimensions with a standard Gaussian prior.The training objective combines reconstruction error, a KL term, and an auxiliary alignment loss.Source: §2 · p. 2

Dataset

  • The VAE is trained on approximately 44,600 real-world German-highway maneuvers, labeled into six safety-critical scenario types.The data totals about 1,735 recording hours and includes cut-in, cut-out, and cut-through maneuvers in both directions.Source: §2 · p. 2

3 Proposed Evaluation Protocol

The protocol evaluates generative scenario models across five complementary layers: four inspect representations and network layers, while the fifth checks generated trajectories against physical constraints. Together, these analyses examine what structure the model encodes, how it can be steered, and whether its outputs are physically plausible.

3 Proposed Evaluation Protocol

  • The five layers ask whether latent representations align with physical descriptors, add structure beyond data statistics, organize maneuvers through the decoder, support targeted interventions, and yield realizable trajectories.Layers 1–4 inspect internal representations and network layers; Layer 5 evaluates generated scenarios against physical constraints.Source: §3 · p. 3

3.1 Layer 1: Latent-Physical Probing

  • Layer 1 compares linear and nonlinear probes of latent vectors, using per-dimension regressions and random-vector controls to assess physical-parameter encoding and its distribution across dimensions.The probing gap indicates nonlinear entanglement; random-vector controls test whether high R2 reflects genuine representations rather than probing artifacts.Source: §3.1 · p. 3

3.2 Layer 2: Baseline Comparison

  • Layer 2 compares parameter probes on learned representations with probes on PCA scores to assess whether the model offers interpretability beyond statistical compression.PCA is treated as an optimal linear compression in mean-squared-error terms; a near-zero advantage over PCA means no interpretability benefit for that parameter.Source: §3.2 · pp. 3–4

3.3 Layer 3: Decoder Analysis

  • Layer 3 traces decoder representations with activation clustering and layer-wise probes of physical parameters and full trajectories.The analyses locate where scenario-level structure emerges and measure which quantities are decodable at each layer.Source: §3.3 · p. 4

3.4 Layer 4: Controllability via Latent Intervention

  • Layer 4 intervenes on latent dimensions using traversals, polynomial fits, and nudges to characterize directional consistency, response shape, and practical sensitivity.The combined measures indicate which dimension to adjust, whether steering requires nonlinear correction, and the nudge magnitude associated with a practically significant change.Source: §3.4 · pp. 4–5

3.5 Layer 5: Physical Plausibility Evaluation

  • Layer 5 checks generated trajectories against friction-circle, lateral-acceleration, and lateral-jerk constraints, reporting physical validity relative to the training data.Relative validity distinguishes generation artifacts from faithful reproduction of aggressive maneuvers present in the training distribution.Source: §3.5 · p. 5

4 Evaluation Results on Guided Variational Autoencoder

The guided-VAE evaluation examines how physical structure is represented across the latent space and decoder, whether latent interventions offer control, and whether generated trajectories satisfy physical constraints.

4.1 Layer 1: Latent-Physical Probing

  • Boundary conditions are almost perfectly linearly encoded in z, while phase-wise accelerations and durations require nonlinear probes for substantial recovery.Duration linear-probe R2 is at most 0.11; nonlinear probing gaps reach 0.55.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Layer 2: Baseline Comparison

  • The VAE beats PCA on four boundary conditions, especially lateral position, but PCA performs better for derived velocity quantities including ∆v and adur.VAE advantages are +0.086 for yinit and +0.113 for yfinal; PCA advantages are −0.266 for ∆v and −0.193 for adur.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Layer 3: Decoder Analysis

  • The six scenario types form distinct decoder-activation clusters from the first dense layer onward, with the CTL cluster between CIL and COL in projection.Left and right variants show mirrored activation signatures.Source: §4.3 · pp. 6–7, Figure 1 · p. 7
  • Boundary-parameter linear recoverability follows a U-shaped pattern across decoder depth, while full trajectories remain recoverable at every layer.Boundary-parameter R2 exceeds 0.95 at the latent input and first dense layer, collapses in middle convolutional layers, then exceeds 0.99 after the final dense layer; trajectory-level R2 is 0.99–1.00 throughout.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Layer 4: Controllability via Latent Intervention

  • Five latent dimensions show clear monotonic traversal responses: z0 for yinit, z4 for ∆v and adur, z5 for velocity, and z9 for yfinal.These dimensions match axis-aligned dimensions identified by probing.Source: §4.4 · p. 8, Table 4 · p. 8
  • At δ=0.2, practical sensitivity concentrates in five dimension–parameter pairs; global response fits are degree four, while scenario-specific fit degrees range from one to three.Global degree-four fits have R2 values from 0.34 to 0.77; for yfinal, cut-out left is linear while cut-through scenarios require cubic fits.Source: §4.4 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.5 Layer 5: Physical Plausibility

  • Five of six scenarios match training-data validity within ±5 percentage points; CTL validity drops 9.8 points, including a 21.7-point jerk-validity drop (98.3% to 76.6%).The authors suggest CTL's lower training-data frequency and jerk's sensitivity to small velocity errors may contribute.Source: §4.5 · p. 8, Table 8 · p. 9

5 Discussion

The protocol reveals representational and physical behaviors that output-level evaluations miss, while a detailed comparison with WOSAC and Waymo E2E is provided in Section A.9.

5 Discussion

  • The protocol shows that high output quality does not guarantee physical consistency, motivating physics-informed training.Layer 5 evaluates whether generated trajectories are physically executable.Source: §5 · p. 9
  • Layer 1 finds partial kinematic alignment, while Layer 2 indicates learned structure beyond statistical baselines that remains dataset-biased.Partial alignment can make intervention harder because target quantities may not map to clean latent directions.Source: §5 · p. 9
  • Layer 3 shows information reorganizes across decoder depth, suggesting interventions may work better where target properties are constructed.Layer 4 finds some physical quantities linearly controllable, while others need nonlinear or scenario-dependent mappings, making latent steering unreliable.Source: §5 · p. 9
  • A detailed comparison of the protocol with WOSAC and Waymo E2E appears in Section A.9.The discussion notes that standard evaluations can suggest competitive generation performance without revealing transparent, controllable, or physically valid representations.Source: §5 · p. 9

6 Conclusion

The conclusion presents a five-layer protocol for evaluating generative scenario models beyond output quality and reports that it exposes behaviors standard metrics miss. It recommends evaluating structural, controllability, and physical dimensions, alongside design improvements within the studied setting.

6 Conclusion

  • Applied to baseline architectures trained on highway maneuver data, the protocol revealed partial kinematic alignment, nonlinear and scenario-dependent control responses, and physically invalid outputs.These behaviors were not captured by standard metrics.Source: §6 · p. 9
  • The five-layer protocol evaluates whether models learn meaningful driving structure, respond controllably to interventions, and generate physically executable trajectories.Source: §6 · p. 9
  • The findings suggest evaluating structural, controllability, and physical dimensions and considering stronger kinematic supervision, scenario-calibrated control, layer-specific interventions, and physics-informed training.The conclusion frames these as directions for future generative driving models.Source: §6 · p. 9

A.1 Related Work

Prior work offers output-fidelity and interpretability tools, but the supplied review identifies a gap in systematic analysis of generative maneuver models grounded in physical kinematics and output-level plausibility.

A.1 Related Work

  • Learning-based scenario models can generate diverse, realistic scenarios, but their high-dimensional latent dimensions often lack recognizable kinematic meanings, complicating targeted generation and verification.Parameterized expert models are interpretable but labor-intensive to extend and struggle to cover rare scenarios [26–28].Source: §A.1 · p. 12
  • Output evaluations use reconstruction error, distributional similarity, ADE, and FDE [33], while some compare acceleration and jerk distributions [34] [35].Distribution comparisons do not test executability under tire-friction, actuator, and stability limits; links to ADAS validation outcomes remain limited [14].Source: §A.1 · p. 12
  • Interpretability tools include latent traversal [16], linear probing [15], disentanglement metrics [36], and mechanistic analysis, mostly developed outside autonomous-driving generation.Driving interpretability work has focused mainly on perception and decision-making, with some latent analysis of discriminative maneuver prediction models [39] [40].Source: §A.1 · p. 12
  • The review identifies no prior systematic interpretability analysis of generative maneuver models combining physically meaningful kinematics with output-level physical-plausibility checks.Source: §A.1 · p. 12

A.2 Mathematical Model

The mathematical model supplies physically interpretable maneuver quantities and describes highway interactions through scenario types and three temporal phases: preparation, execution, and stabilization.

A.2 Mathematical Model

  • A parametric model extracts compact, physically interpretable maneuver quantities to correlate with the generative model’s latent representation.Source: §A.2 · p. 13
  • The model covers cut-in, cut-out, and cut-through highway interactions, with the actor vehicle moving laterally relative to the ego vehicle.Figure 2 gives an overview of the considered scenario types.Source: §A.2 · p. 13, Figure 2 · p. 13
  • The pre-maneuver phase captures approach and preparation, when the actor establishes conditions such as gap, speed, and lateral position.Lateral position remains stable while longitudinal dynamics may already change with intent.Source: §A.2 · p. 13
  • The active maneuver phase captures lateral displacement, modeled by a sigmoid for its S-shaped profile, when lateral rate, time-to-collision, and relative velocity reach critical values.The sigmoid is described as a physically motivated choice based on progressive steering input.Source: §A.2 · p. 13
  • The post-maneuver phase captures stabilization, including whether the actor maintains speed, decelerates, or accelerates after entering the new lane.Post-maneuver longitudinal behavior affects the ego vehicle’s sustained risk exposure.Source: §A.2 · p. 13

A.2.1 Model Definition

A maneuver has lateral-position and longitudinal-velocity channels over horizon T, divided into pre-maneuver, active-maneuver, and post-maneuver phases.

A.2.1 Model Definition

  • Each maneuver has two channels over horizon T: lateral position y(t) relative to the ego-lane center and longitudinal velocity v(t).Source: §A.2.1 · p. 14
  • t1 marks the end of the pre-maneuver phase at tpre.Source: §A.2.1 · p. 14
  • t2 ends the active-maneuver phase at tpre + tdur, and T adds the post-maneuver duration tpost.Source: §A.2.1 · p. 14

A.2.2 Lateral Motion Model

Lateral displacement is modeled as a sigmoid-based lane transition between constant pre- and post-maneuver positions, with exact boundary continuity and a scalar measure of maneuver aggressiveness.

A.2.2 Lateral Motion Model

  • The lateral-motion model holds the initial position before the maneuver, transitions smoothly with a normalized sigmoid, then holds the final position; Figure 3 depicts idealized lateral motion.Source: §A.2.2 · p. 14, Figure 3 · p. 14
  • Normalization gives y(t1) = yinit and y(t2) = yfinal exactly; the transition midpoint is t1 + tdur/2, with σ(±3) ≈ 0.95 and σ(∓3) ≈ 0.05.Source: §A.2.2 · p. 14
  • Mean lateral rate is displacement divided by active-maneuver duration; its magnitude indicates aggressiveness and its sign indicates direction, positive leftward and negative rightward.Source: §A.2.2 · p. 14

A.2.3 Longitudinal Velocity Model

Longitudinal velocity uses piecewise constant accelerations across maneuver phases, supporting extraction of intuitive physical parameters while retaining acceleration discontinuities at phase boundaries.

A.2.3 Longitudinal Velocity Model

  • Velocity uses piecewise constant accelerations over the three phases, with boundary velocities determined by continuity; Figure 4 depicts idealized longitudinal motion.Source: §A.2.3 · p. 15, Figure 4 · p. 15
  • Total velocity change is ∆v = vend − vstart = apre · tpre + adur · tdur + apost · tpost.Source: §A.2.3 · p. 15
  • The piecewise velocity profile is continuous (C0), but acceleration changes instantaneously at phase boundaries; quintic polynomials or jerk-limited ramps can provide C2 continuity.Source: §A.2.3 · p. 15
  • The parametric model is used only to extract maneuver parameters, which depend on macro-structure rather than transition smoothness at phase boundaries.Source: §A.2.3 · p. 15
  • The model has nine independent parameters and three derived quantities that add no degrees of freedom, including final velocity, velocity change, and lane-change aggressiveness.Source: §A.2.3 · p. 15, Table 9 · p. 16

A.2.4 Parameter Extraction

This section examines trajectory-parameter extraction and the evaluation of learned representations in a guided VAE, including fit limitations, latent organization, layer-wise analyses, robustness, and design choices. It also demonstrates the protocol on simple GAN, transformer, and GRU examples, reporting selected layers and probed representations.

A.2.4 Parameter Extraction

  • Parameter extraction identifies lateral phase boundaries using equal thirds for lane-keeping or 5%-95% shift thresholds otherwise, then reads endpoint geometry and fits phase accelerations.The 5%-95% thresholds isolate lateral motion without a sigmoid fit, and shift sign determines direction. Acceleration estimates use independent least-squares fits, equivalent to maximum-likelihood fitting under additive Gaussian noise; derived quantities are computed in closed form.Source: §A.2.4 · p. 16

A.2.5 Analysis of Three-phase Model Fit

  • The three-phase model fits lateral position moderately (mean R2 ≈0.85) but underfits velocity (mean R2 ≈0.21, DW ≈0).Its piecewise-linear velocity assumption smooths away within-phase curvature and leaves it in residuals; the extracted parameters summarize average acceleration rather than actual time-varying behavior.Source: §A.2.5 · pp. 16–17

A.3 Dataset- Additional Details

  • The dataset combines 50 Hz ECU and surrounding-traffic signals across six maneuver types, while the guided VAE assigns scenario type to a latent dimension to support controllable generation.Maneuvers are classified using longitudinal-distance rules followed by lateral-distance TSF refinement. The model uses ztype for class identity and zrest for remaining trajectory variability, with a class-conditional prior and alignment loss.Source: §A.3 · p. 17, §A.4 · p. 18

A.4.1 Model Performance

  • Validation results indicate preserved trajectory fidelity and scenario consistency, while z0 forms class-specific latent modes and free dimensions z2, z5, and z9 show multimodality.Generated and measured samples closely overlap in PCA projections; the passage attributes CTL's higher MSE to its smaller sample count. z5's distribution closely matches training velocity profiles without explicit supervision.Source: §A.4.1 · p. 19, §A.4.2 · p. 20

A.5 Robustness Check in Train/Test Split

  • Under a recording-level 70/30 split, z0 classification error is 0.67%, compared with 1.0% for the maneuver-level split.Whole recordings were partitioned with stratification by dominant scenario, preventing any recording from contributing segments to both sets.Source: §A.5 · p. 20, Table 14 · p. 20

A.6.1 Layer 1

  • Latent encoding is primarily but not strictly axis-aligned: z5 carries overall speed, while timing and phase accelerations are distributed and nonlinear.z5 predicts vstart and vend at nearly identical R2 ≈0.75; yfinal is carried by z9, while ∆v shares z4 with adur. No single dimension reaches R2 = 0.06 for phase durations.Source: §A.6.1 · pp. 21–22

A.6.2 Layer 3

  • Layer-wise probes recover full trajectories with R^2 ∈[0.99, 1.00], while complex CTL velocity detail emerges only in later convolutional layers.COL profiles are recoverable from the first dense layer onward; for CTL, the deceleration trough forms at intermediate layers and aligns at the final convolutional layer. Random matched Gaussian inputs yield roughly 4× higher RMSE, supporting that probe outputs are signal-driven.Source: §A.6.2 · pp. 23–25

A.7.1 Architectural Choice

  • The β-VAE did not improve parameter separation, and latent dimension 10 was retained as the most stable overall choice among dimensions 6, 10, and 12.Dimension 12 improved some velocity terms but performed worse for ∆v and adur; dimension 6 performed well on some parameters but was less balanced overall.Source: §A.7.1 · p. 25, §A.7.2 · p. 25, Table 18 · p. 25

A.8.1 Evaluation Results on GAN

  • GAN probes recover velocity parameters strongly (R2lin > 0.8), while Layer 2 finds vstart concentrated on one latent dimension at 0.67 versus 0.61 for PCA.Lateral and timing parameters are weaker, and lateral and acceleration parameters are spread across multiple latent dimensions and weaker per-axis than PCA.Source: §A.8.1 · p. 26, Table 22 · p. 27, Table 23 · p. 27
  • GAN trajectories fail physical-plausibility checks in all six classes, with PVR dropping 69–89 percentage points versus training and ending in single digits.Lateral jerk is a primary failure (pjy 14–26%); longitudinal acceleration and friction also fail roughly half the time, while lateral velocity is largely satisfied.Source: §A.8.1 · p. 26, Table 24 · p. 28

A.8.2 Evaluation Results on Transformer-based

  • The transformer’s mean-pooled state already encodes several trajectory features nearly linearly; SAE features add no clear improvement, and PCA matches or slightly exceeds the raw representation on strongly encoded variables.For both transformer and SAE, Layer 5 shows large physical-plausibility drops in every scenario; results vary with prefix length. The lightly trained architectures are illustrative, not strong benchmarks.Source: §A.8.2 · pp. 27–28, Table 27 · p. 29

A.8.3 Evaluation Results on World-Model-Inspired GRU Rollout

  • In the GRU rollout, beginning-of-trajectory variables decode best at ht10, end variables at ht99, and middle variables around ht50.At ht99, the hidden state matches PCA on strongly encoded end variables but underperforms it on beginning-of-trajectory properties; the model also has low physical plausibility.Source: §A.8.3 · p. 28, Table 32 · p. 31
  • At ht99, the GRU matches PCA for yfinal, vend, and ∆v, but loses early-trajectory information; its generated trajectories show low physical plausibility.The GRU progressively overwrites early state, whereas PCA retains information from the full flattened trajectory.Source: §A.8.3 · p. 29, Table 34 · p. 32, Table 35 · p. 32

A.9 Comparison of WOSAC and Waymo E2E benchmarks with the proposed five-layer evaluation protocol

  • The VAE ranks best on both WOSAC kinematic NLL and Layer 5 PVR, while the GAN, transformer, and world model have near-zero PVR despite the GAN’s lower WOSAC NLL than the other two.WOSAC provides a relative distributional score; PVR reports the percentage of generated trajectories satisfying physical-dynamics constraints. The comparison uses WOSAC’s kinematic subset and excludes interaction/map metrics such as collision, off-road, and TTC.Source: §A.9 · pp. 29–30, Key observations · p. 30, Table 37 · p. 33

A.10 How to Apply the Protocol to a Model

  • The protocol aims to locate kinematic information in a model and determine whether it can explain or influence generation; it requires a trained model, held-out real trajectories, and a physical-parameter extractor.The extractor is not fixed: the protocol is stated to work with any expert model.Source: §A.10 · p. 30, §A.10.1 · p. 31

A.10.2 Procedure

  • Application requires defining z and the generation path g, constructing latent and parameter matrices, then running Layers 1–5 with architecture-specific representations and interventions.For architectures without a single explicit latent, the procedure gives representative vectors such as the final recurrent hidden state or transformer mean-pooled state, and asks for at least one robustness alternative.Source: §A.10.2 · p. 31

Related papers

Loading 2610.01581v1…