PaperlayerView exact revision on arXiv

Source-linked AI summary

C-DFF: Concept-Level Diagnostics for Mechanism Parity in Simulation-Based Validation of Autonomous Driving Perception

Danial Safaei, Siddartha Khastgir, Matthew Higgins, Jeroen Ploeg, Mohsen Alirezaei, Son Tong, Ludo Gielen

arXiv:2610.33760v1cs.CVcs.AIcs.RO

TL;DR

Similar outputs on real and synthetic inputs do not establish that an autonomous-driving perception system relies on the same visual evidence. C-DFF compares decisive evidence in a shared semantic concept space and, across three evaluated SUTs, distinguishes their mechanism-parity patterns while identifying concept shifts for debugging.

  • Problem

    Matching real–synthetic task outputs does not establish whether a system relies on the same evidence, leaving a mechanism-parity question for simulation-based validation.

    Source: §I · p. 1

  • Method

    C-DFF maps decisive evidence from a frozen SUT into semantic concepts using off-the-shelf segmentation and compares concept-importance vectors for paired real–synthetic inputs.

    Source: §I · p. 1

  • Results

    Across three frozen SUTs, C-DFF differentiated the systems and exposed concept-level shifts; output-value context metrics showed that output agreement does not by itself establish mechanism parity.

    Source: §I · p. 1, A. Main quantitative result · p. 4, §VII · p. 7

  • Takeaways & Limitations

    C-DFF provides semantic diagnostics that the paper reports as useful for validation triage and simulator debugging on KITTI–Virtual KITTI 2.

    Source: §VII · p. 7

    Imperfect pose, timing, or object-layout correspondence can conflate domain effects with pairing noise, so pair-quality filtering remains important for future datasets.

    Source: B. Threats to validity and practical… · p. 7

Abstract

from arXiv · show

Simulation is widely used in autonomous-driving validation, but matching task outputs between real and synthetic images does not guarantee that a system relies on the same visual evidence. This creates a validation gap for transfer, especially in safety-critical perception functions. We present Concept-Decisive Feature Fidelity (C-DFF), a concept-level mechanism-parity diagnostic for paired real-synthetic evaluation. Starting from decisive evidence extracted from a frozen system-under-test (SUT), C-DFF projects that evidence into a shared semantic concept space using off-the-shelf segmentation and compares real and synthetic concept-importance vectors. The result is not only a scalar mechanism-parity score, but also a concept-shift diagnostic that identifies which scene concepts become more or less decisive in simulation. We evaluate C-DFF on KITTI $\leftrightarrow$ Virtual KITTI 2 with 2126 matched pairs across three frozen SUTs (steering regression, YOLOP drivable-area, YOLOP lane-line). Across these tasks, C-DFF is SUT-discriminative and yields interpretable concept-level summaries that support validation triage and simulator debugging. When reported alongside task-specific output-value context metrics, C-DFF shows that output agreement does not by itself establish mechanism parity. These results position C-DFF as a practical semantic diagnostic layer for validation and verification workflows in simulation-based AV testing.

I. Introduction

Output agreement between real and synthetic driving scenes can conceal shifts in the evidence a perception system uses. C-DFF translates decisive evidence into semantic concept comparisons and reports concept shifts for validation and debugging.

I. Introduction

  • C-DFF addresses the gap between matching real–synthetic outputs and relying on the same evidence, which matters because spurious or non-transferable cues can weaken simulation-based conclusions.The paper frames this as a mechanism-parity question for validation and verification.Source: §I · p. 1
  • C-DFF projects decisive evidence from a frozen SUT into a shared semantic concept vocabulary using off-the-shelf segmentation, then compares real and synthetic concept-importance vectors.The pipeline produces a scalar mechanism-parity score and concept-shift summaries.Source: §I · p. 1, Figure 1 · p. 2, A. Contributions • · p. 2

A. Contributions •

  • C-DFF was evaluated on 2126 matched KITTI–Virtual KITTI 2 pairs across steering, YOLOP drivable-area, and YOLOP lane-line SUTs, and was reported as SUT-discriminative.The study reports task-specific output-value context metrics alongside C-DFF.Source: §I · p. 1, A. Contributions • · p. 2

I. Introduction

  • The paper positions C-DFF as a semantic diagnostic layer for interpreting mechanism mismatches, unlike DFF’s use of decisive-map comparisons in a synthetic-data calibration loop.C-DFF does not introduce a generator-calibration objective.Source: §I · p. 2

A. Contributions •

  • Per-pair and dataset-level concept-shift summaries identify which scene concepts become more or less decisive in synthetic imagery.These diagnostics are intended to support simulator, renderer, and synthetic-data debugging.Source: §I · pp. 1–2, A. Contributions • · p. 2
  • C-DFF is proposed for validation workflows as a scalar gate plus a ranked semantic diagnostic for screening, triage, and requirement-style monitoring.The contribution list also reports the multi-SUT evaluation and task-specific output-value context metrics.Source: A. Contributions • · p. 2

II. Problem Formulation

The formulation evaluates concept-level mechanism parity for frozen systems on matched real–synthetic pairs. Pair correspondence is a first-order requirement because misalignment can inflate the measured distance.

II. Problem Formulation

  • The paired-dataset assumption is that each synthetic image represents the same underlying scenario as its real counterpart, with viewpoint, sequence identity, and frame index matched as closely as the benchmark permits.Pairing quality is described as a first-order requirement.Source: §II · pp. 2–3
  • Each SUT is evaluated with fixed weights and preprocessing; C-DFF assesses and diagnoses the system rather than training or adapting it.Source: §II · p. 3
  • For each frozen SUT and matched real–synthetic pair, the goal is to measure whether the system relies on similar semantic evidence across domains.The outputs are a scalar mechanism-parity score and a concept-shift diagnostic indicating which scene concepts changed in importance.Source: §II · p. 3

III. Method: Concept-Decisive Feature Fidelity (C-DFF)

C-DFF extracts decisive evidence from a frozen system-under-test, maps it into semantic concepts, and compares real and synthetic concept-importance vectors. Its evidence extractor uses direct image-space perturbation to produce diagnostic maps without updating SUT weights.

III. Method: Concept-Decisive Feature Fidelity (C-DFF)

  • C-DFF estimates decisive evidence, projects it into a semantic concept space, and compares real and synthetic concept-importance vectors.Source: §III · p. 3

A. Decisive evidence maps

  • A counterfactual image is optimized to move a task-dependent scalar toward a target while keeping perturbations sparse and smooth; the resulting channel-mean absolute perturbation forms the decisive map.The optimization extracts a diagnostic map for the fixed SUT and does not update its weights.Source: A. Decisive evidence maps · p. 3
  • Unlike DFF [4], this paper uses a direct perturbation-based objective because its focus is semantic diagnosis rather than generator calibration or estimator comparison.Source: A. Decisive evidence maps · p. 3
  • Steering uses the scalar steering output, while YOLOP drivable-area and lane-line use the selected head’s mean probability mask as the task-dependent summary.One counterfactual optimization is run per image and SUT.Source: A. Decisive evidence maps · p. 3

B. Semantic concept segmentation

C-DFF assigns semantic labels with an off-the-shelf segmenter and aligns those labels with decisive evidence on the geometry consumed by the SUT. The aligned maps are pooled to a common grid for concept aggregation.

B. Semantic concept segmentation

  • An off-the-shelf semantic segmenter supplies per-pixel concepts; the concrete setup uses Mask2Former with the Cityscapes-19 taxonomy, without training or tuning the segmenter.Source: B. Semantic concept segmentation · p. 3

C. Deterministic alignment and pooling

  • Concept labels are warped through the SUT’s deterministic preprocessing transform with nearest-neighbor interpolation; on a 16 × 16 pooling grid, masks are average-pooled and decisive evidence is resized using nearest-neighbor interpolation.Source: C. Deterministic alignment and pooling · p. 3

D. Concept-importance vector

The concept-importance vector records area-normalized decisive evidence for each semantic concept. Each component measures the average decisive mass assigned to that concept on the pooled grid.

D. Concept-importance vector

  • Each concept’s importance is its average decisive mass on the pooled grid, so larger values mean that concept’s pixels carry more perturbation evidence needed to change the fixed SUT output.Source: D. Concept-importance vector · p. 3

E. C-DFF distance between real and synthetic

C-DFF compares real and synthetic concept-importance vectors with mean squared error. The paper also reports operational pass rates and concept-shift summaries to reveal which concepts become more or less decisive in synthetic imagery.

E. C-DFF distance between real and synthetic

  • Mean squared error is the primary C-DFF distance between real and synthetic concept-importance vectors, chosen as a symmetric measure that preserves magnitude and penalizes larger concept changes more strongly.Source: E. C-DFF distance between real and… · pp. 3–4

F. Thresholding and pass-rate

  • Held-out pass rates count test pairs whose C-DFF is at or below a threshold calibrated from the 90th or 95th percentile on a calibration split.These thresholds are operational summaries for a fixed split, not safety guarantees.Source: F. Thresholding and pass-rate · p. 4

G. Diagnostics: concept shift vectors

IV. Experimental Setup

The evaluation uses 2,126 frame-matched KITTI–Virtual KITTI 2 pairs and three frozen perception systems spanning steering, drivable-area, and lane-line tasks.

IV. Experimental Setup

  • 2,126 frame-level pairs are matched by shared sequence ID and frame index across five sequences, with three sequences for calibration and two held out.The images use a front-facing monocular camera at 1248 × 376; pairing noise is treated as a validity threat.Source: A. Data and frozen SUTs · p. 4
  • The frozen SUTs are a PilotNet-style steering regressor and YOLOP drivable-area and lane-line heads.Steering inputs are resized to 200 × 66 and YOLOP inputs to 640 × 384; both are normalized to [0, 1], with no inference augmentation.Source: A. Data and frozen SUTs · p. 4

B. Implementation details

The implementation combines frozen-SUT counterfactual perturbation explanations with off-the-shelf semantic segmentation, and reports C-DFF alongside task-specific output-value context metrics. The main held-out result is that C-DFF separates the three SUTs, while output agreement alone does not establish mechanism parity.

B. Implementation details

  • Counterfactual explanations use direct image-space perturbation optimization, converted to decisive maps by channel-mean absolute difference, with settings fixed before held-out evaluation.The off-the-shelf segmenter uses Cityscapes-19 concepts and is not trained or adapted; its predictions are warped through the SUT transform before pooling.Source: B. Implementation details · p. 4

C. Metrics reported

  • C-DFF is the primary metric, while steering, drivable-area, and lane-line output-value metrics provide within-SUT context rather than mechanism-parity measurements.The output-value metrics differ by task, so their absolute values are not intended for cross-SUT ranking.Source: C. Metrics reported · p. 4

A. Main quantitative result

  • C-DFF separates the three frozen SUTs on the held-out split; task-specific output-value metrics are context only and do not establish mechanism parity.The table reports held-out C-DFF, calibration-split thresholds, pass rates, and task-specific OV; OV is not intended for cross-SUT ranking.Source: A. Main quantitative result · p. 4, Table II · p. 5

B. Diagnostics and additional analyses

Additional analyses examine segmentation-induced shifts, calibrated C-DFF distributions, and qualitative per-pair diagnostics. The concept-shift summaries identify which scene concepts gain or lose decisiveness, while the qualitative panels are triage aids rather than primary evidence.

B. Diagnostics and additional analyses

  • The synthetic-side label-source diagnostic recomputes concept importance using VKITTI2 ground-truth labels while holding the decisive map fixed.It covers only a held-out cache subset, maps VKITTI2 classes approximately to Cityscapes concepts, and ignores unmapped labels.Source: B. Diagnostics and additional analyses · p. 5
  • Held-out C-DFF percentiles are reported alongside calibrated thresholds, which are descriptive split-specific summaries rather than universal safety bounds.The reported held-out q90 values are Steering 4.36 × 10^-4, YOLOP–DA 1.67 × 10^-4, and YOLOP–LL 4.53 × 10^-5.Source: B. Diagnostics and additional analyses · p. 5, Figure 2 · p. 5
  • Building becomes less decisive for both YOLOP heads; the strongest positive and negative mean concept shifts otherwise differ across Steering, YOLOP-DA, and YOLOP-LL.The ranked shifts provide semantic targets for simulator and synthetic-data debugging.Source: B. Diagnostics and additional analyses · p. 5
  • Low-C-DFF examples typically preserve decisiveness over road-aligned concepts, while high-C-DFF examples show redistribution toward non-transferable or overemphasized concepts.The qualitative panels are intended as semantic triage aids, not primary evidence.Source: B. Diagnostics and additional analyses · p. 5

VI. Discussion

C-DFF supports screening with a scalar score and debugging with ranked concept shifts, while its interpretation depends on paired data and diagnostic components with practical limits. These constraints bound how its results should be used in validation.

A. Operational interpretation and workflow

  • Held-out mean concept shifts give simulator, renderer, or synthetic-data debugging a ranked list of concepts over- or under-emphasized for a given SUT.Table V describes the practical output as a scalar gate plus a ranked concept-shift diagnostic.Source: A. Operational interpretation and… · p. 5, Table V · p. 7
  • DFF compares decisive maps and can support calibration, whereas C-DFF translates mechanism mismatch into semantic summaries for debugging and V&V triage.The two methods can be used together, with DFF as a fine-grained signal and C-DFF as its semantic interpretation layer.Source: A. Operational interpretation and… · p. 5
  • C-DFF can screen or support regression testing with a thresholded score, but thresholds are scene-dependent and not universal bounds.The workflow also supports concept-level checks for monitoring critical concepts such as road, lane, and sidewalk.Source: A. Operational interpretation and… · pp. 5–6

B. Threats to validity and practical constraints

  • Perturbation-based decisive maps can vary with hyperparameters and random seeds despite fixed optimization settings, clamping, and pooling.The decisive-map extractor is treated as a modular approximation, not a ground-truth causal oracle.Source: B. Threats to validity and practical… · pp. 6–7
  • Figure 3 encodes mean concept shifts by SUT and concept, with signs indicating whether a concept becomes more or less decisive in synthetic imagery.Table IV lists the largest shifts numerically; selected Figure 4 examples aid semantic triage but are not prevalence estimates.Source: Figure 3 · p. 6, Figure 4 · p. 6, Table IV · p. 7
  • Concept scores depend on the segmenter, whose errors or domain shift can bias scores and inflate shifts; the ground-truth check is one-sided because equivalent real-side labels are unavailable.The comparison with VKITTI2 ground truth covers a held-out synthetic subset.Source: B. Threats to validity and practical… · p. 7
  • Imperfect real–synthetic correspondence can make C-DFF conflate domain effects with pairing noise, so pair-quality filtering matters beyond the cloned-trajectory benchmark setting.The paper identifies viewpoint, sequence identity, and frame index as pairing considerations.Source: B. Threats to validity and practical… · p. 7
  • Perturbation-based CF-XAI is computationally expensive, making cached decisive maps and segmentations essential for scaling to larger benchmarks or more SUTs.Source: B. Threats to validity and practical… · p. 7

VII. Conclusion

C-DFF is a semantic diagnostic layer for paired real–synthetic validation of frozen autonomous-driving perception models. On KITTI ↔ Virtual KITTI 2, it differentiated evaluated SUTs and exposed concept shifts useful for validation triage and simulator debugging.

  • On KITTI ↔ Virtual KITTI 2, C-DFF differentiated the evaluated SUTs and exposed concept-level shifts useful for validation triage and simulator debugging.Source: §VII · p. 7

Related papers

Loading 2610.33760v1…