PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty Evaluation to Support Safety of the Intended Functionality Analysis for Determining the Applicability of Performance Targets Across Triggering Conditions in ML-Based LiDAR Object Detection

Milin Patel, Rolf Jung

arXiv:2610.35092v1cs.CVcs.LGcs.RO

TL;DR

A threshold selected under one condition carries a statistical guarantee only for data distributed like its selection data, yet SOTIF analysis must evaluate triggering conditions. This paper tests transferred thresholds and derives a lower bound on independent units; its PointPillars evaluation finds condition-dependent target failures and shows that sequence-level evidence is constrained by the available data.

  • Problem

    A threshold's guarantee applies only to data distributed like the selection data, leaving open whether it meets its target after transfer to a triggering condition.

    Source: §1 · p. 2

  • Method

    The paper tests transferred thresholds for target applicability at a stated error probability and derives a lower bound on the independent units needed to show the target is met.

    Source: §3.5 · p. 11, §7 · p. 21

  • Results

    For PointPillars ensembles, frame-level transfers produced not-applicable outcomes, while sequence-level evaluation yielded almost none and required more units than the validation sets provide at 10^-2.

    Source: §6.1 · p. 19, §7 · p. 21

  • Takeaways & Limitations

    A target met under one condition counts as evidence under another only after testing there, alongside retention and the number of independent units.

    Source: §6.1 · p. 19, §7 · p. 21

    With dependent frames, recorded sequences are the independent units, and the minimum required at 10^-2 exceeds the KITTI and nuScenes validation sets.

    Source: §6.1 · p. 19

Abstract

from arXiv · show

For safety of the intended functionality (SOTIF) analysis according to ISO 21448, a machine learning object detector for light detection and ranging (LiDAR) point clouds can meet a performance target, a limit on the mean per-frame share of wrong detections among accepted detections, by rejecting detections above an uncertainty threshold. A threshold selected on data of one condition carries a statistical guarantee only for data distributed like its selection data, whereas SOTIF analysis covers every triggering condition. This paper presents an uncertainty evaluation in which the threshold is transferred into a triggering condition, a test decides at a stated error probability whether the target is applicable there, not applicable or not determined, and a lower bound counts the independent units an applicable outcome requires. For PointPillars ensembles on KITTI and nuScenes, counted in recorded sequences, as the dependence between frames requires, neither dataset held enough independent units for any target limit, and no transfer had a not applicable outcome. Counted in frames, the same data gave not applicable outcomes at limits fixed in advance after transfers into far distance, into rain and between two separately trained detectors, thresholds that met the target limit after a transfer rejected most correct detections, and in rain the area under the receiver operating characteristic curve did not decrease. A performance target met under one condition therefore counts as evidence under a triggering condition only after a test there, reported with its share of accepted correct detections and its number of independent recorded sequences.

1 Introduction

A threshold selected under one condition guarantees the false acceptance proportion only for data distributed like its selection data, while SOTIF analysis requires evaluating performance under triggering conditions. The paper tests threshold transfers between conditions and examines the evidence and data needed to judge whether a performance target applies.

1 Introduction

  • A wrong detection can prompt braking or steering, and rejecting uncertain detections to bound the false acceptance proportion also withholds rejected correct detections from the motion planner.The target bounds the mean per-frame share of wrong detections among accepted detections.Source: §1 · p. 2

1.1 Research Questions

  • The research questions ask whether transferred thresholds preserve the target, which detection changes accompany failures and whether AUROC shows them, and how many independent units are needed.The comparisons cover distance, weather, and dataset changes.Source: §1.1 · p. 3

1.2 Contributions and Scope

  • The paper contributes an applicability test and independent-unit lower bound, evaluates transfers across conditions, and examines detection changes and the limits the datasets can demonstrate.The dataset change is an experimental control because it also changes the sensor, location, object mix, and trained detector; the paper proposes no new indicator, detector, or threshold-selection method.Source: §1.1 · p. 3, §1.2 · pp. 4–5

2 Related Work

Prior work addresses SOTIF performance targets, statistical risk control, LiDAR uncertainty, and condition-specific perception evidence, but the cited work does not test whether a threshold meeting its limit in one condition still meets it in another.

2.1 Performance Targets in SOTIF Processes

  • Prior SOTIF work discusses acceptance criteria, performance targets, hazardous scenarios, and safety arguments, but does not examine target transfer across conditions.The paper frames its question as whether a target met under one condition is also met under another.Source: §2.1 · p. 5

2.2 Statistical Risk Control and Its Validity Under Distribution Shift

  • LTT and related risk-control methods provide guarantees tied to calibration data, while subset-specific guarantees can become uninformative when subsets are small.This paper uses LTT without extending it and tests its threshold after transfer to another condition.Source: §2.2 · pp. 5–6

2.3 Uncertainty of LiDAR Object Detectors

  • Prior LiDAR work studies detector uncertainty, calibration under dataset shift, and cross-country accuracy, but does not test thresholds across conditions.Source: §2.3 · p. 6

2.4 Condition-Specific Evidence for Perception

  • Studies of condition-specific perception address risk decomposition, operating situations, corruption severity, distance adaptation, sensor degradation, or triggering-condition search, but none tests cross-condition threshold compliance.The paper reports no other such study in a documented search.Source: §2.4 · p. 6

3 Test of the Applicability of a Performance Target Across Conditions

The applicability test transfers a threshold selected in one condition unchanged to another and uses an FAP interval to decide whether the target applies there. Its procedure covers uncertainty measurement, threshold selection, outcome interpretation, possible causes of a limit being exceeded, and the minimum independent-unit count.

3 Test of the Applicability of a Performance Target Across Conditions

  • A transferred threshold is tested in the use condition, where an FAP confidence interval determines whether the target is applicable, not applicable, or not determined.If no threshold is selected in the selection condition, the test reports that outcome instead.Source: §3 · p. 6, §3.3 · p. 10

3.1 Uncertainty Indicators and the False Acceptance Proportion

  • The FAP is the mean across frames of each frame’s share of false acceptances among accepted detections; a frame with no accepted detections contributes zero.The uncertainty indicators are computed from ensemble-member outputs, and a correct detection is accepted when its indicator does not exceed the threshold.Source: §3.1 · pp. 7–8

3.2 Selection of a Threshold per Condition

  • Candidate thresholds are tested in ascending order with an HB p-value, and selection stops at the first failure; the guarantee assumes independent, identically distributed frames.The selected threshold is the last passing candidate before that failure, and its guarantee applies to the selection condition’s distribution.Source: §3.2 · pp. 8–9

3.3 Decision on Applicability in the Condition of Use

  • The target is applicable when U≤α, not applicable when L>α, and not determined when L≤α<U; each erroneous decided outcome has probability at most δ/2 under the stated independence assumptions.The confidence interval is computed on test data separate from threshold selection data.Source: §3.3 · p. 10

3.4 Changes in the Detections That Can Lead to an Exceeded Limit

  • A limit can be exceeded after transfer through weaker separation, more wrong detections, shifted indicator values, or a different spread of detections across frames.AUROC reflects separation but not the wrong-detection share or frame-level spread, so it may remain unchanged despite an exceeded limit.Source: §3.4 · p. 10

3.5 Minimum Number of Independent Units to Show That a Limit Is Met

  • At δ′=0.05, showing a limit is met requires at least about 3/α independent units per condition and part; this necessary zero-failure bound does not apply to a not-applicable outcome.The bound assumes the most favorable case of units without false acceptances; any false acceptance raises the required count.Source: §3.5 · p. 11

4 Detectors, Datasets and Conditions of the Study

The study evaluates PointPillars ensembles on KITTI and nuScenes across distance, weather, and dataset-and-detector changes. Its condition assignments, data splits, independent-unit choices, and pre-specified limits define how threshold transfers are tested.

4.1 Ensemble Detectors and Rule for Merging Member Boxes

  • Each ensemble uses four separately seeded PointPillars networks and forms a detection only when at least three members contribute a matching box.The vote is an initial rejection rule, applied before uncertainty thresholds.Source: §4.1 · p. 11

4.2 Assignment of Distance, Weather and Dataset Conditions

  • Far distance and rain are potential triggering conditions; distance is assigned from each box centre, while rain scenes are labeled from nuScenes descriptions.Near and far detections can come from the same frame; rain labels do not specify intensity and may also differ in location and traffic.Source: §4.2 · p. 12
  • The dataset-and-detector transfer compares separately trained KITTI and nuScenes ensembles, so it does not isolate a single environmental change.Distance is also examined within each dataset, yielding four pairs of conditions across the three change types.Source: §4.2 · p. 12

4.3 Division of the Frames and Choice of the Independent Unit

  • Frames are evaluated as independent units when randomly split, or whole recorded sequences are split and counted as independent units to address within-sequence dependence.The sequence-level estimate weights each sequence equally; a third run splits by sequence but counts frames as units.Source: §4.3 · pp. 12–13

4.4 Limits, Error Probability and Predictions Fixed in Advance

  • Before outcomes were computed, the authors fixed δ=0.05 and target limits α=10^-2, 10^-3, and 10^-4; exploratory limits defined no performance target.Within-dataset splits use grid, selection, and test parts; dataset-and-detector transfers use fixed candidate thresholds.Source: §4.3 · p. 12, §4.4 · p. 14

5 Results of the Applicability Test on KITTI and nuScenes

The frame-level evaluation found selected not-applicable outcomes after transfers, but the sequence-level evaluation could neither establish that transferred thresholds met target limits nor that they failed. Across distance, weather, and dataset changes, those outcomes accompanied different detection shifts; the independent-unit analysis shows why frame counts overstate the available evidence.

5.1 Applicability of the Performance Target in the Condition of Use (RQ1)

  • In the sequence-level evaluation, none of 240 transfers showed that a transferred threshold met or failed a target; frame-level outcomes included not-applicable results after transfers into far distance, rain, and between datasets.At 10−2, transfer of u_s from nuScenes to KITTI was not applicable on all five splits; no weather transfer had that outcome, and applicable transferred thresholds retained at most 29% on every split.Source: §5.1 · pp. 14–16

5.2 Changes in the Detections That Accompany a Not Applicable Outcome (RQ2)

  • Except for one transfer, not-applicable outcomes accompanied conditions with a larger share of wrong detections; distance weakened separation, while rain outcomes occurred without decreased u_e AUROC.Rain outcomes also accompanied more wrong detections and a different spread over frames; these changes accompanied, but were not shown to cause, the outcomes. Between datasets, shifted indicator values and more wrong detections accompanied them.Source: §5.2 · pp. 16–17

5.3 Independent Units Needed to Show That a Limit Is Met (RQ3)

  • At 10−2, showing a limit met requires 299 independent units for selection and 368 for testing; the sequence-level datasets could show no target limit met.The 150 nuScenes validation scenes fall short of the 299-unit selection minimum; KITTI's 20–22 drives were too few for any limit used, while frame-level data met the minimum at 10−2 but none at 10−3.Source: §5.3 · pp. 17–18, Figure 6 · p. 18

6 Discussion

The discussion explains how applicability-test outcomes inform SOTIF evidence and recording plans, then qualifies the findings through construct, internal, statistical, and external validity limits.

6.1 Consequences for the SOTIF Analysis

  • Neither validation set can establish a target limit at 10^-2 using independent recorded sequences, and frame-level evidence overstates what dependent frames support.The minimum sequence count grows linearly with 1/α and logarithmically with 1/δ′; recording rare conditions more often supplies test units but alters the all-frame FAP.Source: §6.1 · p. 19
  • A condition-specific threshold requires that condition to be known at run time, while a common threshold avoids that signal but uses stricter selection across conditions.Weather requires a rain signal whose errors enter α allocation; a common threshold for near and far yielded applicable or not determined outcomes, with lower near-distance retention.Source: §6.1 · p. 19
  • The target limits accepted wrong detections, so the safety argument also needs a target for annotated cars reaching the motion planner.The missed-object target can be tested under the same conditions and with the same interval, applying Equation (5) to its own limit.Source: §6.1 · p. 20

6.2 Threats to Validity

  • The wrong-detection definition counts some boxes on annotated cars and unannotated objects as wrong, and the near/far boundary varies by detector and split.On KITTI, 1,536 accepted wrong detections lay inside the camera image without qualifying overlap; the share that were unannotated cars was not measured.Source: §6.2 · p. 20
  • The IoU 0.5 matching rule differs from both benchmark matching rules, and anchoring detections on the first ensemble member omits cars it misses.Averaging headings can also misplace merged boxes near ±180° or when headings differ by about 180°; KITTI ignore logic is only partly reproduced, with no equivalent nuScenes annotation.Source: §6.2 · p. 20
  • Dependent frames make frame-level evidence too strong, while sequence-level aggregation discards within-sequence variation and the HB bound ignores variance of sequence means.Scenes from one nuScenes log may also be dependent; AUROC estimates lack intervals, and rain estimates rely on 24 scenes.Source: §6.2 · p. 20
  • External validity is limited to one detector architecture, deep ensembles, and one object class; KITTI and nuScenes do not represent a product’s ODD.Measured FAPs and retentions are specific to these detectors, and product evidence requires tests on data from its ODD.Source: §6.2 · p. 20

7 Conclusion and Future Work

The conclusion proposes testing threshold transfer across conditions and sizing evidence by independent units. In PointPillars experiments, sequence-level data were insufficient to establish any target limit, and no transfer was not applicable.

7 Conclusion and Future Work

  • Neither validation set had enough independent recorded sequences to establish a target limit, and no transfer was not applicable at sequence level.At frame level, transferred thresholds exceeded target limits after distance and dataset changes, and after weather changes only at limits above target limits; applicable transfers rejected most correct detections.Source: §7 · p. 21
  • A threshold meeting a target in one condition counts as evidence under a potential triggering condition only after testing there and reporting retention and independent-unit count.A not-applicable outcome identifies a condition where the detector with that threshold fails the target at the stated error probability; the minimum unit count can inform recording plans.Source: §7 · p. 21
  • Future work will extend testing to occlusion, car-like structures, further detector architectures and object classes, and thresholds that adapt at run time.It will also estimate within-sequence dependence and examine whether an annotation-free indicator monitor detects changes like those found between datasets and in rain.Source: §7 · p. 21

Statements and Declarations

The authors disclose funding, competing interests, data and code availability, and generative-AI use.

Statements and Declarations

Loading 2610.35092v1…