PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability Characterization for N-version Object Detection

Shunsuke Nagao, Fumio Machida

arXiv:2610.12149v1cs.SE

TL;DR

Conventional metrics do not capture object-wise error overlap and diversity across N-version object-detection results independently of voting. This paper proposes Cov OD and Cer OD and evaluates them in a Carla simulator case study. The reported findings describe how the metrics can inform version combinations, voting strategies, and incremental system construction.

  • Problem

    Conventional metrics do not capture object-wise error overlap across N-version detection results independently of a voting strategy.

    Source: A. Performance metrics for object… · p. 3

  • Method

    The paper proposes Cov OD and Cer OD for evaluating N-version object detection using object-level granularity.

    Source: §I · p. 1

  • Results

    Architecturally diverse combinations achieve higher Cov OD, while architecturally similar combinations achieve higher Cer OD, with the trade-off amplified as more diverse models are added.

    Source: B. Results for Two and Three-Version… · p. 9

  • Takeaways & Limitations

    The metrics can guide version selection, voting-strategy choice, and incremental construction of N-version systems through pairwise analysis.

    Source: §I · p. 2

    The evaluation uses a custom Carla simulator dataset, so generality to other datasets and application domains is not fully validated.

    Source: §VII · p. 11

Abstract

from arXiv · show

N-version object detection (OD) is an approach to diversifying detection results using multiple models or input frames and reducing detection errors by aggregating individual results. Diversity and consistency across multiple detection results are critical information for characterizing the reliability of possible configurations of N-version OD systems. However, existing performance metrics such as mAP and Accuracy fail to capture these factors, as they are defined solely on the final outcome after aggregation. To overcome this limitation, we propose two reliability metrics particularly defined for N-version OD, namely coverage of errors in OD (Cov_OD) and certainty of accurate prediction in OD (Cer_OD), which can be computed from individual detection results without relying on voting strategies. We empirically demonstrate the unique features of the proposed metrics through a case study of N-version OD for a vehicle in an autonomous-driving simulator. We show that the proposed metrics can be used for 1) selecting version combinations with complementary error characteristics, 2) choosing an effective voting strategy based on diversity and consistency profiles, and 3) guiding incremental construction of N-version OD systems via pairwise two-version analysis. These results highlight the importance of the metrics that can guide the design of reliable N-version OD applications.

I. INTRODUCTION

N-version object detection combines multiple inference results, but conventional metrics do not capture object-wise error overlap and depend on the voting strategy. Cov OD and Cer OD provide voting-independent reliability measures, which the paper evaluates in a simulator case study and uses to derive system-design guidance.

I. INTRODUCTION

  • N-version machine learning uses multiple versions of inference results at runtime to improve the reliability of system outputs.The paper situates its object-detection study within prior multi-version applications to classification, malware detection, object detection, and autonomous-vehicle perception.Source: §I · p. 1
  • Conventional object-detection metrics depend on voting, while classification-oriented Cov and Cer do not count multiple object errors within a frame individually.This motivates finer-grained, object-wise reliability measures for N-version detection.Source: §I · p. 1
  • Cov OD and Cer OD quantify object-wise detection reliability and capture error overlap and result diversity without requiring a specific voting strategy.Cov OD measures detections covered by correct detections; Cer OD measures correct detections across component versions.Source: §I · pp. 1–2
  • In a Carla simulator case study, Cov OD and Cer OD were consistently higher than their Cov and Cer baseline counterparts.The authors report that the proposed metrics capture object-wise detection errors for finer-grained evaluation.Source: §I · p. 2
  • Exhaustive two- and three-version comparisons show that Cov OD and Cer OD can guide version selection, voting-strategy choice, and incremental system construction.The proposed metrics are used to identify complementary error characteristics, select a voting strategy based on diversity and consistency profiles, and build three-version systems from pairwise analysis.Source: §I · p. 2

II. RELATED WORKS

Prior work develops object-detection metrics and voting methods, but these generally evaluate one detector or combine detections into a single output. The section explains that multi-detector evaluation therefore depends on how results are merged, potentially overlooking disagreement among versions.

II. RELATED WORKS

  • Prior object-detection metrics address localization, error costs, or safety, but focus on evaluating a single detector rather than multiple detection results.Examples include LRP [14], OC-Cost, and safety-oriented metrics,,,.Source: §II · p. 2
  • Object-detection voting methods such as NMS, PRAE [24], and WBF merge multiple detections into one output using a specific combination rule.The paper instead aims to evaluate reliability and diversity without assuming a voting strategy.Source: §II · p. 2

A. Performance metrics for object detection task

  • For multiple detectors, mAP and Accuracy require a voting strategy to determine the final output, tying their evaluation to that choice.Affirmative includes all grouped inferences, consensus includes groups meeting the specified threshold, and unanimous includes groups with detections from all versions.Source: A. Performance metrics for object… · p. 3

B. Reliability metrics for N-version ML systems

Classification-oriented Cov and Cer capture whether errors are shared across versions, but treat all errors in an image as one error. The paper motivates object-wise metrics by showing how this can obscure which objects are detected across modules.

B. Reliability metrics for N-version ML systems

  • Cov measures the percentage of inputs for which one or more modules make correct inferences, while Cer measures the percentage for which all modules are correct.Both metrics characterize reliability across multiple versions.Source: B. Reliability metrics for N-version… · p. 3
  • Classification-oriented Cov and Cer count errors per image, regardless of how many false detections, misclassifications, or omissions occur.Consequently, the total number of object-detection errors is not accurately counted.Source: B. Reliability metrics for N-version… · p. 3
  • In the illustrated two-result example, person detections overlap, a bike appears in only one result, and a dog appears in neither.Because neither module perfectly detects all target objects, the existing metrics give Cov and Cer equal to zero, despite partial object-level coverage and consistency.Source: B. Reliability metrics for N-version… · pp. 3–4
  • The example motivates object-wise reliability metrics that count detection errors at finer granularity.For the example, the text states that object-wise Cov should be 2/3 and Cer should be 1/3.Source: B. Reliability metrics for N-version… · p. 4

IV. PROPOSED METRICS

The paper extends Cov and Cer to characterize reliability in N-version object detection while accounting for multiple target objects. It defines module-level detection and error sets, then uses common and combined errors to construct the metrics.

IV. PROPOSED METRICS

  • Cov_OD and Cer_OD extend N-version ML reliability metrics to account for multiple target objects in an image.The paper defines reliability as the probability that the N-version system does not output errors.Source: §IV · p. 4
  • Each module produces true-positive, false-positive, and false-negative object sets, whose union represents its detected objects.The notation also defines each module’s error set for an input image.Source: §IV · p. 4
  • The metrics use the intersection and union of module error sets to characterize shared and distributed detection errors.The paper introduces these error-set operations before defining the extended metrics.Source: §IV · p. 4

A. Coverage of error in object detection

Cov_OD reflects whether at least one module detects an object correctly, while Cer_OD reflects whether all modules do so. Their error proportions are computed over the input images and object detections.

A. Coverage of error in object detection

B. Certainty of accurate prediction in Object Detection

  • Cer_OD represents the proportion of unanimous correct inferences across modules, with higher values indicating more such unanimity.It reflects the proportion of incorrect inferences made by one or more modules across the input images.Source: B. Certainty of accurate prediction in… · p. 4

C. Interpretation of Proposed Metrics

Cov_OD and Cer_OD describe different bounds on achievable reliability, and their gap indicates how much voting strategy can exploit module complementarity. Unlike conventional performance metrics, they account for relationships among errors and detections across versions.

C. Interpretation of Proposed Metrics

  • High Cov_OD with sufficiently low Cer_OD indicates potential for high accuracy with minimal redundancy through appropriate voting.A high Cer_OD is not automatically preferable because it can indicate aligned detection tendencies and limited redundancy benefits.Source: C. Interpretation of Proposed Metrics · p. 5
  • Cov_OD(M)−Cer_OD(M) gives the range of reliability exploitable through voting and reflects module complementarity.Common errors set the upper bound because voting cannot recover them, while unanimously correct detections set the lower bound because they are never discarded.Source: C. Interpretation of Proposed Metrics · p. 5

D. Metrics Comparison

  • The Cov_OD diagram’s gray region indicates inferences that may yield a correct output, whereas Cer_OD’s gray region indicates inferences that guarantee one.Their dotted regions denote common errors for Cov_OD and the union of module errors for Cer_OD.Source: D. Metrics Comparison · p. 5, Figure 3 · p. 5
  • Conventional metrics such as mAP depend on the chosen voting strategy and do not reflect error diversity across modules.The comparison distinguishes strategy-specific TP, FP, and FN outcomes from the proposed metrics’ coverage relationships.Source: D. Metrics Comparison · pp. 5–6

V. EVALUATION

The evaluation uses Carla’s Town03 driving scenario to compare N-version object-detection configurations built from different camera inputs or different detection models.

A. Dataset and Construction of N-version Object Detection Systems

  • Carla 0.9.13 Town03 supplies the driving environment and automatically generated ground-truth bounding boxes for the shared evaluation dataset.The simulator supports varied weather and times of day; this study uses its original image dataset.Source: A. Dataset and Construction of… · p. 6

1) Input-diverse N-version system:

  • Five front-facing RGB cameras each capture 2,000 images, and YOLOv8n is evaluated as camera views are incrementally added.All cameras have a 60-degree viewing angle; the setting varies the inputs while holding the detector fixed.Source: 1) Input-diverse N-version system · p. 6, Table I · p. 6

2) Model-diverse N-version system:

B. Reliability metrics

The evaluation compares Cov and Cer with Cov OD and Cer OD across input- and model-diverse systems, alongside single-version mAP baselines.

B. Reliability metrics

  • Reliability comparisons use Cov, Cer, Cov OD, Cer OD, and mAP under three voting strategies, with bounding-box overlap determined at an IoU threshold of 0.5.Source: B. Reliability metrics · p. 6

D. Comparison with Cov and Cer

  • Cov OD and Cer OD consistently exceed Cov and Cer in both diversity settings, capturing object-level coverage and consistency that frame-level metrics can miss.Cov OD counts partial object-level error coverage, while Cer OD captures partial consistency when only some objects are detected correctly across versions.Source: Table III · p. 7, Table IV · p. 7, D. Comparison with Cov and Cer · p. 7

C. Single-Version Baseline Evaluation

  • Single-version baselines compare five models on the center view and YOLOv8n across camera views; mAP declines as camera displacement from center increases.RetinaNet has the highest model mAP at 0.588; YOLOv8n’s center-view mAP is 0.522, compared with 0.206 for left2.Source: C. Single-Version Baseline Evaluation · pp. 6–7, Table II · p. 7

E. Comparison with mAP

The comparison shows that mAP depends on voting strategy and diversity setting, whereas Cov OD and Cer OD characterize N-version configurations independently of voting.

  • With model-diverse systems, affirmative and consensus mAP rises through four versions then falls at five, while the proposed metrics remain monotonic across strategies.Source: Table VI · p. 8, E. Comparison with mAP · p. 8
  • With input-diverse systems, mAP decreases as versions increase under all three voting strategies, with unanimous voting degrading more than affirmative or consensus voting.Source: E. Comparison with mAP · p. 8
  • Cov OD and Cer OD provide voting-agnostic characterization, while mAP varies with both the voting strategy and diversity setting.Source: E. Comparison with mAP · p. 8

VI. PRACTICAL DESIGN INSIGHTS FROM PROPOSED METRICS

Analysis of two- and three-version systems links architectural diversity and consistency to reliability profiles, offering practical guidance for configuration and voting choices.

VI. PRACTICAL DESIGN INSIGHTS FROM PROPOSED METRICS

  • The section uses two- and three-version analyses to derive guidance for selecting model combinations, choosing voting strategies, and incrementally extending configurations.Source: §VI · p. 8

A. Experimental Setup

  • The analysis evaluates all 21 two-version and 35 three-version systems drawn from seven candidate models, computing Cov OD, Cer OD, and mAP under three voting methods.The candidate pool adds YOLOv5n and YOLOv11n to the five models used in the preceding evaluation.Source: A. Experimental Setup · p. 8

B. Results for Two and Three-Version Systems

  • Architecturally diverse combinations achieve higher Cov OD, while structurally similar combinations, especially within the YOLO family, tend to achieve higher Cer OD.Cov OD generally rises with heterogeneous combinations, whereas Cer OD tends to decrease when extending from two to three versions as diversity increases.Source: B. Results for Two and Three-Version… · pp. 8–9
  • Configurations with low Cov OD and high Cer OD tend to show little variation in mAP across voting strategies and often have consistently low mAP.The authors recommend checking Cov OD and Cer OD for sufficient diversity and error coverage rather than relying on mAP alone.Source: B. Results for Two and Three-Version… · pp. 8–9

C. Relationship Between Cov OD, Cer OD, and Voting Strategies

Across three-version configurations, Cov OD and the Cov OD−Cer OD gap relate to mAP differently across voting strategies. Pairwise analysis also suggests how to choose a model when extending a system.

C. Relationship Between Cov OD, Cer OD, and Voting Strategies

  • r = 0.719 and 0.608 for Cov OD versus mAP under affirmative and consensus voting, compared with r = 0.412 under unanimous voting.Figure 6 plots these relationships across all 35 three-version configurations.Source: C. Relationship Between Cov OD, Cer… · p. 9, Figure 6 · p. 11
  • Cer OD has weak or slightly negative correlations with mAP under affirmative and consensus voting (r = 0.119 and −0.098), but a moderate positive correlation under unanimous voting (r = 0.472).The authors relate higher Cer OD to similar correct detections and errors, which can limit complementary detections under affirmative and consensus voting.Source: C. Relationship Between Cov OD, Cer… · p. 9, Figure 6 · p. 11
  • The Cov OD−Cer OD gap correlates positively with mAP under affirmative and consensus voting (r = 0.336 and 0.403), but not under unanimous voting (r = −0.050).The gap quantifies model complementarity; the authors describe its unanimous-voting pattern as consistent with Cov OD.Source: C. Relationship Between Cov OD, Cer… · p. 9, Figure 6 · p. 11
  • Before choosing a voting strategy, compute Cov OD, Cer OD, and their gap; prefer affirmative or consensus when Cov OD or the gap is high, and unanimous when Cer OD is high.This avoids relying on mAP, which requires a voting strategy to be selected first.Source: C. Relationship Between Cov OD, Cer… · p. 9

D. Incremental Construction of N-Version Systems via Pairwise Analysis

  • 0.947 was the highest three-version Cov OD, achieved by adding RT-DETR to FCOS and SSD, whose two-version Cov OD was 0.929.Adding YOLOv11n instead reached Cov OD 0.941; its pairwise Cer OD with FCOS and SSD was 0.721 and 0.715, versus 0.712 and 0.675 for RT-DETR.Source: D. Incremental Construction of… · p. 10
  • When extending a system, choose a candidate with low Cer OD against every existing model: low pairwise Cer OD indicates distinct detection behavior and can increase added error coverage.High pairwise Cov OD alone may not ensure that the new model adds coverage beyond what the existing pair already provides.Source: D. Incremental Construction of… · pp. 10–11

VII. LIMITATION

The study limits its evidence to a Carla simulator dataset, a limited model-combination set, and three voting strategies. Its conclusion reports design insights, with validation on public datasets identified as future work.

VII. LIMITATION

  • All experiments use a custom Carla simulator dataset, so generality to other datasets and application domains is not fully validated.The dataset has accurate annotations.Source: §VII · p. 11
  • The correlation and two-to-three-version transition analyses cover combinations of seven models, while practical systems may have many more and the observed relationships may not generalize.Source: §VII · p. 12
  • The study evaluates only three voting strategies; whether the metrics guide advanced strategies such as Weighted Box Fusion remains open.Source: §VII · p. 12

VIII. CONCLUSION

  • The conclusion states that Cov OD and Cer OD extend Cov and Cer to object-level granularity and provide more accurate, meaningful evaluations than previous metrics.The authors also report actionable design insights for N-version object detection.Source: §VIII · p. 12
  • Future work plans to extend validation to public datasets such as KITTI and COCO and explore efficient ways to identify image and model combinations that maximize the metrics.Source: §VIII · p. 12
Loading 2610.12149v1…