PaperlayerView exact revision on arXiv

Source-linked AI summary

Epistemic Uncertainty-Aware Defect Detection for Quality Control in Medical Device Manufacturing

Raham A. Butt, Marco Romanelli, Roche C. de Guzman

arXiv:2610.09057v1cs.AI

TL;DR

Defect detection in medical-device manufacturing must contend with epistemic uncertainty, which can make decisions unreliable in a high-stakes setting. The paper analyzes a cost-sensitive rejection strategy that lets a model abstain from uncertain predictions. Its experiments report lower decision risk on retained predictions and applicability to real-world device-report data.

  • Problem

    Epistemic uncertainty can affect defect-detection decisions in medical-device manufacturing, where errors can carry substantial costs and patient-safety risks.

    Source: §I · p. 1

  • Method

    The framework uses a cost-sensitive rejection strategy that lets the model abstain from uncertain predictions rather than force a decision for every input.

    Source: §VI · p. 8

  • Results

    Synthetic-benchmark experiments show that abstaining on a relatively small fraction of predictions can reduce decision risk, and real-world medical-device report data demonstrate applicability beyond controlled synthetic settings.

    Source: §VI · p. 8

  • Takeaways & Limitations

    Deferring uncertain decisions offers a practical mechanism for incorporating uncertainty into automated quality assurance and a foundation for downstream analysis and human review.

    Source: §VI · p. 8

    Real-world benchmarks capturing both manufacturing defects and their detection uncertainty remain limited, so generalizability across devices, defect types, and manufacturing settings requires broader evaluation.

    Source: §V · p. 8

Abstract

from arXiv · show

Objective: We investigate whether accounting for epistemic uncertainty can improve the reliability of automated defect detection in medical device manufacturing. Methods: We consider a machine learning framework that operates on heterogeneous manufacturing and device-report data represented with Knowledge Graphs. To mitigate errors arising from uncertainty in the decision model, we analyze a principled rejection strategy to abstain from predictions whose estimated epistemic uncertainty exceeds a specified threshold. We evaluate the approach using standard synthetic benchmarks and real-world medical device report data. Results: The theoretical results establish the validity of the method characterizing the regimes under which it is expected to be effective. Empirically, the rejection strategy enables explicit control of coverage, that is, the proportion of samples for which the model issues predictions, while improving performance on the retained samples. On 266,170 real-world FDA MAUDE device reports, a 10% abstention rate reduces classification error by 48%, and more aggressive rejection (approximately 70% coverage) yields near-perfect accuracy on the retained samples. On standard synthetic manufacturing benchmarks, abstaining on 9% of the decisions, our approach reduces the risk up to 63% compared with the standard no-abstention approach. Conclusions: Abstaining from predictions with high epistemic uncertainty can provide a practical tool for controlling the reliability of machine learning-based defect detection, especially in high-stakes medical device manufacturing applications. Significance: Uncertainty-aware defect detection may support safer and more reliable quality assurance in medical device manufacturing by identifying cases that require additional inspection rather than issuing potentially harmful predictions.

I. INTRODUCTION

Defect misclassification in medical-device manufacturing can carry substantial costs and patient-safety risks, while epistemic uncertainty can undermine defect detection. The paper investigates abstaining from uncertain predictions as a way to address this risk.

I. INTRODUCTION

  • A prior Knowledge Graph-based, three-head cascade models defect and mechanism likelihoods and parameter-level risk, but may be sensitive to epistemic uncertainty from limited training information.The heads detect defects, identify mechanisms, and estimate associated risks.Source: §I · p. 1
  • Misclassification can trigger costly false-positive alerts or missed-defect risks to patients, making reliable defect detection especially important.The introduction also notes that complex devices may limit traditional defect-detection methods.Source: §I · p. 1
  • The paper investigates detection failures arising from epistemic uncertainty and a principled approach to mitigating them.It frames the problem around high-stakes decisions where errors can have substantial costs or patient-safety consequences.Source: §I · p. 1
  • In a toy example, reducing coverage by 19% lowers classification risk on retained predictions by 50%.Rejected observations cluster near decision boundaries and generally have smaller softmax margins than retained observations.Source: Figure 1 · p. 2
  • The proposed framework allows abstention on uncertain predictions to improve robustness while accounting for the costs of erroneous decisions.The authors note that abstained cases could receive further human or automated scrutiny, though that follow-up is outside this work's scope.Source: §I · p. 2

A. Contributions

The paper contributes a reproducible real-world medical-device benchmark, an analysis of epistemic-uncertainty-related detection failures, and a cost-sensitive abstention approach. It also provides theoretical characterization and evaluates the approach on synthetic benchmarks.

A. Contributions

  • The work introduces a reproducible medical-device case-study benchmark available as a Knowledge Graph through Proto-OKN,.The authors state that code for reproducing the experiments and benchmark generation is available.Source: A. Contributions · p. 2
  • The authors investigate detection failures from epistemic uncertainty in semiconductor root-cause analysis and extend the analysis to medical-device defect detection, where errors can carry costs and patient-safety risks.They analyze a cost-sensitive approach that incorporates uncertainty handling into training and allows abstention.Source: A. Contributions · p. 2
  • The theoretical results characterize regimes where the abstention approach is expected to be effective and provide a mathematical-programming foundation.Synthetic-benchmark experiments span varying class-posterior overlap and five random seeds, with reported effectiveness as overlap increases and robustness to random initialization.Source: A. Contributions · p. 3

II. RELATED WORK

Prior work applies AI to defect detection in medical devices and semiconductor manufacturing, while methods for selective classification address uncertain predictions. This paper focuses on epistemic uncertainty in decisions and targets a gap in real-world medical-device data analysis.

II. RELATED WORK

  • Prior medical-device work applies AI to software defects and manufacturing inspection, including porosity detection and methods addressing textured backgrounds, small defects, and domain shifts [2]–.Semiconductor studies also address defect detection, fault classification, anomaly prediction, and structured root-cause explanations,,,,.Source: §II · p. 3
  • Existing abstention and selective-classification methods include dedicated architectures, perturbed training data, specialized training procedures, and post-training uncertainty measures –.The cited post-training approaches quantify uncertainty without modifying the model, but may be limited for overconfident models,.Source: §II · p. 3
  • The work focuses on epistemic uncertainty arising from models’ decisions, which the discussed prior works do not explicitly account for, and extends decision-theoretic principles of with theoretical insights.These insights include an interpretable characterization of the abstention threshold and a complete mathematical programming characterization through associated optimality conditions.Source: §II · p. 3
  • The authors identify a gap in work on epistemic uncertainty and selective prediction for MAUDE-related real-world medical-device data.Proto-OKN, integrates sources including FDA 510(k) and MAUDE,, enabling investigation with heterogeneous device data.Source: §II · p. 3

III. FRAMEWORK

The framework formulates defect detection at head H1 as multiclass classification and modifies the training objective with a reservation output for abstention. A positive hyperparameter controls the prediction–abstention trade-off, and the analysis characterizes abstention thresholds and degenerate cases.

III. FRAMEWORK

  • The analysis focuses on head H1 as a multiclass classification task, because its errors can produce false or missed defect alarms and affect costs and risk.The authors state that the analysis extends naturally to other classification heads.Source: §III · p. 3
  • The predictor selects the class with the highest inferred posterior probability, while training generally minimizes empirical cross-entropy on the training set.The inferred posterior approximates the unknown true class posterior.Source: §III · p. 4
  • The training objective is modified by adding a reservation variable as an additional class label, allowing the model to defer predictions when uncertainty is high.The reservation variable is constrained to be nonnegative.Source: §III · p. 4
  • The positive hyperparameter k controls the trade-off: higher values encourage predictions, while lower values promote abstention.The reformulation and analysis characterize the abstention threshold and identify degenerate cases.Source: §III · p. 4

A. Illustrative example

The reservation output lets the classifier reject uncertain observations, especially near decision boundaries. In the illustrative example, reducing coverage substantially lowers classification risk on retained predictions.

A. Illustrative example

  • The example uses a three-class classifier with an additional reservation output and rejects an observation when its largest class probability does not exceed the reservation probability.The model has two hidden layers and is trained for 150 epochs.Source: A. Illustrative example · p. 4
  • The reservation output allows the model to reject observations concentrated around decision boundaries, where competing classes are difficult to distinguish.These regions are associated with greater predictive uncertainty.Source: A. Illustrative example · p. 4
  • At approximately 81% coverage, classification risk falls by approximately 50% on retained predictions.The selective risk curve illustrates the trade-off between coverage and risk.Source: A. Illustrative example · p. 4
  • Rejected observations are predominantly associated with smaller classification margins, indicating weaker separation between competing classes.The reservation mechanism trades coverage for lower classification risk.Source: A. Illustrative example · p. 4

B. Optimization problem with the reservation variable

The theoretical analysis characterizes how the reservation variable and cost multiplier shape optimal predictions. It identifies regimes where full reservation or ordinary class-probability predictions are optimal, as well as mixed and degenerate cases.

B. Optimization problem with the reservation variable

  • The analysis uses true posterior probabilities, with results for empirical one-hot labels available as a special case.The proofs use Karush-Kuhn-Tucker optimality conditions.Source: B. Optimization problem with the… · p. 5
  • When the optimal reserve probability increases or the cost multiplier k decreases, the adaptive threshold rises and more class probabilities become inactive.The threshold determines which class probabilities remain active at the optimum.Source: B. Optimization problem with the… · p. 5
  • For k < 1, full reservation is optimal; for k > C, every optimal solution has prediction coordinates equal to the true class probabilities.The mixed regime is 1 < k < C.Source: B. Optimization problem with the… · p. 5
  • At k = 1 and k = C, the optimizer need not be unique, and distinct prediction–reservation allocations may have the same optimal value.Source: B. Optimization problem with the… · p. 5

IV. EXPERIMENTS

The experiments examine synthetic defect-classification benchmarks and a real-world medical-device case study. Across the evaluation, the reported accuracy and risk apply only to predictions the model retains, while coverage records the fraction of samples receiving predictions.

IV. EXPERIMENTS

  • The evaluation covers the baseline synthetic benchmark, a more challenging version with increased class-conditional overlap, and a real-world medical-device case study.The real-world benchmark is publicly accessible through Proto-OKN.Source: §IV · p. 5
  • Results are reported over five independent runs using accuracy or empirical risk and coverage, the fraction of samples on which the model predicts.Accuracy and risk are computed only on samples receiving predictions.Source: §IV · p. 5
  • Higher accuracy or lower risk on retained samples indicates that abstention preferentially rejects predictions more likely to be misclassified.The framework may also abstain on some samples that would have been correctly classified.Source: §IV · p. 5
  • Figure 2 contrasts the benchmark from [7] with a harder setting that increases overlap between class-conditional feature distributions.Source: Figure 2 · p. 6

A. Synthetic Benchmark

Across two synthetic settings, abstention improves accuracy on retained predictions, and risk falls as coverage decreases in both baseline and higher-overlap cases.

B. Experiment on real medical devices data

On MAUDE reports, abstention improves retained-prediction accuracy at 90% target coverage; stronger rejection can approach perfect accuracy but entails fewer predictions.

V. LIMITATIONS AND FUTURE WORK

The paper identifies limited real-world benchmarks as a constraint on evaluating epistemic uncertainty in medical-device defect detection. MAUDE provides a report-level severity-classification testbed, but its reporting and duplication characteristics limit what results establish about device failures and deployment.

V. LIMITATIONS AND FUTURE WORK

  • Broader evaluation across devices, defect types, and manufacturing settings remains necessary to establish generalizability.The paper identifies benchmark availability as a key limitation and encourages development of larger, more diverse, application-relevant benchmarks.Source: §V · p. 8
  • MAUDE report volumes reflect reporting behavior, device utilization, and regulatory requirements, not confirmed failure incidence.Narratives vary in completeness and may contain duplicates, so the benchmark is report-level severity classification under realistic label noise, not an estimate of device failure rates.Source: §V · p. 8
  • Exact narrative matching removes duplicates, but near-duplicate event reports may cross random train–test splits.The paper names near-duplicate detection and temporal splits as extensions that could better approximate deployment conditions.Source: §V · p. 8
  • The authors point to Proto-OKN and existing repositories as resources for building and sharing more diverse medical-device datasets.Such datasets could support more comprehensive evaluations and a stronger empirical foundation for quality-assurance use.Source: §V · p. 8

VI. CONCLUSION

The conclusion presents abstention based on epistemic uncertainty as a principled way to reduce risk from unreliable defect-detection decisions, with implications for downstream analysis and review. The appendix supports the method’s optimization with results on the training objective, probability simplex, and class-activation threshold.

VI. CONCLUSION

  • Abstaining on uncertain predictions provides a principled way to reduce risk from unreliable defect-detection decisions that may propagate into root-cause analysis and risk assessment.The paper evaluates the approach on synthetic benchmarks and real-world medical-device report data.Source: §VI · p. 8
  • Deferring uncertain decisions offers a practical mechanism for incorporating uncertainty into automated quality assurance and connecting decisions with downstream analysis and review.The paper cautions that predictive performance alone may not adequately characterize reliability in high-stakes defect detection.Source: §VI · p. 8

APPENDIX

  • The appendix distinguishes true conditional class probabilities used in theoretical optimization from observed one-hot targets used for empirical loss.The conditional population objective averages the empirical loss over the distribution of labels given the input.Source: APPENDIX · p. 10

A. Auxiliary results

  • The auxiliary results establish concavity of the objective and explain that its constrained maximization over the probability simplex admits a global maximum.The simplex is convex, closed, and bounded.Source: A. Auxiliary results · p. 11
  • Setting the reserve output to zero makes the formulation equivalent to standard training with cross-entropy loss.The equivalence follows through minimizing the Kullback–Leibler divergence between true and predicted distributions.Source: A. Auxiliary results · p. 11

B. Proof of Theorem 3.1

  • The proof characterizes a class as inactive when its probability falls below a threshold, with k defined as the reserve-output to class-prediction cost-multiplier ratio.Source: B. Proof of Theorem 3.1 · p. 12

C. Proof of Theorem 3.2

Theorem 3.2 characterizes reservation choices by the cost multiplier k: full reservation is optimal below 1, while no reservation is optimal above the class count C. The boundary values k = 1 and k = C may admit multiple optimal allocations.

C. Proof of Theorem 3.2

  • At the boundary cases k = 1 and k = C, the optimizer need not be unique: distinct prediction–reservation allocations may attain the same optimal value.For k = 1, the proof gives an example with the posterior concentrated on a single class. For k = C, every member of a family indexed by different α values is optimal.Source: C. Proof of Theorem 3.2 · pp. 12–14
  • The proof analyzes k < 1, k > C, and both boundary cases k = 1 and k = C separately.Source: C. Proof of Theorem 3.2 · pp. 12, 14
  • For k < 1, full reservation is optimal; for k > C, every optimal solution has no reservation and predicts according to the posterior distribution.The cost-multiplier mixed-regime is 1 < k < C.Source: C. Proof of Theorem 3.2 · pp. 13–14

D. Additional numerical results

E. Benchmark generation

  • The synthetic benchmark generator uses process and defect-map inputs to produce a synthetic labeled dataset.The listed inputs include feature means and scales, a correlation matrix, a causal defect map, class priors, and a target Bayes error.Source: E. Benchmark generation · p. 14

Related papers

Loading 2610.09057v1…