PaperlayerView exact revision on arXiv

Source-linked AI summary

Distribution-free false-alarm calibration and chance-corrected spatial evaluation for industrial anomaly detection

Jie Deng

arXiv:2608.15090v1cs.CVcs.LG

TL;DR

AUROC and raw mask overlap do not specify selected-threshold false-alarm performance or account for chance spatial correspondence. The paper combines a distribution-free tolerance threshold with a paired-minus-crossed spatial test, finding positive spatial-evidence lift on Magnetic Tile Defect but not in most ISP-AD cells.

  • Problem

    AUROC and raw mask overlap do not specify selected-threshold false-alarm performance or distinguish defect-specific localization from chance spatial correspondence.

  • Method

    The study combines a distribution-free tolerance threshold with a paired-minus-crossed spatial test that compares matched and crossed image–mask overlap.

  • Results

    Eight of nine ISP-AD detector–modality cells had no positive bootstrap lower bound for spatial-evidence lift, whereas both Magnetic Tile Defect representations showed positive lift.

  • Takeaways & Limitations

    The results support reporting operating-point performance and chance-corrected spatial evidence separately from AUROC and raw mask overlap.

  • Takeaways & Limitations

    The empirical study uses two public industrial datasets rather than multi-site production streams, and its FPR statement depends on image-level independent sampling.

Abstract

from arXiv · show

Studies of industrial visual inspection commonly report the area under the receiver operating characteristic curve (AUROC) and the overlap between anomaly maps and defect masks. Neither measure specifies the false-alarm rate at a selected threshold, while recurrent defect locations and mask geometry can inflate overlap. We combine a distribution-free upper tolerance threshold with a paired-minus-crossed spatial test. This test compares each detector's score-contributing locations with the matched defect mask and with masks from other images; the difference in rates defines spatial-evidence lift relative to the empirical chance-overlap rate. We evaluate three detectors on 120 point-defect images from three ISP-AD modalities and three fixed data splits. Of 378 alarms, 230 overlap the matched mask. Paired and crossed rates are nevertheless similar in eight of nine detector--modality cells; only DINOv2--ASM has a positive 95\% bootstrap lower bound (lift 0.259, 95\% interval 0.159--0.347). On the independent Magnetic Tile Defect dataset, the same analysis gives lifts of 0.203 (0.169--0.236) for Wide ResNet-50 (WRN50) patch memory and 0.231 (0.202--0.262) for Vision Transformer B/16 (ViT-B/16) patch memory, with one-sided permutation $p=10^{-5}$ for both. When crossed masks are restricted to the same defect class, the lifts remain 0.185 and 0.210. Exact sample planning shows that, with 150 calibration normals, a 95\%-confidence distribution-free claim is supported only for target false-positive rates of 1.98\% or higher; a 1\% target requires at least 299 normals. The results support reporting operating-point performance and chance-corrected spatial evidence alongside AUROC and raw mask overlap.

1 Introduction

The study frames industrial anomaly detection as requiring both auditable operating-point false-positive control and spatial evaluation beyond raw mask overlap. It proposes distribution-free threshold calibration and chance-corrected spatial-evidence testing while keeping ranking, operating-point, and overlap performance distinct.

  • Introduction: AUROC summarizes ranking performance but does not specify false-positive behavior at an operating threshold.The introduction motivates explicit threshold evaluation for unsupervised anomaly detectors used in manufacturing quality control.
  • Introduction: Raw localization overlap can be inflated by recurrent positions, centered crops, mask geometry, dilation, or the number of selected patches.High overlap may reflect responses to common image regions rather than image-specific defects.
  • Introduction: A one-sided order-statistic tolerance limit links anomaly scores to finite-sample, distribution-free false-positive claims.The study derives normal-sample requirements for target false-positive rates between 0.1% and 10%.
  • Introduction: Paired-minus-crossed permutation testing estimates spatial-evidence lift beyond empirical chance overlap by removing the original image–mask pairing.The crossed-mask reference preserves observed mask geometry and detector locations while breaking their original pairing.
  • Introduction: The procedure is evaluated on ISP-AD and Magnetic Tile Defect across convolutional and transformer representations, repeated splits, and matched sensitivity analyses.Detector architectures remain unmodified, with the aim of making threshold selection and spatial interpretation auditable.

2 Related work

Prior work established industrial anomaly detectors, benchmark datasets, threshold-calibration methods, and spatial evaluation metrics. This section motivates combining distribution-free operating-point control with chance-corrected spatial analysis for industrial inspection.

  • PatchCore, PaDiM, DINOv2, AnomalyDINO, and EfficientAD represent memory-based, distributional, transferable-feature, few-shot, and low-latency anomaly-detection approaches.PatchCore compares pretrained patches with nominal memory, PaDiM models position-wise Gaussians, and EfficientAD combines teacher–student and autoencoder branches.
  • MVTec AD and VisA provide broad benchmarks, while ISP-AD and Magnetic Tile Defect add production-modality and masked multi-class defect settings.MVTec AD 2 highlights threshold maintenance under industrial distribution change.
  • Threshold-free ranking does not specify deployed alarm rates, motivating one-sided order-statistic tolerance limits that bound normal-score tails without parametric assumptions.Probability-of-detection studies likewise evaluate binary detection at a selected threshold and report uncertainty conditional on flaw properties.
  • Pixel AUROC, region overlap, and point-mask intersection capture different spatial properties, while cross-image negatives help address spatial bias and empirical mask geometry.The adapted analysis uses complete defect masks and detector-native contributor sets in a thresholded, unconditional alarm-and-mask event.

3 Materials and methods

The study evaluates fixed feature-based detectors on point defects using distribution-free thresholding and chance-corrected spatial analysis. Independent Magnetic Tile evaluation and prespecified sensitivity procedures test whether matched-mask overlap exceeds empirical crossed-mask overlap.

  • External evaluation: The prespecified Magnetic Tile statistic crosses masks across all 388 nonempty-mask images, while class and class–filename-group stratification are post-hoc sensitivity analyses.Within-stratum resampling examines sensitivity to defect class, location, and repository-group structure without revising the primary decision rule.
  • Study design: 120 ISP-AD point-defect files were evaluated with three detectors, three modalities, and three deterministic seeds, yielding 1080 repeated evaluations.Point defects were selected because their small extent makes accidental overlap and patch-grid resolution especially important.
  • Operating-point calibration: Distribution-free thresholding uses the least conservative order-statistic rank under target tail probability α and confidence 1 −δ, with deployment rule S > τ.The guarantee requires independent calibration units and is conditional on a fixed normal distribution; it does not extend to batch dependence or domain shift.
  • Operating-point calibration: 150 calibration normals with α = 0.10 and δ = 0.05 give rank r = 142 and achieved confidence 0.9693; independent normals estimate realized FPR without refitting.The primary setting uses 200 independent ISP-AD normals per modality and seed and 786 Magnetic Tile normals per seed for Wilson intervals.
  • Spatial evaluation: Spatial evidence compares score-contributing grid cells with matched defect masks and crossed masks after any-positive pooling and one-cell mask dilation.Crossing preserves score maps, alarms, mask sizes, and locations while breaking image–mask identity; bootstrap intervals and permutation tests quantify the lift.

4 Results

Results show that operating-point and chance-corrected spatial metrics reveal distinctions obscured by AUROC or raw overlap. Spatial evidence is weak across most ISP-AD cells but positive on Magnetic Tile Defect, while finite-sample guarantees constrain low-FPR claims.

  • Primary ISP-AD results: 4.8%–6.8% mean FPRs remained below the nominal 10% target across nine ISP-AD cells, while DINOv2–LSM-2 recall varied from 45.5% to 87.9%.DINOv2–LSM-2 had mean AUROC 0.974 and point-defect alarm recall 70.7%; AUROC varied approximately 0.006 across seeds.
  • Chance-corrected spatial evidence: 60.8% of 378 alarms overlapped corresponding masks, but eight of nine detector–modality cells had bootstrap lower endpoints at or below zero after crossed-mask comparison.For DINOv2–LSM-2, paired and crossed rates were 68.7% and 68.0%, yielding lift 0.007 (p = 0.769).
  • Chance-corrected spatial evidence: 0.259 was the only strictly positive ISP-AD lift, from DINOv2–ASM, with paired rate 0.382, crossed rate 0.123, and 95% CI 0.159–0.347.Detector-blocked permutation p values were 0.00373 for PatchCore, 10^-5 for DINOv2, and 0.91031 for PaDiM.
  • Magnetic Tile Defect: 0.203 and 0.231 were the Magnetic Tile Defect lifts for WRN50 and ViT-B/16 patch memory, respectively, with both one-sided permutation tests at p = 10^-5.The corresponding paired-minus-crossed rates were 0.260–0.058 for WRN50 and 0.301–0.069 for ViT-B/16; confidence intervals were 0.169–0.236 and 0.202–0.262.
  • Magnetic Tile Defect: 0.185 and 0.210 were the class-restricted lifts for WRN50 and ViT-B/16, while class–exp restrictions yielded 0.199 and 0.225 among 386 eligible defects.All intervals remained above zero, and within-stratum permutation tests reached the Monte Carlo minimum.
  • Finite-sample calibration limits: 299 representative, effectively independent normal calibration units are required for a 95%-confidence claim near 1% FPR; with 150 normals, 1% is unsupported.With m = 150 and rank reselected, 10%, 5%, and 2% claims were supported, while holding r = 142 fixed supported only targets down to 9.417%.

5 Discussion

The discussion separates ranking, thresholded detection, raw overlap, and chance-corrected spatial-evidence metrics because they answer distinct questions. It also identifies limits from spatial bias, calibration design, sample planning, and study-specific measurement choices.

  • Metric interpretation: AUROC, alarm recall, raw alarm-and-overlap, and spatial-evidence lift answer distinct questions and require information not supplied by preceding metrics alone.AUROC concerns score ranking, alarm recall concerns threshold exceedance, raw overlap records mask intersection, and lift compares against crossed-mask overlap.
  • Metric interpretation: 97.1% raw overlap-given-alarm for DINOv2–LSM-2 coexists with nearly identical unconditional paired and crossed rates.On Magnetic Tile Defect, paired rates remain above crossed rates for both feature representations; the crossed comparison tests matched-image specificity rather than replacing localization metrics.
  • Spatial-reference limitations: The crossed reference preserves contributor counts and locations, alarm frequency, mask size, dilation, and recurrent mask position, but cannot remove every spatial bias.Class- and filename-group matching can further restrict comparisons when metadata are available, while the test does not establish causal visual evidence.
  • Calibration assumptions: 150 normal images cannot support a 95%-confidence guarantee at a 1% false-positive rate when the nominal target is lowered from 10% to 1%.Additional data are needed for lower targets and for studying drift, threshold stability, and economic cost; evaluation performance may nevertheless exceed the target because the guarantee concerns repeated calibration samples.
  • Study limitations: The empirical evaluation uses two public datasets rather than multi-site production streams, and its crossed protocol was developed on ISP-AD before Magnetic Tile results were inspected.Study-specific dilation, contributor reducers, and a high-dimensional covariate model may change results; raw-overlap coefficients remain sensitive to mask geometry and are not interpreted as physical capability limits.

6 Conclusion

The study combines a distribution-free tolerance threshold with a paired-minus-crossed spatial test for industrial anomaly detection. On ISP-AD, eight of nine detector–modality cells lacked a positive bootstrap lower bound for spatial-evidence lift despite 230 raw overlaps.

  • Method: The method combines a distribution-free tolerance threshold with a paired-minus-crossed spatial test.The threshold addresses normal-tail probability at a selected operating point, while the spatial test estimates overlap beyond crossed-mask rates.
  • Method: The tolerance threshold provides a finite-sample statement about normal-tail probability at a selected operating point.
  • Findings: Eight of nine detector–modality cells had no positive bootstrap lower bound for spatial-evidence lift despite 230 raw overlaps.

A Tolerance-limit proof with ties

The proof establishes a distribution-free upper-tail guarantee for a tolerance threshold under iid calibration, including when the score distribution has atoms and ties. It derives the guarantee by relating calibration order statistics to a binomial upper-tail probability.

  • Quantile and order-statistic guarantee: P0(S > S(r)) ≤ α follows when at least k = m − r + 1 calibration scores are at least the population quantile q.With q = inf{x : F(x) ≥ 1−α}, the order-statistic condition implies S(r) ≥ q and therefore P0(S > S(r)) ≤ α.
  • Binomial calibration argument: The count of calibration scores at least q is binomial with success probability p ≥ α under iid calibration.Its upper-tail probability is therefore at least the Binomial(m, α) tail used in Eq. (1), proving Eq. (2).
  • Effect of ties: Atoms and ties make the strict decision conservative.The proof notes that P0(S > q) ≤ α while P0(S ≥ q) ≥ α, so ties affect the strict-threshold behavior.

B Reproducibility checklist

The reproducibility protocol fixes data sources, seeds, calibration parameters, contributor definitions, crossed baselines, uncertainty procedures, and external evaluation settings. It also specifies inference units and EfficientAD tier handling without cross-modality merging.

  • Calibration and thresholds: The primary threshold protocol uses α = 0.10, δ = 0.05, m = 150, r = 142, and strict S > τ.These fixed values define the main calibration and decision rule.
  • Spatial implementation: Primary ISP contributors are PatchCore maximum cell and DINOv2 and PaDiM top 1%, with native-grid pooling and one-cell mask dilation.Contributor selection and mask processing are fixed across the primary ISP analysis.
  • Chance-overlap baseline: The crossed baseline uses all off-diagonal mask–map pairings within detector and modality, while seed matrices are averaged before image-level inference.Post-hoc sensitivities restrict crossing to defect class and class–exp strata.
  • Uncertainty and inference: Uncertainty uses 5,000 paired-image bootstrap replicates and 99,999 one-sided assignment permutations.These procedures quantify interval uncertainty and permutation significance for the spatial analysis.
  • External evaluation: The external protocol uses 16 support normals, 150 calibration normals, all remaining normals, all defects, and WRN50 and ViT-B/16 maximum-distance patch memories.Four empty masks are excluded only from spatial inference.

C Secondary EfficientAD score-validity check

A secondary EfficientAD-S check used the official small-teacher configuration to assess score validity. One ASM run produced finite raw maps but undefined normalized scores because affine normalization had zero range.

  • Configuration: EfficientAD-S used official small-teacher weights, ImageNette penalty images, 256 × 256 inputs, batch size one, Adam learning rate 10^-4, weight decay 10^-5, and 10,000 training steps.The budget was selected after an LSM-2 development run; LSM-1 and ASM were held-modality evaluations under the fixed protocol.
  • Score validity: 0.4036534727 was both the 90th and 99.5th percentile of the support-normal student–teacher map, making the official affine-normalization denominator zero.The ASM run for seed 20260802 completed training but did not produce a valid normalized anomaly score.
  • Score validity: Finite raw branch maps contrasted with NaN or infinite normalized calibration and evaluation scores required by the protocol.Table 9 retained the valid/attempted denominator while averaging only runs with defined scores.

D Numerical failure-handling rule

The pipeline treats a run as valid only when all required calibration and evaluation scores are finite; failures are reported diagnostically and retained in attempted-run accounting rather than assigned zero AUROC.

  • D Numerical failure-handling rule: A run is valid only if every required calibration and evaluation score is finite; undefined normalizers, non-finite scores, missing checkpoints, and incomplete evaluations are reported with dataset role and seed.Failed runs remain in the valid/attempted denominator and are not assigned zero AUROC because AUROC is undefined.
  • D Numerical failure-handling rule: Diagnostic alternative scores may be reported post hoc but cannot replace the primary result.
Loading 2608.15090v1…