PaperlayerView exact revision on arXiv

Source-linked AI summary

How Many Categories Are Enough? Distribution-Free Certification Limits for Few-Shot Anomaly Thresholds

Gia Huy Thai, Nguyen Thai Anh

arXiv:2610.00236v1cs.LG

TL;DR

Few-shot anomaly detectors are evaluated by ranking, but deployment also needs thresholds with controlled false-alarm rates, raising the question of how much normal evidence—and which statistical unit—can certify a threshold for an unseen category. The paper audits target-only LOIO calibration and develops a category-count feasibility calculus alongside CRESS, a source-category certification protocol. Target-only alarms can be unstable under shift, and available category counts leave CRESS fail-closed even when image-unit analyses select positive thresholds.

  • Problem

    The paper asks how much source evidence, measured in which statistical unit, is needed to certify an anomaly threshold for a new category.

    Source: §1 · pp. 1–2

  • Method

    The paper derives category-count feasibility limits and uses CRESS to split source categories into reference, proposal, and certification roles for threshold auditing.

    Source: §1 · p. 2

  • Results

    At α=0.20, target-only LOIO reaches empirical FAR 0.341 on Gaussian-corrupted MVTec at k=4, while category-unit CRESS returns τ⋆=0 across 960 frozen configurations with three or four certification categories.

    Source: §7 · p. 16

  • Takeaways & Limitations

    Image-unit and category-unit analyses support different risk claims: image bounds concern the selected source mixture, while a new-category claim requires category units and additional transfer conditions.

    Source: §6 · p. 16, §7 · p. 16

    The audit covers MVTec, VisA, MPDD, and four controlled corruptions rather than the full spectrum of production shifts.

    Source: §6 · p. 16

Abstract

from arXiv · show

Few-shot anomaly detectors are judged by ranking metrics, yet deployment requires an alarm threshold with a controlled false-alarm rate (FAR). We ask how much normal evidence, in images or category units, is needed to certify such a threshold for an unseen category. Using a frozen DINOv2 principal component analysis (PCA) residual ranker on 15 MVTec and 12 VisA categories under four corruption types, we show that target-only leave-one-image-out (LOIO) calibration is resolution-limited and shift-fragile: rank values cannot fall below $1/(k+1)$, and at the attainable level $\alpha=0.20$, empirical FAR reaches 0.341 on Gaussian-corrupted MVTec at $k=4$, 1.7 times the nominal level. A category-count feasibility calculus is then derived: even with all-zero category losses and no multiplicity charged, any deterministic, uniformly valid, distribution-free 95% upper confidence bound (UCB) requires at least 14, 29, and 59 independent and identically distributed (iid) category draws at $\alpha=0.20$, $0.10$, and $0.05$; these counts are necessary but not sufficient. The Cross-category Reliability Estimation with Source Support (CRESS) protocol splits source categories into disjoint reference, proposal, and certification roles. With only three or four certification categories, all 960 frozen configurations return the fail-closed threshold $\tau^\star=0$, and the smallest category-level UCB is 0.950. Image-unit analyses of the same archive select positive thresholds in 36.7% to 60.3% of target cells; these bounds hold for the selected source mixture, not for the marginal risk of a new-category draw. The contribution is a quantitative feasibility boundary and an estimand-aware protocol specifying when source evidence can, and cannot, support a transferable reliability claim.

1. Introduction

Target-only few-shot calibration can reach an attainable alarm level yet remain shift-fragile, while distribution-free source certification for a new category requires enough independent category evidence. The paper introduces a feasibility calculus and CRESS protocol to distinguish what image evidence and category evidence can support.

1. Introduction

  • At α=0.20, empirical FAR reaches 0.341 on Gaussian-corrupted MVTec at k=4, approximately 1.7 times the intended level.This is a benchmark aggregate across categories and five support-sampling seeds under one declared synthetic shift, not a population-level failure probability.Source: §1 · p. 1
  • Target-only LOIO ranks lie on the finite grid {j/(k+1)}, and crossing its resolution floor does not guarantee stable FAR under shift.The paper distinguishes attainable operating levels from exchangeability conditions needed for false-alarm validity.Source: §1 · p. 1
  • With only three or four certification categories, category-unit CRESS returns τ⋆=0 across all 960 frozen configurations.Image-unit analyses select positive thresholds in 36.7% to 60.3% of target cells, but the two analyses address different risk profiles.Source: §1 · p. 2
  • Even with all-zero category losses and no multiplicity, deterministic uniformly valid distribution-free 95% UCBs require at least 14, 29, and 59 iid category draws at α=0.20, 0.10, and 0.05.These are necessary, not sufficient, counts; Figure 1 previews the feasibility calculation and the available three or four certification categories.Source: §1 · p. 2, Figure 1 · p. 3
  • CRESS separates source categories into reference, proposal, and certification roles, then reports image-mixture and category-level conclusions without substituting one for the other.The source protocol retains zero fallbacks and audits whether the available archive can support a nonzero certificate.Source: §1 · p. 2

2. Related Work

Prior few-shot anomaly detectors provide ranking mechanisms, while calibration, conformal inference, and risk-control methods supply related statistical tools. This paper applies those principles to cross-category threshold certification and separates its contribution from the inherited rankers and statistical primitives.

2.1. Few-Shot Industrial AD

  • Raw score scales from few-shot industrial anomaly rankers do not by construction provide a certified false-alarm-rate interpretation.The paper positions its work as complementary to ranking methods, examining reliability of their decision thresholds instead.Source: §2.1 · p. 3

2.2. Frozen-Feature Subspace Detectors

  • Frozen DINOv2 PCA residuals were already established as an effective few-shot ranking mechanism, so this work begins after residual computation.SubspaceAD [20] is cited for the prior ranking mechanism.Source: §2.2 · p. 3

2.3. Calibration and Decision Reliability in Few-Shot AD

  • A calibrated probability summary alone does not constitute a risk guarantee for a declared alarm rule under a specified shift.Prior calibration and perturbation sensitivity findings motivate separating ranking quality from operational false-alarm behavior.Source: §2.3 · p. 3

2.4. Conformal Prediction and Risk-Controlled Calibration

  • CRESS applies established candidate-testing and one-sided-bound principles using disjoint source roles, distinguishing selected-mixture image risk from marginal new-category risk.The paper claims neither a new nonconformity score nor a generic confidence bound.Source: §2.4 · p. 4

3. Method

The method fixes the statistical unit and separates target-only and source-assisted reliability routes around a frozen DINOv2 PCA residual scorer. Its image and category analyses target distinct risk quantities, with category-level transfer requiring additional conditions.

3.1. Problem Setup

  • The framework returns a fixed raw anomaly score and optional pixel map, plus target-only LOIO reliability when available or a source-assisted threshold with a certificate.Target test labels are reserved for evaluation rather than detector fitting, threshold proposal, or certification.Source: §3.1 · p. 4
  • Image-unit analysis targets mean alarm loss for the retained source-image mixture, whereas category-unit analysis equally weights category losses for a new draw from the declared category meta-population.Category units include support construction and held-out normal-view sampling; additional images, support seeds, or corruption views do not add independent category units.Source: §3.1 · p. 4

3.2. Frozen Subspace Scoring and Localization

  • Frozen DINOv2 ViT-S/14 patch features are used to fit a PCA subspace, and patch residuals form the anomaly map.The image-level ranking score aggregates the largest residuals using a top-1% mean, a smoothed maximum.Source: §3.2 · pp. 4–5

3.3. Target-Only LOIO Reliability

Target-only LOIO constructs normal reference scores by leaving out each support image, then ranks a test score against those references. The resulting quantity is an audited rank value, not an automatic finite-sample guarantee.

3.3. Target-Only LOIO Reliability

  • Each support image is scored by a subspace fitted on the other k−1 supports, producing target-normal nonconformity evidence for ranking a test image.The test image is scored using the subspace fitted on all k supports, making the construction asymmetric and a LOIO approximation rather than a split-conformal p-value with an automatic finite-sample guarantee.Source: §3.3 · p. 5
  • Figure 2 separates the shared frozen-subspace scoring substrate from reuse of that scorer to construct target-only LOIO rank values.Source: §3.3 · p. 5

3.4. Target-Only Resolution

The target-only LOIO rank has a finite attainable grid, so its alarm cannot operate below the grid’s minimum value.

3.4. Target-Only Resolution

  • The LOIO rank lies on {j/(k+1): j=1,…,k+1}, making the alarm identically zero for α < 1/(k+1).This algebraic resolution limit requires no exchangeability assumption.Source: §3.4 · p. 5
  • At k=4 the attainable floor is 0.2, while at k=8 it is approximately 0.111.The analysis reports the largest grid point not exceeding each nominal α and its corresponding empirical FAR.Source: §3.4 · p. 5
  • Randomization can smooth the deterministic grid under relevant exchangeability assumptions, but cannot by itself address support-to-test shift.The floor is a finite-rank resolution property, not evidence that the underlying ranker cannot separate anomalies.Source: §3.4 · p. 5

3.5. Source Archive Preparation

Source archive preparation normalizes scores using support-only statistics, then selects or aggregates source views by declared condition; the scoring and target-side workflow use frozen DINOv2 PCA residuals and LOIO ranks.

3.5. Source Archive Preparation

  • Support-only residuals are summarized by their median and MAD; when k=1, a patch-split support calibration residual replaces undefined LOIO calibration.At k=1 the single residual gives zero MAD, so the denominator defaults to ε=10^-6; these cases are treated as stress tests.Source: §3.5 · p. 5
  • Source views are selected after normalization: matched-condition mode requires condition metadata, clean-source mode uses clean normals, and condition-agnostic mode takes each base image’s median across conditions.Mismatched-condition routing uses the lexicographic successor with wraparound; condition-agnostic aggregation avoids treating repeated corruption views as independent observations.Source: §3.5 · p. 5

Image AUROC, AP

  • Frozen DINOv2 patch features and a support-fitted PCA residual scorer produce the image-ranking score and optional inherited pixel map.The pixel map is not a certification target or a localization claim in this study.Source: Figure 2 · p. 6
  • The target-side workflow applies the scorer within LOIO folds to obtain target-normal calibration scores and rank values, whose minimum is 1/(k+1).Because calibration scores use k−1 supports while the test score uses all k, the rank is empirically audited rather than assumed to have automatic finite-sample validity.Source: Figure 2 · p. 6

3.5. Source Archive Preparation

  • Eligible source categories are split before candidate construction into nonempty, disjoint reference, proposal, and certification sets using the 0.50/0.25/0.25 allocation.The hash-based category permutation is independent of observed scores and outcomes, and all views from one category remain in one partition.Source: Image AUROC, AP · p. 6

3.6. Nested CRESS Threshold Selection

CRESS builds a source-reference rank map, proposes thresholds from proposal categories, and evaluates them on separate certification categories. Its image and category analyses target different estimands, and failure to certify returns a zero threshold.

3.6. Nested CRESS Threshold Selection

  • The pooled normalized reference scores define an empirical source-reference rank map, not an assumed cross-category conformal transformation.Its finer grid comes from the larger reference-score pool rather than the k target supports.Source: §3.6 · p. 6
  • Distinct proposal ranks form the ordered candidate set, capped at 20 evenly spaced thresholds when necessary; certification data do not determine the candidates.The same reference map is applied to certification images.Source: §3.6 · p. 6
  • The image-unit loss is an alarm indicator, while category-unit loss averages alarm indicators across a certification category’s views.Certification images are treated as normal, so any alarm is a false alarm; category losses allow arbitrary dependence among archived views within a category.Source: §3.6 · p. 6
  • Image-unit bounds target the declared source-image mixture, whereas category-unit bounds target expected archive-average loss for an iid category draw; the estimands are not interchangeable.Equating the category expectation with a new-view alarm probability additionally requires representative within-category sampling.Source: §3.6 · p. 6
  • Category certification uses a family-adjusted Hoeffding UCB, while image sensitivity uses a Clopper–Pearson bound with error allocation across levels, unit types, and candidates.The image analysis assumes iid Bernoulli alarms from the induced source-image mixture, which the stratified archive does not guarantee.Source: §3.6 · p. 7
  • CRESS returns the largest passing threshold or τ⋆=0, a deterministic fail-closed fallback; target test images and labels enter none of the selection stages.The workflow is summarized as source-archive preparation, category splitting, candidate proposal, certification, and threshold selection.Source: §3.6 · p. 7

3.7. Certificate Scope, Feasibility, and Transfer Boundary

Category-level certification is constrained by the number of iid category units: even zero observed losses require substantial sample sizes for a distribution-free 95% UCB. CRESS certifies source risk, with target-category guarantees requiring additional transfer conditions.

3.7. Certificate Scope, Feasibility, and Transfer Boundary

  • The conditional source certificate holds simultaneously across reported levels and both unit definitions when certification units are independent samples from their declared source-unit population.The image model conditions further on fitted certification scorers; category losses are bounded in [0,1].Source: §3.7 · p. 7
  • At 95% confidence, even all-zero losses require at least 14, 29, and 59 iid categories to certify α=0.20, 0.10, and 0.05 without multiplicity.These are necessary lower bounds for deterministic, uniformly valid, distribution-free UCBs, not sufficient sample sizes for CRESS.Source: §3.7 · p. 7
  • The all-zero category-count calculations are design limits, and meeting a listed count does not guarantee a positive CRESS threshold.The distribution-free result applies to deterministic, uniformly valid bounds based exclusively on iid bounded category losses; alternative assumptions require separate validity justifications.Source: Table 1 · p. 8, §3.7 · p. 8
  • With only three or four iid categories and zero losses, the optimistic multiplicity-free 95% UCB floors are 0.632 and 0.527; these budgets do not suffice for CRESS.At α=0.20, three and four categories support at most 48.8% and 59.0% confidence in the multiplicity-free scenario.Source: §3.7 · p. 8
  • Repeated images or support seeds do not increase the number of independent certification categories, and the available source pools fall below the 14-category minimum at α=0.20.Maintaining nonempty reference and proposal roles limits certification categories to at most 12, 9, and 13 for pools of 14, 11, and 15.Source: §3.7 · p. 8
  • Source-risk dominance permits a bound for a specific target category; iid sampling from the same category meta-population supports only a marginal new-category claim.The latter also requires representative within-category sampling to extend to the marginal alarm probability of a new normal view.Source: §3.7 · pp. 8–9

4. Experiments

The experiments assess few-shot anomaly reliability across datasets, corruption conditions, and source-transfer settings. They distinguish ranking metrics from operational certification and define category-level analyses and reproducibility checks.

4.1. Datasets and Few-Shot Splits

  • Within-MPDD certification is omitted because excluding the target leaves only five eligible source categories, below the implementation’s six-category minimum.MVTec, VisA, and MPDD provide the study’s evaluation and transfer settings; nested support sets are formed from permuted normal training images.Source: §4.1 · p. 9

4.2. Evaluation Protocols

  • Target corruption tests cover 15 MVTec and 12 VisA categories at k=4/8, across five seeds and four corruptions; support images remain clean.CRESS deterministically assigns source categories to disjoint reference, proposal, and certification roles; target labels are used only to score outcomes.Source: §4.2 · p. 9
  • The frozen operational gate requires positive category-level thresholds in at least 80% of target cells, then checks FAR, power, and related criteria.The 80% cutoff is a usability criterion, not a confidence level, and the protocol does not provide a simultaneous 95% guarantee over the full grid.Source: §4.2 · p. 9

4.3. Baselines and Ablations

  • PCA64 is the default reliability substrate; direct pooled-source conformal is an uncertified baseline, while the controlled memory bank isolates scoring and storage trade-offs.The controlled memory-bank implementation is not an official PatchCore or AnomalyDINO reproduction; published values are used for AnomalyDINO-S and WinCLIP.Source: §4.3 · p. 10

4.4. Metrics

  • Ranking uses raw-score image AUROC and AP, while reliability measures include FAR, power, alarm precision, normal-rank CDFs, positive-threshold rates, and retained state.Paired corruption views measure residual displacement relative to shared support-derived calibration scores; category-level correlations are descriptive, not causal.Source: §4.4 · p. 10

4.5. Reproducibility and Claim Rules

  • Reproducibility records preserve protocol, model, data, prediction, support, corruption, and source-partition details, with automated integrity checks across 3,300 view cells.The checks cover 1,500 MVTec, 1,200 VisA, and 600 MPDD cells.Source: §4.5 · p. 10

5. Results

Target-only LOIO shows dataset- and corruption-dependent false-alarm behavior, while category-level certification is infeasible with the available source categories. Image-unit analyses can select positive thresholds, but they address source-mixture risk rather than marginal new-category risk.

5.1. Shift-Induced False-Alarm Inflation

  • At α=0.20, MVTec’s average FAR is 0.278 at k=4 and 0.200 at k=8, while VisA’s is 0.142 and 0.094, respectively.The effective threshold is 0.20 for k=4 but 1/9 for k=8; lower nominal levels can be unattainable because rank values lie on a finite grid.Source: §5.1 · p. 10, Table 2 · p. 11
  • 0.341 FAR occurs on Gaussian-corrupted MVTec at k=4 and α=0.20, about 1.7 times nominal; this is a benchmark aggregate, not a population-level failure probability.The MVTec category median is 0.215, with 9 of 15 categories above 0.20.Source: §5.1 · pp. 10–11, Table 3 · p. 11

5.2. Category-Level Residual Displacement

  • In MVTec, Gaussian noise and JPEG compression shift normal residual scores upward, whereas VisA’s corresponding median shifts remain near zero at both support sizes.In MVTec, category-level residual-displacement correlations with FAR inflation range from 0.825 to 0.914 across the reported corruptions and support sizes; the analysis does not establish causality.Source: §5.2 · p. 12

5.3. Category-Count Feasibility

  • 14, 29, and 59 iid category draws are necessary all-zero-sample floors for deterministic distribution-free 95% UCBs at α=0.20, 0.10, and 0.05, not sufficient sample sizes.Under the favorable frozen allocation, the lower limits are 22, 46, and 94; the Hoeffding rule requires 60, 240, and 958 draws.Source: §5.3 · p. 12

5.4. CRESS Boundary Test: Categories versus Images

  • With only three or four certification categories, all 960 strict CRESS configurations return τ⋆=0; the smallest candidate UCB is 0.950.The zero-threshold result persists when k=1 stress cells are excluded, and the 960 configurations are not independent replications.Source: §5.4 · p. 13, Table 4 · p. 14
  • Image-unit analyses select positive thresholds in 36.7% to 60.3% of target cells, but their risk bound concerns the selected source mixture, not a new-category draw.The image analysis uses an idealized conditional iid source-image assumption and does not create additional independent categories.Source: §5.4 · pp. 13–14

5.5. Supporting Ranking Substrate and Storage

  • Compared with the controlled DINOv2 NN ranker, PCA64 has lower AUROC on MVTec and higher AUROC on VisA at k=1, 4, and 8.The reported differences are −0.011, −0.005, and −0.003 on MVTec, and +0.019, +0.008, and +0.009 on VisA; external references use their respective protocols.Source: §5.5 · p. 14, Table 6 · p. 15
  • The aggregation ablation covers only three MVTec and four VisA categories; its largest displayed AUROC gap from the best tested aggregator is 0.018.Moderate top-fraction pooling outperforms max pooling on MVTec, while top-0.5% is optimal on VisA; the subset does not establish full-dataset invariance.Source: §5.5 · p. 15, Table 7 · p. 15

6. Limitations

The guarantees apply only to deterministic, uniformly valid, distribution-free UCBs based on iid bounded category losses, and their category counts are necessary rather than sufficient. The empirical scope also depends on the sampling unit, source-risk assumptions, and benchmark shifts considered.

6. Limitations

  • 14, 29, and 59 iid category draws are optimistic necessary limits at α=0.20, 0.10, and 0.05, not sufficient CRESS budgets.The bounds do not account for multiplicity; alternative methods may use fewer categories but require distinct validity analyses.Source: §6 · pp. 15–16
  • Proposition 1 covers the declared candidate family within one fixed experimental cell, not a simultaneous 95% event across the reported grid.Source: §6 · p. 16
  • Category-level interpretation treats whole categories, including support construction and held-out views, as iid draws from a defined meta-population; realized-target transfer additionally requires source-risk dominance.The empirical audit covers MVTec, VisA, MPDD, and four controlled corruptions, not the full spectrum of production shifts.Source: §6 · p. 16

7. Conclusion and Future Work

The study finds that threshold reliability depends on both the amount and the statistical unit of normal evidence. Its feasibility analysis distinguishes category-level certification from image-mixture sensitivity and motivates matching guarantees to their estimands.

7. Conclusion and Future Work

  • At α=0.20, target-only LOIO reaches empirical FAR 0.341 on Gaussian-corrupted MVTec at k=4, while the finite floor 1/(k+1) does not ensure exchangeability under shift.Source: §7 · p. 16
  • 14, 29, and 59 iid category draws are necessary even with all-zero losses for multiplicity-free deterministic distribution-free 95% UCBs at α=0.20, 0.10, and 0.05.The strict audit had three or four certification categories; category-unit CRESS returned τ⋆=0 throughout 960 frozen configurations, consistent with the feasibility boundary rather than 960 independent failures.Source: §7 · p. 16
  • Image-unit sensitivity selected positive thresholds in 36.7% to 60.3% of target cells under an idealized selected-mixture iid model, estimating a different estimand from category draws.Source: §7 · p. 16
  • The study's practical discipline is to distinguish ranking from threshold reliability, identify attainable operating points, and match the certification unit and guarantee to the estimand.Source: §7 · p. 16
  • Future work should test category-rich archives and real production shifts, and validate assumptions for parametric, hierarchical, or side-information models.Source: §7 · p. 16

Funding

The study reports no specific grant funding.

Funding

  • The research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors.Source: Funding · p. 16

Related papers

Loading 2610.00236v1…