PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixed-Prior Decision Risk for Open-Set Recognition

L. A. Erlygin, P. D. Proskura, A. A. Zaytsev

arXiv:2609.35043v1cs.CV

TL;DR

Selective open-set recognition needs uncertainty scores that rank decisions by their actual error risk, but KL summaries of gallery-aware posteriors can disagree with that ordering. MPRisk scores decision-conditioned risks and rejection non-specificity using four validation-tuned weights. Across nine image, audio, and text benchmarks, it is best or tied-best at all image and audio operating points and most text operating points, with bootstrap-confirmed gains over HolUE on five benchmarks.

  • Problem

    Selective OSR needs scores aligned with the error risk of the chosen decision, but information-gain summaries of gallery-aware posteriors are not generally ordered by decision risk.

    Source: §3.2 · p. 5, §3.4 · p. 6

  • Method

    MPRisk reuses the Bayesian posterior to score decision-conditioned false-acceptance, misidentification, and false-rejection risks, plus rejection non-specificity, with four nonnegative validation-tuned weights.

    Source: §7 · p. 18

  • Results

    MPRisk is best or tied-best at all image and audio operating points and most text operating points, with bootstrap-confirmed gains over HolUE on five benchmarks.

    Source: §6 · p. 16, §7 · p. 18

  • Takeaways & Limitations

    Decision-conditioned posterior risks provide a simple basis for selective OSR, while KL features can supply complementary information when sufficient validation data are available.

    Source: §7 · p. 18

    Tuned MPRisk depends on a validation split matched to the test operating point, and robustness under gallery shift and validation/test cost mismatch was not evaluated.

    Source: §6 · p. 18

Abstract

from arXiv · show

In open-set recognition (OSR), a probe must either be identified as one of the known gallery classes or rejected as unknown, so three error types coexist: false acceptance, false rejection, and misidentification. An uncertainty score for selective recognition should rank probes by the risk of the decision the system has made. Bayesian gallery-aware models such as Holistic Uncertainty Estimation (HolUE) summarize the posterior over known and unknown classes by Kullback--Leibler (KL) divergence components and map them to an uncertainty score with a supervised nonlinear calibrator. We show that the KL summary is not generally monotone in decision risk: linear fusion of the KL components tuned on validation data yields negative filtering quality on several benchmarks. We propose MPRisk, a mixed-prior posterior decision-risk score that keeps the same Bayesian posterior but directly scores the error events associated with the selected decision: false-acceptance, misidentification, and false-rejection risks, plus a non-specificity penalty for rejections, enabled by modeling unknown identities as a continuous component. Four nonnegative weights tuned on a validation set suffice for ranking; no nonlinear supervised model is required. Across nine image, audio, and text benchmarks, MPRisk achieves the best or tied-best Prediction Rejection Ratio at every operating point on the image and audio benchmarks and on most text operating points, with bootstrap-confirmed gains over HolUE on five benchmarks (up to $+0.19$ PRR) at comparable or lower runtime.

1. INTRODUCTION

Open-set recognition must identify known probes and reject unknown ones, while selective recognition requires ranking decisions by their error risk. The paper proposes MPRisk to score decision-specific errors directly and evaluates it alongside analyses of KL-based uncertainty.

Motivation

  • Open-set recognition requires identifying samples from known classes while rejecting samples from unknown classes.A standard pipeline accepts a probe when its best similarity exceeds a threshold, then assigns the closest gallery identity; otherwise it rejects the probe.Source: §1 · p. 1
  • Selective recognition ranks decisions by how well uncertainty scores place erroneous decisions ahead of correct ones, amid false acceptance, false rejection, and misidentification.These error types arise because an unknown probe can be accepted, a known probe rejected, or a known probe assigned the wrong identity.Source: §1 · pp. 1–2
  • HolUE summarizes the posterior over known and unknown classes with two KL components and maps them to uncertainty using a supervised nonlinear calibrator.The paper frames the central question as whether information-gain summaries appropriately rank OSR decision risk.Source: §1 · p. 2

Method

  • MPRisk directly scores false-acceptance, misidentification, and false-rejection risks, adding a rejection non-specificity penalty enabled by a continuous unknown-identity component.Four nonnegative validation-tuned weights suffice for ranking, without a nonlinear fitted model.Source: §1 · p. 2

Analysis and evaluation

  • KL summaries are not generally ordered by decision risk, and linear fusion of HolUE KL features performs poorly on several benchmarks while nonlinear calibration recovers useful rankings.The paper evaluates the approach on nine image, audio, and text benchmarks, including ablations, error-type analyses, calibration diagnostics, and runtime measurements.Source: §1 · p. 2

2. RELATED WORK

Related work covers open-set recognition, probabilistic embeddings, and selective prediction, then positions MPRisk as a decision-conditioned approach to uncertainty estimation. It retains a continuous unknown-identity component while distinguishing ranking from probability calibration.

2.1. Open-set recognition and open-world learning

  • Open-set methods commonly improve recognition or unknown detection, whereas this work estimates reliability for a fixed OSR system without changing its embedding model, gallery, or acceptance rule.Prior approaches include classifier-head or activation-statistic modifications and methods targeting known–unknown separation.Source: §2.1 · p. 3

2.2. Open-set recognition in text

  • Text OSR includes intent and topic classification protocols, authorship attribution, and verification, and this work applies the Bayesian model to two modality groups.CLINC150 is cited as a standard intent-classification benchmark; topic datasets are adapted by treating subsets of topics as known.Source: §2.2 · p. 3

2.3. Aleatoric uncertainty and probabilistic embeddings

  • In the fixed-backbone, fixed-gallery pipeline, probabilistic embeddings represent aleatoric uncertainty; MPRisk uses the vMF distribution to average probe-level class probabilities and compute unknown-posterior non-specificity.The SCF model predicts a vMF distribution on the unit hypersphere, where larger concentration corresponds to a higher-quality embedding.Source: §2.3 · p. 4

2.4. Selective prediction and calibration

  • Selective prediction ranks uncertain samples for rejection, while calibration measures whether predicted probabilities match empirical frequencies; the paper evaluates these as separate goals.Under 0–1 loss, the optimal score is posterior error probability rather than an entropy- or divergence-based functional.Source: §2.4 · p. 4

2.5. Relation to prior work

  • The paper identifies its distinction as decomposing OSR uncertainty into decision-conditioned posterior risks while retaining a continuous unknown-identity component.Existing gallery-aware estimators use raw posterior confidence or information-theoretic summaries with supervised calibration; existing selective-prediction scores ignore accept/reject structure.Source: §2.5 · p. 4

3. BACKGROUND AND PROBLEM STATEMENT

Open-set recognition must identify probes from known classes and reject unknown probes, while uncertainty scores should rank decisions by their error risk. The section introduces a Bayesian posterior over known and continuous unknown identities and explains why KL summaries need not track decision risk.

3.1. Open-set recognition protocol

  • The OSR protocol applies a fixed recognition decision, while uncertainty scoring orders probes for possible rejection without changing that decision.The Bayesian class variable combines discrete known classes with a continuous interval for unknown identities; its prior assigns mass β to that continuum.Source: §3.1 · p. 4, §3.3 · p. 5

3.2. What the uncertainty score should measure

  • Uncertainty should rank probes by the error risk of the decision actually made, conditioning on whether the system accepted or rejected the probe.Unknown posterior mass signals false-acceptance risk for accepted probes but supports a correct rejection for rejected probes.Source: §3.2 · p. 5

3.3. Bayesian mixed-prior posterior

  • HolUE summarizes the posterior with two KL components and uses a validation-trained nonlinear calibrator to produce an uncertainty score.The posterior is over known and unknown classes; the probe-level posterior integrates over the SCF embedding distribution, with a mean-embedding approximation used in practice.Source: §3.3 · p. 5, §3.4 · p. 6

3.4. KL summaries and decision risk

  • For N ≥3, entropy and maximum posterior probability need not rank probes alike, and KL components can change meaning across accept and reject decisions.KL aggregates posterior evidence across actions, whereas decision risk concerns the action taken; the relation of KL2 to error can reverse between accepted and rejected probes.Source: §3.4 · p. 6
  • The raw KL summary is nearly uncorrelated with error ordering on IJB-C and anti-correlated on Yahoo Answers.Table 1 also reports inversion rate and KL AUROC/AUPRC as measures of raw-summary ordering and any-error detection.Source: §3.4 · p. 6

4. MPRISK METHOD

MPRisk combines posterior risks conditioned on the chosen OSR decision with a penalty for non-specific rejections. Its four nonnegative components are weighted using validation data, without fitting a nonlinear ranking model.

4.1. Decision-conditioned risk components

  • Three decision-conditioned posterior risks cover false acceptance and misidentification for accepted probes and false rejection for rejected probes.Under equal error costs, their sum is the posterior probability that the taken decision is wrong.Source: §4.1 · pp. 6–7

4.2. Mixed-prior reject non-specificity

  • The non-specificity penalty flags rejections that place substantial mass on unknown identities without concentrating on a specific identity.A diffuse unknown-identity posterior can indicate poor embedding quality, unlike the concentrated posterior induced by a high-quality unknown probe; the penalty is not itself an error probability.Source: §4.2 · p. 7

4.3. Final score, weight tuning, and probability calibration

  • MPRisk adds the four components with nonnegative weights selected on a disjoint validation set, avoiding a nonlinear fitted ranking model.The weights represent relative component costs, and their tuning uses validation error labels through the PRR objective; an optional scalar calibrator preserves ranking up to ties.Source: §4.3 · p. 8

4.4. Computational cost

  • Posterior computation costs O(Kd) per probe, while the risk formula adds O(1) overhead.Weight tuning and probability calibration are offline procedures; MPRisk raw requires no tuning.Source: §4.4 · p. 8

5. EXPERIMENTS

The experiments compare uncertainty estimators across image, audio, and text OSR benchmarks under fixed recognition decisions and disjoint validation splits. They use PRR as the primary filtering metric and compare estimators under matched validation access.

5.1. Datasets, protocols, and baselines

  • The evaluation covers four image and audio benchmarks and five text benchmarks, with fixed recognition decisions and validation splits disjoint from test identities, authors, and topics.The compared methods receive the same validation data for tuning, supervised fitting, and calibration.Source: §5.1 · p. 8

5.2. Metrics

  • PRR measures F1 filtering as probes are removed in decreasing uncertainty order, with 1 for oracle filtering, 0 for random, and negative values for worse-than-random scores.The evaluation also includes per-error detection AUROC, paired bootstrap intervals, calibration error, and runtime.Source: §5.2 · pp. 8–9

5.3. Compared estimators under a fixed validation budget

  • The fixed-validation comparison includes linear KL fusion, tuned simple scores, rejection-gated SCF, supervised error detectors, and a hybrid KL-plus-MPRisk estimator.The supervised logistic and MLP baselines use a 14-dimensional feature vector; the hybrid tests whether information gain and decision risk are complementary.Source: §5.3 · p. 9

5.4. Main results: image, Whale, and audio

MPRisk performs strongly across image, audio, and text filtering benchmarks, though HolUE is stronger on Yahoo Answers. The reported results include gains at higher image and audio FPIR settings and varying text-task performance.

5.4. Main results: image, Whale, and audio

  • At all 12 image, Whale, and audio operating points, MPRisk is best or tied-best; at FPIR 0.2 it reaches 0.91 on IJB-C (+0.18 over HolUE) and 0.94 on Whale (+0.11 over HolUE).MPRisk raw is weaker than the tuned version: 0.44 vs. 0.83 on IJB-C at FPIR 0.1.Source: Figure 2 · p. 10, §5.4 · p. 10

5.5. Main results: text

  • MPRisk performs best on DBPedia, at all five CLINC150 points, at most PAN-20-AV points, and at AGNews extreme points, while HolUE is consistently stronger on Yahoo Answers.Examples include 0.92 vs. 0.81 for HolUE on DBPedia at FPIR 0.3 and 0.37 vs. 0.18 on PAN-20-AV at FPIR 0.1.Source: §5.5 · p. 10, Table 3 · p. 10

5.6. Feature-space study

MPRisk’s decision-risk features combine effectively with linear tuning, unlike KL features, whose linear fusion performs poorly; hybrid KL+MPRisk improves results in several settings.

5.6. Feature-space study

  • On CLINC150 and PAN-20-AV, MPRisk dominates the F1 curve and lowers FPIR fastest, whereas HolUE’s F1 curve is higher on Yahoo Answers.Yahoo Answers’ error mass consists almost entirely of false rejections.Source: Figure 3 · p. 11
  • MPRisk reaches PRR 0.76/0.84/0.91 on IJB-C with four nonnegative weights, while linear KL fusion is negative on IJB-C and IJB-B.HolUE’s nonlinear calibration of the same KL evidence reaches 0.76/0.81 on IJB-C.Source: §5.6 · p. 11, Table 4 · p. 11
  • Tuned logistic regression exceeds MPRisk only modestly and not uniformly, while the 14-input supervised MLP varies substantially across datasets.The logistic comparison includes Whale 0.86 vs. 0.81 at FPIR 0.05 and DBPedia 0.89 vs. 0.72 at FPIR 0.1.Source: §5.6 · p. 11, Table 5 · p. 12
  • Hybrid KL+MPRisk obtains the highest PRR at 9 of 12 image/audio operating points and on AGNews and PAN-20-AV.On PAN-20-AV it reaches 0.43 at FPIR 0.1, +0.25 over HolUE.Source: §5.6 · p. 12, Table 5 · p. 12

5.7. Component ablation

Ablations show that decision-risk components need tuning across decision regimes; the continuous unknown component adds a rejection-specific signal, while aggregate error detection entails dataset-specific trade-offs.

5.7. Component ablation

  • The tuned combination is uniformly near the per-cell maximum across variants, while no fixed weighting serves all decision regimes.Single decision-conditioned components are zero on complementary decision sets and can rank half the probes arbitrarily.Source: §5.7 · p. 12, Table 7 · p. 13

5.8. Is the mixed prior necessary?

  • The continuous unknown component adds a rejection-specific signal and helps explain suspicious rejects, while its aggregate PRR gains are smaller than those from tuning the three standard risks.Under a collapsed unknown class, the first three ordinary risks are unchanged and rNS is unavailable.Source: §5.8 · p. 13, Table 8 · p. 13

5.9. Error-type detection

  • On image and audio benchmarks, tuned MPRisk is the best overall error detector and the best FA and ID detector, with IJB-C FA 0.98 and ID 0.97.It incurs a moderate cost in FR AUROC relative to threshold-based scores.Source: §5.9 · p. 13, Table 10 · p. 14
  • On Yahoo Answers, MPRisk’s FR AUROC is 0.90 versus HolUE’s 0.92, while its FA AUROC is 0.20; on PAN-20-AV it is best for any-error and FR detection at 0.68 and 0.66.The aggregate PRR objective weights rare FA errors little on Yahoo; on PAN-20-AV, MPRisk raw is the best ID detector at 0.91.Source: §5.9 · p. 14, Table 9 · p. 14

5.10. Statistical significance

Bootstrap comparisons show gains over HolUE on several benchmarks, while calibration quality varies and runtime remains comparable or lower for MPRisk.

5.10. Statistical significance

  • Bootstrap intervals exclude zero for MPRisk’s PRR gain over HolUE on IJB-B, Whale, VoxBlink, and DBPedia; Yahoo Answers is a significant loss.The reported gains are IJB-B 0.12 [0.07, 0.17], Whale 0.06 [0.04, 0.08], VoxBlink 0.08 [0.06, 0.10], and DBPedia 0.19 [0.15, 0.25].Source: §5.10 · p. 14, Table 12 · p. 15

5.11. Calibration

  • MPRisk cal has lower ECE on IJB-C, Yahoo Answers, and PAN-20-AV, while HolUE is lower on Whale, VoxBlink, DBPedia, and CLINC150.Calibration does not change the selective-recognition ranking, so calibrated scores provide probability readouts without ranking cost.Source: §5.11 · pp. 14–15

5.12. Runtime

  • Tuned MPRisk and MPRisk cal are comparable to or cheaper than HolUE on all benchmarks and over four times cheaper on PAN-20-AV.PAN-20-AV runtime is 31–32 ms per probe versus 132 ms for HolUE.Source: §5.12 · p. 15

6. DISCUSSION

The discussion attributes MPRisk’s results to decision-conditioned features that combine linearly, while noting task-specific trade-offs, validation requirements, and limits on the tested scope.

6. DISCUSSION

  • MPRisk outperforms calibrated KL scores at 25 of 27 image/audio operating points and on most text operating points, with gains whose bootstrap intervals exclude zero on five benchmarks.The overall comparison includes ties or inconclusive differences on three benchmarks and one clear HolUE win on Yahoo Answers.Source: §6 · pp. 15–16, Table 13 · p. 16
  • On Yahoo Answers, HolUE’s quality-driven KL2 is near-ideal for errors that are almost purely false rejections; a hybrid recovers most of its advantage while preserving MPRisk’s gains elsewhere.Tuning λ against a deployment cost matrix is the other proposed remedy.Source: §6 · p. 16
  • The continuous unknown component contributes a rejection-specific signal unavailable to a collapsed reject class and makes suspicious rejections interpretable.The interpretation is high unknown mass without a specific unknown hypothesis.Source: §6 · p. 16
  • With limited validation data, tuned MPRisk is the lower-risk choice and MPRisk raw is a tuning-free fallback; abundant validation errors make hybrid or logistic detectors reasonable.On moderate validation sets, performance tracks how closely the hypothesis class matches the task rather than raw capacity.Source: §6 · p. 16
  • Tuned MPRisk depends on validation matched to the test operating point, and robustness under gallery shift and validation/test cost mismatch was not evaluated.The method also inherits Bayesian-posterior and SCF embedding-distribution assumptions.Source: §6 · p. 18

7. CONCLUSION

MPRisk directly ranks open-set recognition decisions by their posterior error risks, with four validation-tuned weights. Across nine benchmarks, it is best or tied-best at all image and audio operating points and most text operating points, with bootstrap-confirmed gains over HolUE on five benchmarks.

7. CONCLUSION

  • MPRisk scores false-acceptance, misidentification, and false-rejection risks for the selected decision, adding a closed-form non-specificity penalty for confident but poorly supported rejections.Modeling unknown identities as a continuous component enables the penalty; four nonnegative validation-tuned weights suffice for ranking.Source: §7 · p. 18
  • Across nine image, audio, and text benchmarks, MPRisk is best or tied-best at all image and audio operating points and most text operating points, with bootstrap-confirmed gains over HolUE on five benchmarks at comparable or lower runtime.The authors report that combining MPRisk with KL scores yields further gains in several settings.Source: §7 · p. 18
  • Linear fusion of KL features performs poorly, while MPRisk components remain competitive with supervised error detectors; future work includes cost-matrix tuning and transfer to selective generation and hallucination detection.The authors suggest decision-conditioned posterior risks as a simple basis for selective OSR, with KL features providing complementary information when validation data are sufficient.Source: §7 · p. 18

Related papers

Loading 2609.35043v1…