PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning with a Reject Option: A survey

Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, Jesse Davis

arXiv:2107.11277v3cs.LGcs.AI

TL;DR

Models can make costly mistakes because they always predict, even for uncertain or unfamiliar inputs. This survey formalizes rejection types and organizes evaluation, architectures, learning techniques, applications, and connections across the field, concluding with a higher-level overview of machine learning with rejection.

  • Problem

    Always predicting can produce serious mispredictions on ambiguous or unfamiliar inputs, including in safety-sensitive applications.

  • Method

    The survey formalizes ambiguity and novelty rejection and structures the field by reviewing evaluation strategies, architectures, and learning techniques.

  • Results

    The paper provides a higher-level overview of existing research on machine learning with rejection.

  • Takeaways & Limitations

    The survey identifies safety-sensitive domains as application areas where rejection supports cautious predictions.

  • Takeaways & Limitations

    There is no standard benchmark set, and custom or proprietary data make rejection approaches difficult to compare.

Abstract

from arXiv · show

Machine learning models always make a prediction, even when it is likely to be inaccurate. This behavior should be avoided in many decision support applications, where mistakes can have severe consequences. Albeit already studied in 1970, machine learning with rejection recently gained interest. This machine learning subfield enables machine learning models to abstain from making a prediction when likely to make a mistake. This survey aims to provide an overview on machine learning with rejection. We introduce the conditions leading to two types of rejection, ambiguity and novelty rejection, which we carefully formalize. Moreover, we review and categorize strategies to evaluate a model's predictive and rejective quality. Additionally, we define the existing architectures for models with rejection and describe the standard techniques for learning such models. Finally, we provide examples of relevant application domains and show how machine learning with rejection relates to other machine learning research areas.

1 Introduction · 2 The learning with reject problem setting

Machine learning with rejection extends prediction models with abstention when inputs are likely to produce errors. The survey formalizes this problem through model architectures and two rejection types: ambiguity and novelty.

  • 1 Introduction: Always returning a prediction can be harmful when classes are difficult to distinguish or test examples differ substantially from training data.These risks are especially consequential in applications such as medicine.
  • 1 Introduction: Rejecting uncertain predictions can improve performance on retained examples and increase user trust by avoiding mispredictions.The survey organizes its overview around eight key research questions and identifies characteristics that structure rejection methods.
  • 2 The learning with reject problem setting: In the standard supervised setting, the learner approximates an unknown target function f from i.i.d. training data by selecting a hypothesis with low expected risk.The target space may be discrete, continuous, or probabilistic, and risk is defined using a suitable loss function.
  • 2.1 Models with a reject option: A rejection model extends the output space with ® and represents prediction as a pair (h, r), where the rejector abstains when the predictor faces heightened misprediction risk.The rejector may use the input, confidence or probability values, or both.
  • 2.1 Models with a reject option: The survey identifies three architectures: separated rejectors filter inputs independently, dependent rejectors assess predictor outputs, and integrated rejectors combine prediction and rejection in one model.Integrated models treat rejection as an additional class, while separated and dependent designs preserve distinct predictor and rejector roles.
  • 2.2 Types of rejection: The survey distinguishes ambiguity rejection, caused by ambiguous targets or model limitations, from novelty rejection, caused by insufficiently represented regions of the feature space.Ambiguity often occurs near classification decision boundaries, whereas novelty reflects limited or absent training data.
  • 2.2.1 Ambiguity Rejection: Ambiguity rejection addresses regions where the predictor cannot capture the true relationship because the target is nondeterministic, data are erroneous, or the hypothesis space excludes the target function.Examples include overlapping classes, high target variance, missing features, and using linear models for nonlinear concepts.
  • 2.2.2 Novelty Rejection: Novelty rejection abstains on examples sufficiently different from training data, whose distributional mismatch or rarity prevents reliable prediction.This is particularly important in medical applications with incomplete coverage; in sleep-stage scoring, accuracy was 66.8% for children versus 77.7% for adults, and rejection mitigated incorrect predictions for most children.

3 Evaluating models with rejection

Models with rejection must balance predictive accuracy on accepted examples against coverage, while evaluation can target a fixed rejection rate, the performance–rejection trade-off, or costs. Prediction, rejection, and combined-quality metrics each capture different aspects, with combined metrics offering holistic assessment but less diagnostic detail.

  • Core objectives: A model with rejection aims for high accuracy on predictions it returns and high coverage across test examples, but these goals compete.Coverage is equivalently associated with a low rejection rate, while restricting predictions to confident cases can increase accuracy by reducing coverage.
  • Evaluation categories: Evaluation uses three broad approaches: fixed-rejection-rate metrics, performance–rejection trade-off curves, and cost functions.Fixed-rate evaluation can use standard metrics on non-rejected examples; trade-off curves plot rejection rate against predictor performance; cost functions require prediction, misprediction, and rejection costs.
  • Fixed rejection rate: At a given rejection rate, prediction quality measures accepted-example performance, rejection quality measures rejection of misclassified examples, and combined quality assesses both.Prediction quality may use accuracy, F-scores, or fairness metrics, whereas combined quality can combine prediction and rejection quality into a single measure.
  • Evaluation limitations: Combined-quality metrics provide holistic assessment but make it difficult to identify whether the predictor or rejector causes poor performance, while cost functions are interpretable but require domain-based cost setting.Cost functions also support coherent optimization and testing, but their usefulness depends on the user specifying appropriate costs from domain knowledge.
  • Performance–rejection trade-off: Accuracy-Reject Curves vary rejection rates from 0% to 100% and plot rejection rate against prediction quality, with higher curves indicating better performance.When curves cross, overall performance can be assessed using the area under the curve; in the illustrated case, a proposed model outperforms baselines only below a rejection rate of 0.01.

4 Separated rejector

Separated rejectors independently filter examples, typically for novelty rejection, before invoking a predictor. They are learned using density estimation, one-class classification, or novelty scoring, with a threshold controlling abstention.

  • Architecture: Separated rejectors typically target novelty rejection and compute r: X → R independently of the predictor’s output.They can also support ambiguity rejection in some cases.
  • Architecture: Examples with r(x) < τ are rejected; otherwise, the model uses predictor h to make a prediction.The threshold τ therefore determines whether each example is abstained on or predicted.
  • Learning the rejector: Learning separates rejector training from predictor training and commonly uses p(X) estimation, one-class classification, or novelty-scoring methods.These approaches align with identifying unlikely, unexpected, or out-of-distribution examples.
  • Learning the rejection threshold τ: The threshold τ may be set using domain knowledge, adversarial examples, or estimated novelty contamination when the number of novelties is unknown.One example sets τ to reject all introduced adversarial examples.
  • Benefits and drawbacks: Separated rejectors are predictor agnostic, can augment existing predictors, and reduce the number of predictions by filtering examples first.The reduced prediction workload is especially advantageous when prediction has high computational cost.

5 Dependent rejector

Dependent rejectors use a confidence function derived from the predictor and input to abstain below a threshold, supporting ambiguity and novelty rejection. Designing them requires selecting an appropriate confidence measure and setting its rejection threshold.

  • Dependent rejector: A dependent rejector analyzes predictor outputs through a confidence function ch: X → [0, 1] and rejects examples when confidence falls below τ.The confidence function estimates how likely the predictor is to make a correct prediction; score functions can be transformed into [0, 1].
  • Learning a dependent rejector: The learning task consists of choosing a confidence function that reflects predictive confidence and setting the rejection threshold τ.The confidence measure should indicate target variability or predictor bias for ambiguity rejection, and similarity to training data for novelty rejection.
  • Types of confidence function: Conditional-probability confidence functions support ambiguity rejection, while class-conditional densities support novelty rejection by identifying rare samples.Common density estimators include generative predictors such as Gaussian Mixture Models, while probability estimates may be obtained by post-processing predictor outputs.
  • Types of confidence function: Sensitivity-based confidence measures permit ambiguity rejection by quantifying prediction robustness to perturbations of model parameters or test examples.Methods may use prediction variance across perturbed predictors, invariance under small noise or transformations, or the minimum perturbation that changes the predicted label.
  • Learning the rejection threshold τ: Thresholds can be set using domain knowledge, user constraints, or empirical objectives; a single global τ is simple, transparent, and usually effective.Chow’s rule learns the optimal τ by minimizing a risk function containing expected error and rejection rates.

6 Integrated rejector

An integrated rejector combines prediction and rejection in one model, making their roles indistinguishable and treating rejection as part of the output or decision process. It can be learned through model-agnostic objectives or model-specific adaptations, with joint optimization offering benefits but requiring careful design.

  • Architecture: An integrated rejector combines the predictor and rejector into a single model, so their respective roles cannot be distinguished.Conceptually, rejection may be represented as an additional output.
  • Learning approaches: Integrated rejectors are learned either with model-agnostic objectives penalizing predictions and rejections or by adapting a specific predictor to include rejection.The model-agnostic approach can use potentially any existing optimizer, whereas the model-specific approach integrates rejection into the predictor’s decision process.
  • Model-agnostic learning: For classification, specialized cost-based or surrogate losses jointly optimize prediction and rejection, while other tasks use ad-hoc loss functions.The rejection cost satisfies Cr ∈ (0, 1/2] in the classification setting; surrogate losses can support efficient optimization when discrete losses are computationally difficult.
  • Model-agnostic learning: Adding a K + 1 reject class incorporates rejection directly into multiclass outputs, and lowering Cr increases the chance of rejection.The reject class can be learned without actual examples by assigning ambiguous examples positive rejection scores, or with artificially generated rejection examples such as adversarial examples.
  • Model-specific learning: Model-specific integrated rejectors include SVMs that reject points between class-prioritized hyperplanes and neural networks that modify output layers or use a dedicated rejection head.In integrated SVMs, examples between the learned hyperplanes form the rejection region; neural networks can reject when a sigmoid rejection head is below 0.5 or when class-prioritized networks disagree.
  • Benefits and drawbacks: Jointly optimizing prediction and rejection can improve performance because the components affect each other, but designing the architecture remains a potential drawback.The integrated design is also a unique model, while the supplied passage truncates the discussion of its additional drawbacks.

7 Combining multiple rejectors

Combining multiple rejectors enables models to perform multiple rejection types, including ambiguity and novelty rejection. The appropriate combination strategy depends on whether the rejectors’ rejection regions overlap.

  • 7 Combining multiple rejectors: Combining multiple rejectors enables simultaneous ambiguity and novelty rejection, whereas most rejectors target a single rejection type.Overlapping rejection regions require deeper analysis to determine the underlying rejection type.
  • 7 Combining multiple rejectors: When rejection regions do not overlap, rejectors can be combined with a logical or-rule that rejects an example if any rejector rejects it.The resulting rejection can be assigned the corresponding rejection type when the rejectors’ regions do not overlap or the type is not needed.
  • 7 Combining multiple rejectors: When rejection regions overlap, a simple or-rule may not identify the rejection reason because different rejectors can abstain for different reasons.Existing approaches therefore carefully select the order in which rejectors are evaluated.
  • 7 Combining multiple rejectors: Overlapping rejectors are typically organized in multi-step architectures that stack rejectors or use multiple models with rejection.The survey cites both rejector-stacking and multiple-model approaches as examples.

8 Applications of machine learning models with rejection

Machine learning with rejection supports cautious predictions in safety-sensitive applications, although practical studies remain limited. Existing work spans biomedical, engineering, economics, and image-recognition applications, using ambiguity rejection, novelty rejection, or both.

  • Applications: Rejection is applied in safety-sensitive domains to avoid potentially serious consequences from incorrect decisions, including accidents, industrial breakdowns, and incorrect medical diagnoses.The survey notes that practical application papers remain limited despite these motivations.
  • Biomedical applications: Biomedical research primarily uses ambiguity rejection for disease detection and classification, with medical experts verifying predictions when model confidence is insufficient.High-confidence detection results are automatically translated into diagnoses; otherwise, expert verification is required.
  • Biomedical applications: Biomedical studies also use supervised novelty rejection for brain-activity classification and both rejection types for sleep-stage scoring.Confidence metrics derived from neural-network classifier activity support sleep-stage rejection.
  • Engineering applications: Engineering applications reject ambiguous gas classifications when two classifiers disagree, requiring consensus before classification.A similar disagreement-based ambiguity rejection technique is also used in defect detection.
  • Economics applications: Economics applications include classifying dollar bills with LVQ using confidence metrics for both rejection types and evaluating rejection techniques for loan decisions.The two applications address currency-value classification and whether to grant a loan.
  • Image recognition applications: Image-recognition systems use confidence-based dependent rejectors for text styles and handwritten numbers, supporting both ambiguity and novelty rejection.Another application identifies walkers from their footprints.

9 Link to other research areas

Learning with rejection is closely connected to uncertainty quantification, anomaly detection, active learning, class-incremental learning, delegation, and meta-learning. These fields provide related uncertainty estimates, detection mechanisms, training strategies, deployment adaptations, alternative architectures, and rejector guidance.

  • Uncertainty quantification: Uncertainty quantification distinguishes aleatoric uncertainty from epistemic uncertainty, both of which can motivate ambiguity or novelty rejection.Aleatoric uncertainty reflects data randomness, whereas epistemic uncertainty reflects limited knowledge, such as small training sets or incorrect model bias.
  • Uncertainty quantification: Learning with rejection can leverage calibrated uncertainty estimates to abstain when uncertainty is high, although calibration is not always necessary.Calibrated estimates meaningfully convey uncertainty levels and can be important for rejection decisions.
  • Anomaly detection: Anomaly detection is closely linked to novelty rejection, but unsupervised rejectors require specialized confidence metrics because hard decision surfaces may not exist.Anomaly detectors are often used within separate rejector architectures, while unsupervised settings challenge confidence measures designed for supervised classification.
  • Active learning: Active learning addresses uncertainty during training to reduce labeling costs, whereas learning with rejection addresses uncertainty at test time to prevent mispredictions.Rejected test examples can be queried when an oracle is available, identifying new data types or fine-tuning the decision boundary.
  • Class-incremental learning: Novelty rejection and class-incremental learning share an open-world focus, but class-incremental learning targets novel classes and retrains models using detected examples.Novelty-rejected examples can be incorporated into incremental pipelines, including as prototypes in a k-Nearest Neighbors model.
  • Meta-learning and delegation: Meta-learning can guide rejection by deriving knowledge about classifier strengths and weaknesses and identifying features informative for determining when to reject.Delegation is another related approach, sending low-confidence examples to a more specialized classifier rather than usually having the user inspect them.

10 Conclusions and perspectives

The survey organizes machine learning with rejection around formal rejection conditions, evaluation strategies, architectures, learning techniques, and applications. It identifies standardization, partial abstention, and extensions beyond classification as key directions for future research.

  • Key findings: The survey distinguishes ambiguity rejection, for examples with ambiguous target values, from novelty rejection.Ambiguity can arise from a nondeterministic feature–target relation or limitations of the hypothesis space.
  • Key findings: Model evaluation must consider the trade-off between predictive quality and the proportion of rejected examples.The survey categorizes evaluation methods by fixed rejection rate, overall prediction–rejection trade-off, and cost functions.
  • Key findings: The survey categorizes rejection systems as separated, dependent, or integrated rejector architectures.Separated rejectors are usually learned independently; dependent rejectors use confidence functions and require a rejection threshold, while integrated rejectors are learned within the predictive model.
  • Future research directions: There is no standard benchmark set for comparing models with rejection, because studies commonly use custom or proprietary data.The survey also notes that applying multiple rejector-evaluation strategies provides a better view of model performance.
  • Future research directions: Future research should investigate partial abstention and rejection algorithms for regression, forecasting, clustering, semi-supervised learning, and self-supervised feedback loops.Partial abstention extends rejection from complete predictions to parts of structured outputs such as multi-label predictions; only a handful of studies address non-classification domains.

Declarations · Funding

The paper acknowledges funding from VLAIO, FWO-Vlaanderen, KU Leuven, and the Flemish Government. Support includes Baekeland PhD mandates, an aspirant grant, KU Leuven research funds, and an artificial-intelligence research programme.

  • Funding: Kilian Hendrickx and Dries Van der Plas received VLAIO funding through Baekeland PhD mandates HBC.2017.0226 and HBC.2019.2615.The mandates supported KH and DV, respectively.
  • Funding: Lorenzo Perini received funding from FWO-Vlaanderen through aspirant grant 1166222N.
  • Funding: Jesse Davis was partially supported by KU Leuven research funds C14/17/070.
  • Funding: Lorenzo Perini, Jesse Davis, and Wannes Meert received funding from the Flemish Government.
  • Funding: The Flemish Government funding was provided under the “Onderzoeksprogramma Artifici¨ele Intelligentie (AI.
  • Funding: The funding acknowledgements identify support for individual authors through national, regional, university, and innovation-oriented programmes.

Ethics approval … Authors’ contributions

The authors divided responsibilities across concept development, literature study, categorization, manuscript preparation and revision, funding acquisition, and supervision.

  • Authors’ contributions: JD and WM developed the paper’s concept.
  • Authors’ contributions: KH, LP, and DVdP conducted the literature study.
  • Authors’ contributions: KH, LP, DVdP, WM, and JD performed the categorization.
  • Authors’ contributions: KH, LP, and DVdP prepared the original draft.
  • Authors’ contributions: WM and JD handled review and editing, while all five authors contributed to revision.
  • Authors’ contributions: WM and JD acquired funding and supervised the work.
Loading 2107.11277v3…