Source-linked AI summary
Selective Classification for Deep Neural Networks
Yonatan Geifman, Ran El-Yaniv
TL;DR
Selective classification had not been considered for deep neural networks, motivating methods that reduce error by rejecting uncertain instances while retaining coverage. The paper learns a rejection function and threshold for a trained neural classifier to guarantee a user-specified risk with high probability. On ImageNet, it guarantees 2% top-5 error with probability 99.9% at almost 60% test coverage.
Problem
Selective classification had not yet been considered for deep neural networks, although it aims to reduce error by abstaining while keeping coverage high.
Method
The method learns a rejection function and threshold for a given trained neural classifier to guarantee a desired risk with high probability.
Results
2% top-5 ImageNet error was guaranteed with probability 99.9% and almost 60% test coverage.
Takeaways & Limitations
The empirical study showed that guaranteed risk control is achievable and that the methods can be used for mission-critical deep-learning tasks.
Takeaways & Limitations
The approach starts from a trained neural classifier and studies selective classification only under 0/1 loss, leaving joint training and other losses or regression open.
Abstract
from arXiv · showhide
Selective classification techniques (also known as reject option) have not yet been considered in the context of deep neural networks (DNNs). These techniques can potentially significantly improve DNNs prediction performance by trading-off coverage. In this paper we propose a method to construct a selective classifier given a trained neural network. Our method allows a user to set a desired risk level. At test time, the classifier rejects instances as needed, to grant the desired risk (with high probability). Empirical results over CIFAR and ImageNet convincingly demonstrate the viability of our method, which opens up possibilities to operate DNNs in mission-critical applications. For example, using our method an unprecedented 2% error in top-5 ImageNet classification can be guaranteed with probability 99.9%, and almost 60% test coverage.
1 Introduction
Selective prediction reduces error by abstaining on uncertain instances while preserving coverage, but it had rarely been studied for deep neural networks. This paper learns rejection functions for trained neural classifiers and demonstrates high-confidence risk control on CIFAR and ImageNet.
- Selective prediction trades coverage for lower error by abstaining from predictions when the classifier is uncertain.
- Deep neural networks had not previously been considered for selective prediction, despite the broader reject-option literature.
- The method learns a rejection function for an already-trained neural classifier and selects a threshold to guarantee a desired risk with high probability.
- On CIFAR, softmax response and MC-dropout achieved nearly identical selective-classification results, whereas softmax response was significantly superior on ImageNet.
- 2% top-5 ImageNet error was guaranteed with probability 99.9% at almost 60% test coverage.
2 Problem Setting
A selective classifier combines a standard classifier with a binary selection function that accepts or rejects each input. Its performance is characterized by risk and coverage, with the goal of meeting a target risk while maximizing coverage.
- A selective classifier is a pair (f, g), combining a classifier f with a selection function g.
- The selection function accepts an input when g(x) = 1 and abstains when g(x) = 0.
- Coverage is the probability mass of the non-rejected region, while selective risk measures performance on accepted predictions.
- Risk and coverage can be traded off and summarized by a risk-coverage curve.
- Given a classifier, training sample, confidence parameter, and target risk, the goal is to learn a selection function meeting the risk target while maximizing coverage.
3 Selection with Guaranteed Risk Control
The SGR procedure constructs a selection function around an existing classifier to meet a desired selective-risk target with high-probability guarantees. It uses confidence-ranked rejection, empirical risk bounds, and binary search to select coverage while preserving the guarantee.
- Selection with Guaranteed Risk Control: SGR constructs a selection function for a given classifier and confidence-rate function, without assuming a particular form for the confidence measure.Its goal is guaranteed performance for a specified risk target.
- Selection with Guaranteed Risk Control: The algorithm takes a classifier, confidence-rate function, confidence parameter, target risk, and labeled training set as inputs.It outputs a selective classifier and a risk bound.
- Selection with Guaranteed Risk Control: SGR performs a binary search over confidence thresholds to find the selection rule that guarantees the required risk with sufficient confidence.The search evaluates empirical selective risk and updates the threshold using a numerical bound.
- Selection with Guaranteed Risk Control: The numerical bound used by SGR is the tightest possible in this setting, whereas analytic concentration bounds introduce slack.This supports using the numerical bound for calibrated risk control.
- Selection with Guaranteed Risk Control: The procedure applies uniform convergence across its iterations so the selected risk bound remains valid over the searched selection functions.The proof conditions on the number of accepted examples and completes with a union bound.
4 Confidence-Rate Functions for Neural Networks
The paper evaluates confidence-rate functions for neural networks by ranking examples and inducing threshold-based rejection. It considers softmax response and MC-dropout confidence, while emphasizing ranking quality rather than calibrated probabilities.
- Confidence-Rate Functions for Neural Networks: An effective confidence-rate function should rank examples so that higher confidence corresponds to lower expected loss.The induced risk-coverage curve is used to analyze the effectiveness of a confidence-rate function.
- Confidence-Rate Functions for Neural Networks: The paper notes that the considered confidence-rate functions are not ideal, but reports that they are empirically effective.Their effectiveness is assessed through risk-coverage behavior.
- Confidence-Rate Functions for Neural Networks: Softmax response ranks an input by the maximum softmax output across classes.The function is κ = max_j∈Y(f(x|j)).
- Confidence-Rate Functions for Neural Networks: Softmax responses need not be interpreted as absolute probabilities because selective rejection requires coherent relative ranking.The paper treats them as potentially useful confidence rates despite criticism of their probabilistic interpretation.
- Confidence-Rate Functions for Neural Networks: MC-dropout confidence is defined as minus the response variance obtained from repeated dropout-based forward passes.The variance is measured for the neuron corresponding to the most probable class.
5 Empirical Results
Experiments evaluate selective risk control across CIFAR-10, CIFAR-100, and ImageNet using SR and MC-dropout confidence functions with trained neural networks. The method produces tight risk bounds and substantially reduces error while retaining meaningful coverage, with SR especially effective on ImageNet.
- SR and MC-dropout produce nearly identical risk-coverage profiles on CIFAR-10 and CIFAR-100, while SR is significantly better on ImageNet.ImageNet comparisons cover both top-1 and top-5 tasks.
- At 60% ImageNet top-1 coverage, SR rejection has 10% error versus more than 20% for MC-dropout.
- The SGR routine constructs high-probability risk-controlled selective classifiers for CIFAR-10, CIFAR-100, and ImageNet.Experiments use trained VGG models, with RESNET-50 also evaluated on ImageNet.
- 5.1 Selective Guaranteed Risk for CIFAR-10: 1% error can be guaranteed while covering more than 78% of CIFAR-10.The reported risk bound remains close to the selected target risk, and test risk stays below the bound.
- 5.2 Selective Guaranteed Risk for CIFAR-100: At 67% CIFAR-100 coverage, the method surpasses the reported 18.85% best known result despite using an architecture that is not state of the art.The authors suggest a wide residual network could improve results further.
- 5.3 Selective Guaranteed Risk for ImageNet: On ImageNet, selective classifiers using VGG and RESNET architectures are effective and surpass the reported best known results with an appropriate coverage compromise.RESNET preserves its relative advantage over VGG across target-risk values.
6 Concluding Remarks
The paper presents selective classification with high-confidence risk control and validates it on challenging image datasets. It also identifies important boundaries and open directions, including jointly training classification and rejection functions and extending beyond 0/1 loss.
- The paper presents an algorithm for learning a selective classifier whose risk can be fully controlled and guaranteed with high confidence.
- Empirical studies on challenging image classification datasets show that guaranteed risk control is achievable.
- The methods are presented as immediately usable by deep learning practitioners for mission-critical classification tasks.
- The approach starts from a trained neural classifier rather than jointly training the classifier and selection function.
- Jointly training the classifier and selection function to optimize coverage for a given risk level remains a challenging open question.
- The paper studies selective classification only under 0/1 loss, leaving extensions to other loss functions, regression, and controlled false-positive and false-negative rates open.