SelectiveNet: A Deep Neural Network with an Integrated Reject Option

Yonatan Geifman; Ran El-Yaniv

Source-linked AI summary

SelectiveNet: A Deep Neural Network with an Integrated Reject Option

Yonatan Geifman, Ran El-Yaniv

arXiv:1901.09192v4Jan 26, 2019cs.LGstat.ML

TL;DR

Selective prediction seeks models that can abstain on uncertain inputs, while existing deep-network rejection commonly thresholds confidence from a pretrained predictor. SelectiveNet jointly trains prediction and selection end-to-end for a target coverage, and the paper reports consistently improved risk–coverage performance across classification and regression datasets.

Problem
Selective prediction needs models that can abstain when they are insufficiently confident, but existing deep-network rejection mechanisms mainly rely on confidence thresholds over pretrained predictors.
Method
SelectiveNet is a three-headed deep network that jointly optimizes prediction, selection, and auxiliary prediction end-to-end for a user-specified coverage.
Results
The paper reports consistently improved risk–coverage trade-offs across several classification and regression datasets, including empirical advantages over MC-dropout and SR.
Takeaways & Limitations
SelectiveNet provides an integrated selective classification and regression model optimized for the covered domain, with reported state-of-the-art deep selective-classification results.
Takeaways & Limitations
The authors leave the effects of ensembling and the selection-head architecture for future study, while coverage inaccuracies can be amplified by high-capacity networks.

Abstract

from arXiv · show

We consider the problem of selective prediction (also known as reject option) in deep neural networks, and introduce SelectiveNet, a deep neural architecture with an integrated reject option. Existing rejection mechanisms are based mostly on a threshold over the prediction confidence of a pre-trained network. In contrast, SelectiveNet is trained to optimize both classification (or regression) and rejection simultaneously, end-to-end. The result is a deep neural network that is optimized over the covered domain. In our experiments, we show a consistently improved risk-coverage trade-off over several well-known classification and regression datasets, thus reaching new state-of-the-art results for deep selective classification.

1. Introduction

Selective prediction lets models abstain on uncertain cases, but prior deep-network approaches mainly add confidence-based rejection to pretrained predictors. SelectiveNet instead jointly learns prediction and selection for a specified coverage, with reported advantages across classification and regression.

Motivation: Selective prediction targets mission-critical settings by allowing a model to abstain when it is insufficiently confident.The goal is to learn models that know what they do not know.
Existing approaches: Prior reject-option methods use abstention costs or confidence proxies such as functional margins and nearest-neighbor distances.For deep networks, confidence-based rejection is commonly applied after training a predictor for full coverage.
SelectiveNet: SelectiveNet optimizes an end-to-end neural network with an integrated reject option for a user-specified target coverage.The approach is designed to optimize the required coverage slice rather than rely only on confidence extracted from a pretrained model.
Contributions: The paper introduces a selective loss based on interior-point optimization and a three-headed network for selective classification.The heads support prediction, selection, and auxiliary prediction.
Contributions: SelectiveNet provides a regression alternative to costly MC-dropout or ensemble techniques and shows empirical advantages over four classification and regression datasets.The reported comparisons include significant advantages over MC-dropout and SR.

2. Selective Prediction Problem Formulation

Selective prediction pairs a predictor with a selection function that determines which inputs receive predictions, trading selective risk against coverage. This work focuses on minimizing risk subject to a target coverage within a deep-network hypothesis class.

Problem formulation: A selective model consists of a prediction function f and a selection function g that either returns a prediction or abstains.With hard selection, g(x)=0 denotes abstention; soft selection can support probabilistic or thresholded decisions.
Problem formulation: Coverage is the probability mass of the non-rejected region, while selective risk measures loss over the selected predictions.Their empirical counterparts can be calculated from a labeled sample.
Problem formulation: The risk–coverage curve describes a selective model’s risk as a function of coverage.Risk and coverage can be traded off across the model’s operating profile.
Optimization objective: The paper minimizes selective risk subject to a specified coverage rate 0 < c ≤1 over a hypothesis class Θ.For the deep-network setting, Θ is the parameter set for the network architectures implementing f and g.
Optimization objective: A model controlling coverage and optimizing risk can be converted to one controlling risk and optimizing coverage using an earlier technique.The paper focuses on the coverage-constrained risk-minimization formulation.

3. Related Work

Reject-option research spans several hypothesis classes and learning algorithms, while neural-network methods commonly add confidence-based selection to a pretrained predictor. This paper follows joint learning and focuses on a single classifier, with ensembles noted as an extension.

Reject-option methods: Reject-option techniques have been studied extensively for SVMs, nearest neighbors, boosting, and other hypothesis classes.The literature includes both cost-based and confidence-oriented mechanisms.
Reject-option methods: A common construction adds a selection mechanism to a trained prediction model, whereas joint learning trains the predictor and selection function together.This paper follows the joint-learning approach.
Neural-network methods: For neural networks, a straightforward approach thresholds a confidence score from a model already trained to predict all points.For DNNs, prior work derives selective classifiers from confidence functions and can provide high-confidence selective-risk guarantees.
Neural-network methods: Ensemble-based confidence statistics are another family of rejection signals, but this work focuses on a single classifier.The authors state that extending the approach to ensembles is straightforward and likely to improve results.

4. SelectiveNet

SelectiveNet is a deep architecture that jointly learns prediction and selection, with an auxiliary prediction head supporting end-to-end optimization for a chosen coverage. Its training combines selective and auxiliary losses while enforcing coverage through an interior-point penalty.

4. SelectiveNet: SelectiveNet uses a deep neural architecture with prediction, selection, and auxiliary prediction heads.The main body can use architectures suited to the task; the auxiliary head is used only during training.
4. SelectiveNet: The selection head implements g(x), the prediction head implements f(x), and the auxiliary head h(x) learns a related prediction task.The auxiliary task enriches or enforces relevant feature construction in the shared main body.
4. SelectiveNet: The selective objective is converted into an unconstrained objective using an Interior Point Method penalty for the target coverage.The penalty uses c for target coverage, λ for constraint importance, and Ψ as a quadratic penalty function.
4. SelectiveNet: Training minimizes a convex combination of the selective loss L(f,g) and an auxiliary loss Lh.The auxiliary loss uses the same classification or regression task as the prediction head and ignores coverage considerations.
4. SelectiveNet: The auxiliary head exposes the shared main body to all training instances, preventing premature focus on only the selected fraction.Without h, the model may overfit to the wrong subset before accurate low-level features are constructed.

5. Coverage Accuracy

Deep selective models can miss their target test coverage because constrained training does not guarantee exact generalization. SelectiveNet improves coverage accuracy over SR and can be calibrated with an unlabeled validation set.

5. Coverage Accuracy: Deep constrained training can produce test coverage below or above the target, with high model capacity potentially amplifying these inaccuracies.The optimal selective model should satisfy the coverage constraint with equality.
5. Coverage Accuracy: At a 75% target coverage, realized coverage was 64.16% for SR and 80.17% for SelectiveNet.Both methods violated the target, despite using trained coverage rates from 0.7 through 0.9.
5. Coverage Accuracy: SelectiveNet had a 3.625% average coverage violation versus 11.98% for SR.The authors report significantly better coverage accuracy for SelectiveNet while still motivating post-training calibration.
5. Coverage Accuracy: Post-training calibration estimates a threshold τ from an independent unlabeled validation set and predicts only when g(x) ≥ τ.The threshold is set to the 100(1 − c) percentile of validation-set selection scores.
5. Coverage Accuracy: The Hoeffding bound provides a probabilistic interval [c − ϵ, c + ϵ] for calibrated coverage with confidence at least 1 − δ.The bound applies because the thresholded selection event is a Bernoulli variable.

6. Experimental Design and Details

The experiments evaluate SelectiveNet on three image datasets and one regression dataset, comparing it with SR and MC-dropout. CNN experiments use modified VGG-16, while regression uses a fully connected architecture.

6.1. Datasets: The study evaluates classification on SVHN, CIFAR-10, and Cats vs. Dogs, plus regression on Concrete Compressive Strength.The Concrete dataset contains 1,030 instances, eight numerical features, and one target value.
6.2. Baseline Methods: The proposed method is compared with Softmax Response and Monte Carlo-dropout baselines.SR uses maximum softmax activation, while MC-dropout estimates confidence from variance across multiple dropout passes.
6.3. Architectures: CNN experiments use a modified VGG-16 with one 512-neuron fully connected layer, batch normalization, and dropout.These changes adapt the original architecture to the small datasets and image sizes.
6.3. Architectures: For the CNN setup, the prediction and auxiliary heads are fully connected softmax layers, while the selection head outputs one sigmoid unit.The shared body is VGG-16, with α = 0.5 and λ = 32 used in all experiments.
6.3. Architectures: The regression model uses a 64-neuron ReLU hidden layer, linear prediction and auxiliary heads, and a 16-neuron ReLU selection hidden layer.The selection head ends with one sigmoid-activated neuron.

7. Experiments

Experiments across image classification and regression datasets show that SelectiveNet consistently improves selective risk over standard rejection baselines, with advantages across coverage rates.

Experimental setup: SelectiveNet is evaluated on three image-classification datasets and one regression dataset using architectures adapted to each task.The classification experiments use CNN-based main blocks, while regression uses a fully connected architecture.
Classification: 26.8% is SelectiveNet’s maximal relative advantage over MC-dropout on Cifar-10 at 0.75 coverage.Its relative advantage over SR ranges from 8.5% at 0.95 coverage, while SR and MC-dropout perform similarly.
Classification: 14.07% is SelectiveNet’s maximal relative advantage on SVHN at 0.8 coverage, while methods are statistically indistinguishable at 0.95 coverage.SR and MC-dropout have similar performance across SVHN coverage rates.
Classification: SelectiveNet shows a consistent and significant advantage over SR and MC-dropout on Cats vs. Dogs across the repeated coverage-rate experiments.The same architecture was adapted to the larger 64 × 64 input images for all contenders.
Classification: Risk-coverage curves for Cifar-10 and Cats vs. Dogs demonstrate SelectiveNet’s consistent advantage over SR.The curves compare SelectiveNet and SR across both datasets.
Regression: 26.81% to 30.48% is SelectiveNet’s relative advantage over MC-dropout on Concrete Compressive Strength below 0.9 coverage.SR is unavailable for regression because regression networks lack a softmax layer.

8. Empirical Observations

Additional analyses examine coverage calibration and learned representations. Calibration remains effective near the trained coverage, while SelectiveNet’s representation separates covered instances more than rejected ones.

Coverage Calibration: Coverage calibration evaluates five Cifar-10 SelectiveNet models across training and post-training coverage values using a selective-risk confusion matrix.Each matrix entry is the selective risk of a model trained at one coverage and calibrated to another; values are error percentages averaged over three runs.
Coverage Calibration: The bold-red diagonal indicates that calibration at a model’s trained coverage is nearly optimal among the evaluated SelectiveNet models.Red entries include values within one standard error of each column’s minimum.
Coverage Calibration: Calibration toward coverage close to the trained target causes no significant compromise, whereas off-diagonal settings often yield inferior selective risks.This supports coverage-specific optimization of selective risk.
Representation Structure: t-SNE visualizations compare the embedding representations learned by SelectiveNet and SR for Cifar-10 at 0.7 coverage.The visualization includes rejected and covered instances, with rejected points shown prominently and covered points faded.
Representation Structure: Rejected SelectiveNet instances are weakly separated and often form a central cluster, unlike the relatively well-separated SR clusters.The authors interpret this structure as potentially preserving representational capacity for separating non-rejected points and helping the selection function capture rejected points.

9. Concluding Remarks

The paper concludes that jointly trained prediction and selection produce accurate selective classification and a fast selective-regression approach. It identifies ensembling, selection-head capacity, and active learning as open directions.

Conclusions: Mutual training of prediction and selection focuses the network on instances that will not be rejected in production.This is contrasted with SR and MC-dropout, which add rejection mechanisms to standard full-coverage predictors.
Conclusions: SelectiveNet is reported to provide the most accurate classification for a given abstention rate and the best fast solution for selective regression.For regression, MC-dropout is described as the only other and inferior alternative.
Applications: The authors motivate SelectiveNet for fast-inference deep regression applications such as visual tracking or detection.This motivation is tied to the reported selective-regression result.
Future Work: Future work includes testing ensembles, optimizing the selection-function architecture for datasets and coverage rates, and investigating links with active learning.The paper states that the capacity choice for selection function g was not optimized.