Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification

Zhuoning Yuan; Yan Yan; Milan Sonka; Tianbao Yang

Source-linked AI summary

Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification

Zhuoning Yuan, Yan Yan, Milan Sonka, Tianbao Yang

arXiv:2012.03173v2Dec 6, 2020cs.LGcs.CVmath.OCstat.ML

TL;DR

Large-scale deep AUC maximization lacks sufficient evidence on difficult real-world tasks, especially medical image classification. The paper introduces a robust, scalable AUC margin loss and evaluates it across four medical datasets, achieving strong results including first place in the Stanford CheXpert competition.

Problem
Large-scale deep AUC maximization has been studied mainly through optimization algorithms, while its performance on difficult medical image classification tasks remains insufficiently studied.
Method
The paper proposes a min-max margin-based surrogate for AUC that is more robust to noisy and easy data while retaining scalable stochastic optimization.
Results
The method was evaluated on four medical image classification tasks and achieved 1st place in the Stanford CheXpert competition.
Takeaways & Limitations
The studies demonstrate the effectiveness of the new AUC margin loss and deep AUC maximization on medical image classification tasks.
Takeaways & Limitations
The AUC square loss has adverse effects on well-classified easy data and is sensitive to noisily labeled data.

Abstract

from arXiv · show

Deep AUC Maximization (DAM) is a new paradigm for learning a deep neural network by maximizing the AUC score of the model on a dataset. Most previous works of AUC maximization focus on the perspective of optimization by designing efficient stochastic algorithms, and studies on generalization performance of large-scale DAM on difficult tasks are missing. In this work, we aim to make DAM more practical for interesting real-world applications (e.g., medical image classification). First, we propose a new margin-based min-max surrogate loss function for the AUC score (named as AUC min-max-margin loss or simply AUC margin loss for short). It is more robust than the commonly used AUC square loss, while enjoying the same advantage in terms of large-scale stochastic optimization. Second, we conduct extensive empirical studies of our DAM method on four difficult medical image classification tasks, namely (i) classification of chest x-ray images for identifying many threatening diseases, (ii) classification of images of skin lesions for identifying melanoma, (iii) classification of mammogram for breast cancer screening, and (iv) classification of microscopic images for identifying tumor tissue. Our studies demonstrate that the proposed DAM method improves the performance of optimizing cross-entropy loss by a large margin, and also achieves better performance than optimizing the existing AUC square loss on these medical image classification tasks. Specifically, our DAM method has achieved the 1st place on Stanford CheXpert competition on Aug. 31, 2020. To the best of our knowledge, this is the first work that makes DAM succeed on large-scale medical image datasets. We also conduct extensive ablation studies to demonstrate the advantages of the new AUC margin loss over the AUC square loss on benchmark datasets. The proposed method is implemented in our open-sourced library LibAUC (www.libauc.org).

1. Introduction

The paper argues that large-scale deep AUC maximization can improve medical image classification, addressing scalability and robustness problems in existing AUC surrogate losses. It introduces a robust, scalable AUC margin loss and reports strong results across difficult medical tasks.

Motivation: Medical image classification has advanced through large-scale datasets, including chest X-ray, skin-lesion, and mammography datasets.These datasets support difficult disease-detection and screening tasks.
Motivation: AUC is attractive because it is the default evaluation metric for medical classification and handles imbalanced data by ranking positive examples above negative examples.The paper notes that medical datasets often contain far fewer malignant or positive cases than benign or negative cases.
Problem: Existing AUC methods face scalability challenges, while the AUC square loss can be adversely affected by easy data and sensitive to noisy labels.The paper identifies these two robustness issues as limitations of the commonly used scalable surrogate.
Contribution: The proposed AUC margin loss uses a min-max formulation to improve robustness to noisy and easy data while retaining the AUC square loss’s large-scale optimization advantage.The authors state that existing large-scale optimization algorithms can be adapted with one line of code.
Empirical study: The method was evaluated on four difficult medical image classification tasks and achieved 1st place in the Stanford CheXpert competition.The studies covered chest disease detection, skin-lesion classification, mammography screening, and tumor-tissue identification.
Contribution: The work is presented as the first comprehensive study of deep AUC maximization on large-scale medical image classification datasets.The paper also reports extensive comparisons with standard deep-learning losses and ablations against the AUC square loss.

2. Related Work

Related work studies pairwise surrogate losses for AUC maximization, but exact pairwise optimization can be computationally expensive and approximations may depend on dataset properties.

Optimizing Pairwise Surrogate loss: Pairwise AUC surrogates replace the ranking indicator with a convex loss and optimize relationships between positive and negative examples.The literature includes SVM-based, partial-pair, and online buffer-based approaches.
Optimizing Pairwise Surrogate loss: The SVM approach has O(n^2) complexity for n examples, motivating later methods that use partial pairs or historical data buffers.Partial-pair quality can depend on dataset properties, while small-regret online methods may require very large buffers.

3. Method

The method replaces standard classification and pairwise AUC objectives with a scalable min-max formulation and a robust margin-based surrogate. It preserves stochastic optimization while addressing the square loss’s sensitivity to easy and noisy data, then adapts existing algorithms for deep AUC maximization.

AUC is preferred for imbalanced medical classification because minimizing misclassification error may not maximize it.
Direct pairwise AUC optimization is not scalable because constructing all positive-negative pairs can require O(n^2) complexity.
The AUC square loss enables pair-free stochastic optimization through an equivalent min-max objective, but it adversely affects easy data and is sensitive to noisy labels.
The proposed AUC margin loss replaces the square penalty on the separation term with a squared hinge using a tunable margin m.
A nonnegative dual variable α makes the margin loss equivalent to a min-max problem, allowing stochastic algorithms developed for the square loss to be adapted.
When the mean positive score exceeds the mean negative score by margin m, the loss’s gradient omits α and only pushes scores toward their class means.
For mislabeled examples, reducing m reduces wrong-direction updates, alleviating sensitivity to noisy data.
The method uses PESG to update the model and auxiliary variables, with projection of α onto the nonnegative orthant.

4. Empirical Studies

The empirical studies evaluate robust deep AUC maximization across benchmark datasets and four difficult medical image classification tasks. AUC-M generally outperforms non-AUC losses and AUC-S, with strong CheXpert and Melanoma competition results.

Experimental setup: Experiments cover imbalanced Cat&Dog, CIFAR-10, CIFAR-100, and STL-10 benchmarks plus four medical image classification tasks.The medical tasks include chest disease detection, melanoma classification, breast cancer screening, and tumor-tissue identification.
Benchmark results: AUC-M and AUC-S outperform non-AUC losses in most benchmark cases, while AUC-M is usually better than AUC-S, especially at 1% imbalance.The study also includes ablations with easy and noisy data and examines the non-negative constraint on α.
CheXpert: CheXpert uses 224,316 training X-rays from 65,240 patients, with models pretrained using cross-entropy before DAM training.The competition test set contains images from 500 patients and is maintained privately by the organizer.
Melanoma: DAM predictions formed more separated positive and negative patterns than cross-entropy predictions on the Melanoma training data.The comparison uses the AUC-M+Meta model and a CE-trained model, as shown by the prediction histograms.
Melanoma: 0.9685/0.9438 public/private AUC ranked the Melanoma solution 42nd/33rd among 3,314 teams, and AUC-M outperformed AUC-S and standard losses.The reported values correspond to the public/private testing sets; the final ensemble later reached 0.9505 private AUC.
Other medical tasks: Additional experiments evaluate mammogram breast-cancer screening on DDSM+ and tumor-tissue identification on PathCamelyon using DenseNet121.DDSM+ contains 55,000 training and 13,900 testing mammographic images; PathCamelyon contains 294,912 color histopathology patches.

5. Ablation Studies

The ablations examine batch score normalization, robustness to noisy and easy data, and the non-negative constraint on α. Batch score normalization improves performance, while the α constraint prevents harmful values and supports faster convergence.

5.1 Batch Score Normalization (BSN): Applying batch score normalization improves performance across experiments with 1% and 10% imbalance ratios.The comparison uses DenseNet121 on four benchmark datasets, with and without batch score normalization.
5.2 Robustness to Noisy Data and Easy Data: The noisy- and easy-data ablations test whether AUC-M remains robust when additional mislabeled or easy samples are introduced.The noisy-data construction flips selected labels, while the easy-data experiments add easy positive samples.
5.3 Effect of Alpha Constraint: The non-negative α constraint yields better performance than leaving α unconstrained.The experiment uses C2-IB with 40% easy positive samples and 1% noisy samples, with margin m = 0.1.
5.3 Effect of Alpha Constraint: The α constraint prevents α from entering a bad region, producing faster convergence and a better result.The change in α is illustrated during training, including the first two epochs.

6. Conclusion

The paper concludes that its margin-based AUC surrogate addresses two major issues of AUC square loss and supports robust deep AUC maximization. Experiments across benchmark and real-world medical datasets demonstrate the effectiveness of the margin loss and the deep AUC maximization approach.

6. Conclusion: The proposed margin-based surrogate loss addresses two major issues associated with AUC square loss.The conclusion links the new loss to robust behavior on noisy and easy data.
6. Conclusion: The AUC square loss can be reformulated into expectations over positive and negative examples plus a cross term.The displayed expressions introduce the reformulation and explain the expectation terms using the definitions of a(w) and b(w).
6. Conclusion: The reformulation uses the definitions of a(w) and b(w), independence of positive and negative samples, and the convex conjugate of the square function.The text identifies these as the bases for equalities (e1), (e2), and (e3).

Appendix C. Proof of Theorem 1

The proof section establishes the relationship between the min-max formulation and the AUC margin loss, then analyzes how its constrained α affects updates on easy data. The case analysis distinguishes well-separated scores from insufficient-margin settings.

Appendix C. Proof of Theorem 1: The min-max problem is shown to be equivalent to the AUC margin loss in (6).The proof expresses the objective using positive and negative prediction deviations and a squared margin term.
Appendix C. Proof of Theorem 1: For easy positive data, the unconstrained square-loss update can reduce the positive prediction score and produce an undesirable update.The example changes the score from 1 to 0.9; the analogous negative example increases the negative score from −1 to −0.9.
Appendix C. Proof of Theorem 1: The AUC margin loss constrains α to be non-negative, giving α = max{0, 1 + b −a}, unlike AUC square loss.For AUC square loss, the unconstrained optimum is α = 1 + b −a.
Appendix C. Proof of Theorem 1: When the margin is already sufficient, α = 0 and the margin loss pushes positive and negative data toward their respective mean scores.The sufficient-margin case is defined by m + b ≤ a, and the positive example is pushed toward a.
Appendix C. Proof of Theorem 1: When the margin is insufficient, the constrained formulation increases the score of a misclassified positive example in the worked case.With m + b > a, the update changes the example's prediction from 0.25 to 0.2562.

E.3 Noisy Data for AUC Square Loss

The noisy-data analysis shows that AUC square loss can generate undesirable updates when the model is not good and noisy labels are present. AUC margin loss does not eliminate this issue completely, but its margin parameter can reduce the magnitude of the wrong-direction update.

E.3 Noisy Data for AUC Square Loss: For a noisy sample under an insufficient-margin model, the AUC square-loss gradient can produce an undesirable update.The analyzed case has m + b > a, and the gradient term is positive in the direction associated with the undesirable update.
E.3 Noisy Data for AUC Square Loss: The same undesirable-update phenomenon also occurs for the corresponding noisy negative-data case.The text states that the negative-data case can be developed similarly and yields the same outcome.
E.4 Noisy Data for AUC Margin Loss: AUC margin loss cannot completely resolve noisy-data problems, but reducing m reduces the magnitude of the wrong-direction update.In the analyzed case, the update magnitude is governed by the positive hyperparameter m.

Appendix F. An Example of Sensitivity of AUC

The appendix illustrates how accuracy and AUC can differ on imbalanced data, using examples where ranking quality and classification errors do not coincide.

Accuracy and AUC are illustrated as potentially differing on an imbalanced dataset with a positive ratio of 3/25.The stated accuracy threshold is 0.5.
Example 1 ranks all positive instances above negatives but still misclassifies two negative instances as positive.
Example 2 places one positive instance below seven negative instances and misclassifies one positive and one negative instance.
Example 3 places two positive instances below seven negative instances and misclassifies two positive instances as negative.

Appendix G. Descriptions of Imbalanced Datasets

The appendix describes the datasets used for imbalanced-data experiments and distinguishes original training-set size from manually constructed imbalanced variants.

Dataset descriptions report the number of samples in the original training set as the size of the training set.
Datasets with the suffix “-IB” are manually constructed imbalanced datasets created by randomly removing positive samples.

Appendix H. More Experiments on Benchmark Datasets

The appendix adds benchmark experiments with two network backbones, evaluates testing AUC over epochs, and examines margin selection within a two-stage AUC-maximization framework.

Testing AUC over epochs is evaluated on benchmark datasets using DenseNet121 and ResNet20.
At an imbalance ratio of 10%, benchmark testing AUC is reported for DenseNet121 and ResNet20.The appendix notes that a higher imbalance ratio, such as 10% versus 1%, makes classification easier.
Margin m trades off the margin benefit against robustness to noisy data, so tuning m is important for AUC-M.On benchmark datasets, the average values of m over different random trials are 0.7, 0.8, and 0.7.
The framework first learns representations with CrossEntropy pretraining and then finetunes the classifier decision boundary with AUC margin loss.

Appendix K. Network Architecture for Melanoma Classification

The melanoma architecture combines a CNN with a two-layer neural network and trains the components sequentially while freezing the CNN during DNN training.

The melanoma model uses a mixed architecture combining a CNN, identified as EfficientNet, with a 2-layer neural network.
Training first fits the CNN and then fits the DNN while freezing parameter updates for the CNN.The CNN and DNN use the same configurations according to the passage.