PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification

Zhuoning Yuan, Yan Yan, Milan Sonka, Tianbao Yang

arXiv:2012.03173v2cs.LGcs.CVmath.OCstat.ML

TL;DR

Large-scale deep AUC maximization had not been comprehensively studied on medical image classification tasks. This paper proposes a robust, scalable margin-based AUC loss and evaluates deep AUC maximization across four medical-image tasks, reporting stronger performance than standard deep-learning methods and first place on CheXpert.

  • Problem

    Large-scale deep AUC maximization had not been comprehensively studied on medical image classification datasets, where AUC is a favorable evaluation metric.

    Source: §1 · pp. 2, 4

  • Method

    The paper proposes a margin-based min-max surrogate loss for AUC that is designed to be robust to noisy and easy data while retaining large-scale optimization scalability.

    Source: §1 · p. 3

  • Results

    Across four medical-image tasks, DAM outperformed standard deep-learning methods; on CheXpert, it ranked first with a 2%+ improvement over Stanford’s baseline on private testing data.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The study reports successful use of deep AUC maximization on large-scale medical image classification tasks.

    Source: §1 · p. 4, §6 · p. 16

    The analysis notes that AUC square-loss updates can move truly positive or negative examples in the wrong direction when their scores meet specified conditions.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Deep AUC Maximization (DAM) is a new paradigm for learning a deep neural network by maximizing the AUC score of the model on a dataset. Most previous works of AUC maximization focus on the perspective of optimization by designing efficient stochastic algorithms, and studies on generalization performance of large-scale DAM on difficult tasks are missing. In this work, we aim to make DAM more practical for interesting real-world applications (e.g., medical image classification). First, we propose a new margin-based min-max surrogate loss function for the AUC score (named as AUC min-max-margin loss or simply AUC margin loss for short). It is more robust than the commonly used AUC square loss, while enjoying the same advantage in terms of large-scale stochastic optimization. Second, we conduct extensive empirical studies of our DAM method on four difficult medical image classification tasks, namely (i) classification of chest x-ray images for identifying many threatening diseases, (ii) classification of images of skin lesions for identifying melanoma, (iii) classification of mammogram for breast cancer screening, and (iv) classification of microscopic images for identifying tumor tissue. Our studies demonstrate that the proposed DAM method improves the performance of optimizing cross-entropy loss by a large margin, and also achieves better performance than optimizing the existing AUC square loss on these medical image classification tasks. Specifically, our DAM method has achieved the 1st place on Stanford CheXpert competition on Aug. 31, 2020. To the best of our knowledge, this is the first work that makes DAM succeed on large-scale medical image datasets. We also conduct extensive ablation studies to demonstrate the advantages of the new AUC margin loss over the AUC square loss on benchmark datasets. The proposed method is implemented in our open-sourced library LibAUC (www.libauc.org).

1. Introduction

The introduction motivates deep AUC maximization for imbalanced medical classification and presents a scalable, robust margin-based surrogate loss. It reports strong results across medical imaging tasks and benchmark comparisons.

Introduction

  • AUC is the default evaluation metric in medical classification and is suited to imbalanced data because it ranks positive predictions above negative ones.The introduction notes that AUC can be more sensitive than misclassification error to changes in a few prediction scores.Source: §1 · pp. 2–3
  • Generic pairwise AUC surrogates are not scalable for large datasets, while the scalable AUC square loss can be adversely affected by easy data and noisy labels.The paper proposes the margin loss to address these limitations.Source: §1 · p. 3
  • The AUC min-max-margin loss is designed to resist noisy data, avoid adverse effects from easy data, and retain the scalability of AUC square loss.Its min-max form allows existing large-scale optimization algorithms for AUC square loss to be used with a one-line code change.Source: §1 · p. 3
  • 1st place among 150+ submissions in CheXpert came with a 2%+ improvement over the Stanford baseline on private test data.The Kaggle ensemble ranked 33 of 3,314 teams, while the best single model exceeded the winning team’s best model by more than 2%.Source: §1 · p. 3
  • The empirical study covers chest X-rays, skin lesions, mammograms, and microscopic tumor-tissue images, alongside benchmark comparisons with AUC square, cross-entropy, and focal losses.The introduction also situates this work amid prior large-scale medical image classification studies, including chest X-ray, skin-lesion, and mammography datasets [22] [39].Source: §1 · pp. 1–4

2. Related Work

Prior AUC work moved from costly or approximate pairwise objectives toward min-max formulations of pairwise square loss that support efficient stochastic optimization. Existing deep AUC methods scale further but retain square loss, whose deficiencies this work addresses.

Deep AUC Maximization (DAM)

  • Existing deep AUC methods include buffered online learning for small mammogram datasets and scalable min-max algorithms, but neglect deficiencies of the square loss, [28] [15].The buffered method requires a large sample buffer, limiting it to datasets with a few thousand medical images; federated DAM algorithms were also proposed [14].Source: §2 · p. 4

Optimizing Pairwise Square loss

  • Pairwise square loss supports efficient large-scale stochastic optimization through an equivalent min-max formulation that avoids explicitly constructing positive-negative pairs [12].Later work sought to improve convergence rates for these min-max optimization problems.Source: §2 · p. 4

Optimizing Pairwise Surrogate loss

  • Pairwise AUC surrogates can be costly or approximate: SVM optimization has O(n^2) complexity, while partial-pair and buffered methods have dataset-quality or memory limitations,.The partial-pair approximation may perform poorly with large intra-variance, and buffered online AUC methods need large buffers to achieve small regret.Source: §2 · p. 4

3. Method

The method replaces misclassification-focused training with scalable AUC maximization, proposing a margin-based loss to address weaknesses of the AUC square loss. For difficult medical image tasks, it also uses cross-entropy pretraining before AUC optimization.

3. Method

  • In imbalanced medical datasets, minimizing misclassification error may not maximize AUC, which is a more favorable classifier-evaluation metric for medical applications.This motivates the paper’s focus on AUC rather than standard individual-example loss minimization.Source: §3 · p. 5

3.1 Background on Scalable AUC Maximization

  • AUC measures the probability that a positive sample ranks above a negative one; pairwise training can cost O(n^2), whereas a square-loss min-max formulation enables stochastic updates without constructing pairs.The supplied passages state that such stochastic algorithms scale to hundreds of thousands of examples.Source: §3.1 · pp. 5–6

3.2 Drawbacks of the AUC Square Loss

  • The AUC square loss can worsen classification on easy examples and, with noisy labels, update parameters against truly positive and negative examples.For easy pairs with a score gap above 1, its update pushes the positive score down and the negative score up; noisy-label updates can likewise move in the wrong direction when the scores are sufficiently extreme.Source: §3.2 · pp. 6–7

3.3 The Proposed AUC Margin Loss

  • The AUC margin loss replaces the square loss’s mean-separation term with a squared hinge using a tunable margin m and α ≥ 0.The remaining terms minimize the variance of prediction scores on positive and negative examples; the resulting objective is equivalent to a min-max optimization problem.Source: §3.3 · p. 7
  • With a good model, the nonnegative α constraint limits updates to drawing sampled predictions toward their class means; with a poor model, updates likely push positive scores up and negative scores down.For noisy labels, tuning m smaller can reduce movement in the wrong direction, alleviating sensitivity to noisy data.Source: §3.3 · p. 8

3.4 DAM with the AUC Margin Loss

  • PESG updates the primal variables and α for the margin-loss min-max objective, projecting α onto the nonnegative orthant after updates.The method uses algorithmic regularization γ and periodically updated reference solutions; λ is the weight-decay parameter.Source: §3.4 · pp. 8–9
  • For difficult medical tasks, the two-stage framework first minimizes cross-entropy, then maximizes an AUC surrogate while learning all layers with the final classifier randomly initialized.The authors report that direct AUC optimization from scratch handles simple datasets but has difficulties on complex tasks, motivating the framework.Source: §3.4 · p. 9

4. Empirical Studies

Across benchmark and medical-image tasks, the paper reports that AUC-M generally improves on non-AUC losses and often outperforms AUC-S. The studies cover imbalanced benchmark datasets and four medical classification tasks, including CheXpert and melanoma competitions.

4.1 Performance on Benchmark datasets

  • Benchmark experiments used 1% and 10% imbalance settings, compared AUC-M and AUC-S with CE and focal-loss baselines, and applied batch score normalization before both AUC losses.Results were averaged over five random training sets; the 10% imbalance results are reported in Appendix H.Source: §4.1 · p. 10
  • For DenseNet121/ResNet20 at imratio=1%, AUC-M and AUC-S perform much better than non-AUC-based losses in most cases; AUC-M performs better than AUC-S in most cases, especially in the extremely imbalanced setting with imratio=1%.Source: §4.1 · p. 10

4.2 Medical Image Classification Tasks

  • The medical-image studies span chest X-rays for disease detection, skin lesions for melanoma, mammograms for breast-cancer screening, and microscopic images for tumor-tissue identification.The overview identifies four tasks; the paper reports task and data summaries in Table 2.Source: §4.2 · p. 10, Table 2 · p. 11

4.2.1 CheXpert Competition

  • 0.9305 mean testing AUC earned the DAM AUC-M CheXpert ensemble 1st place; it surpassed CE-based solutions and averaged better than 2.8 of 3 radiologists.The reported AUC-square DAM result was 0.922. The submission was made on Aug. 31, 2020.Source: §4.2.1 · p. 12, Table 3 · p. 12

4.2.2 Melanoma Classification

  • On melanoma EfficientNet-B5 tests, DAM improved over CE and focal loss, while AUC-M outperformed AUC-S; AUC-M predictions also showed more separated positive and negative patterns than CE predictions.The comparison reports public and private testing performance with and without test-time augmentation.Source: §4.2.2 · p. 13, Figure 2 · p. 13, Table 4 · p. 14
  • 0.9685 public and 0.9438 private AUC scores ranked the melanoma ensemble 42nd and 33rd, respectively, among 3,314 teams.For a single model under similar configurations, DAM scored 0.9423 AUC versus 0.9167 for the winning team's single model; a later ensemble combining AUC-M and CE models reached 0.9505 private AUC.Source: §4.2.2 · pp. 13–14, Table 4 · p. 14

4.3 Other Two Medical Classification Tasks

  • AUC-M performed consistently better than baseline methods on DDSM+ mammograms and an imbalanced PathCamelyon tumor-tissue task.DDSM+ training data had 13% imbalance and test data 4%; the constructed PathCamelyon task used 1% imbalance.Source: §4.3 · p. 14, Table 5 · p. 15

5. Ablation Studies

Ablations show that batch score normalization improves performance, AUC-M outperforms AUC-S under noisy and easy-data conditions, and constraining α to be nonnegative yields faster convergence and better results.

5.1 Batch Score Normalization (BSN)

  • Applying batch score normalization improves performance across four benchmark datasets and two imbalance ratios.The experiments use DenseNet121 and compare settings with and without batch score normalization at 1% and 10% imbalance ratios.Source: §5.1 · p. 15, Figure 3 · p. 15

5.2 AUC-Margin Loss

  • On C2-IB ablations, AUC-M outperforms AUC-S by a large margin under noisy ratios of 1% and 5% and with top 10% or 20% easy-sample additions.Source: §5.2 · p. 15, Figure 5 · p. 16
  • Enforcing α ≥0 improves testing AUC and prevents α from entering a bad region, yielding faster convergence and a better result.This comparison uses C2-IB with 1% imbalance, 40% easy positive samples, 1% noisy samples, and margin m = 0.1.Source: §5.2 · pp. 15–16, Figure 4 · p. 16

6. Conclusion · Appendix A. Optimal Values of a, b, α in AUC Square Loss · Appendix B. Reformulation of AUC Square Loss

The conclusion reports that the proposed AUC margin loss is robust to noisy and easy data and that deep AUC maximization succeeds on medical image classification tasks; the appendices derive square-loss auxiliary optima and reformulate the loss.

6. Conclusion

  • The proposed margin-based AUC surrogate addresses two major issues of square loss and is demonstrated to be robust to noisy and easy data.The paper evaluates its methods on four benchmark and four real-world medical datasets.Source: §6 · p. 16
  • Results across four benchmark and four real-world medical datasets demonstrate the new margin loss’s effectiveness and deep AUC maximization’s success on medical image classification.Source: §6 · p. 16

Appendix A. Optimal Values of a, b, α in AUC Square Loss

  • For fixed w, the optimal a and b are the respective positive- and negative-class mean scores; minimizing the a-subproblem yields the variance of the positive-class scores.Source: Appendix A · p. 21
  • The appendix obtains the optimal α by setting the gradient of its subproblem to zero.The displayed derivation expresses the subproblem in terms of p, 1 − p, and class-conditional score expectations.Source: Appendix A · p. 21

Appendix B. Reformulation of AUC Square Loss

  • The square-loss reformulation expands the pairwise expression into positive- and negative-class squared deviations and a cross term, then uses sample independence to remove the cross term.The resulting expression contains conditional squared deviations for the positive and negative classes.Source: Appendix B · pp. 21–22
  • The reformulation defines a(w) and b(w) as class-conditional mean scores, treats them as constants in expectation, and uses the square function’s convex conjugate in the final equality.Source: Appendix B · p. 22

Appendix C. Proof of Theorem 1 … E.2 Easy Data for AUC Margin Loss

The appendices prove the equivalence between the min-max formulation and the AUC margin loss, then use one-dimensional examples to contrast square-loss and margin-loss updates on easy data. The examples show that the margin loss’s nonnegative constraint on α changes the update behavior across margin conditions.

Appendix C. Proof of Theorem 1

  • The proof establishes that minimizing the min-max problem is equivalent to the AUC margin loss, whose final expression explicitly shows a squared hinge loss.The displayed objective includes positive- and negative-class squared deviations and a squared margin term.Source: Appendix C · p. 22

Appendix E. A 1-Dim Example of Easy/Noisy Data for AUC Square and Margin Loss

  • The key difference between the square and margin losses is that α is unconstrained for the square loss but constrained to be nonnegative for the margin loss.At optimality, α = 1 + b - a for square loss and α = max{0, 1 + b - a} for margin loss.Source: Appendix E · p. 23

E.1 Easy Data for AUC Square Loss

  • With w = 1, a = 0.5, and b = -0.5, square-loss updates lower the positive score from 1 to 0.9 and raise the negative score from -1 to -0.9, both undesirable.The examples use easy positive and negative samples x1 = 1 and x2 = -1.Source: §E.1 · pp. 23–24

E.2 Easy Data for AUC Margin Loss

  • When m + b ≤ a, α = 0, yet the margin loss still penalizes well-classified samples by pushing positive and negative predictions toward their respective means.The example states that when the model is good enough, the loss pushes positive data toward a and negative data toward b.Source: §E.2 · p. 24
  • For m + b > a, the margin-loss update raises the score of a positive sample below the margin from 0.25 to 0.2562.The example uses w = 1, m = 1, a = 0, b = -0.5, and x = 0.25; the passage says negative-data examples follow similarly.Source: §E.2 · p. 24

E.3 Noisy Data for AUC Square Loss · E.4 Noisy Data for AUC Margin Loss

For a not-good model, noisy data can produce an undesirable update with AUC square loss; AUC margin loss retains the wrong direction but can reduce its magnitude by lowering the margin hyperparameter.

E.3 Noisy Data for AUC Square Loss

  • Assuming w = 1, when m + b > a (e.g., a = 0.25, b = −0.5), C = 0.25 × 1 + 1 − 0.25 = 1 in the positive direction, so ∇wF aligns with ∇whw(x1) and gives an undesirable update.Source: §E.3 · p. 24
  • Because only y1 is revealed, term C determines ∇wF; the true positive label means hw(x1) can be large.Source: §E.3 · p. 24
  • Here C = hw(x1) − b + α = 1, so ∇wF follows ∇whw(x1) and produces the undesirable update.Source: §E.3 · p. 24
  • Noisy negative data in the same AUC square-loss case also gives an undesirable update.Source: §E.3 · p. 25

E.4 Noisy Data for AUC Margin Loss

  • For AUC margin loss, the not-good-model case uses α = m + b − a, with a = 0.25 and b = −0.5.Source: §E.4 · p. 25
  • For noisy positive data, C = hw(x1) − b + α = m, so lowering the positive hyperparameter m reduces the wrong-direction update but does not eliminate the issue.Unlike the square loss's constant m = 1, m in the margin loss is a hyperparameter.Source: §E.4 · p. 25
  • The same noisy-data situation occurs for negative examples with a not-so-good model under AUC margin loss.Source: §E.4 · p. 25

Appendix F. An Example of Sensitivity of AUC

On an imbalanced dataset, AUC drops dramatically as positive instances rank lower, while accuracy remains unchanged.

Appendix F. An Example of Sensitivity of AUC

  • AUC drops dramatically as positive-instance ranks fall, while accuracy remains unchanged.The examples use 25 samples with a positive ratio of 3/25 and an accuracy threshold of 0.5; they contrast correctly ranked positives with cases where one or two positives rank below seven negatives.Source: Table 6 · p. 25, Appendix F · p. 25

Appendix G. Descriptions of Imbalanced Datasets

Appendix G describes the datasets and explains how imbalanced variants were constructed.

Appendix G. Descriptions of Imbalanced Datasets

  • Datasets labeled “-IB” were manually constructed by randomly removing positive samples.The table’s “size of training set” refers to the original training set’s number of samples.Source: Table 7 · p. 26

Appendix H. More Experiments on Benchmark Datasets · Appendix I. The Choice of Margin m for AUC-M Loss · Appendix J. A Two-stage Training Framework for DAM

The appendices describe additional benchmark AUC evaluations, explain why the AUC-M margin requires tuning and report selected margins, and outline a two-stage DAM training framework.

Appendix H. More Experiments on Benchmark Datasets

  • Figures 6 and 7 plot testing AUC versus epochs on benchmark datasets using DenseNet121 and ResNet20, respectively.Source: Figure 6 · p. 26, Figure 7 · p. 26
  • Table 8 reports benchmark testing AUC for DenseNet121 and ResNet20 at imratio=10%, which is less imbalanced and easier than 1%.Source: Table 8 · p. 27

Appendix I. The Choice of Margin m for AUC-M Loss

  • A smaller margin can reduce sensitivity to noisy data when a weak model’s stochastic gradient points in the wrong direction, so tuning m trades margin benefit against robustness.Source: Appendix I · p. 27
  • On benchmark datasets, average m values across random trials are 0.7 for C2, 0.8 for C10, 0.7 for S10, and 0.5 for C100.Source: Appendix I · p. 27
  • For medical datasets, the best m is 0.8 on Melanoma, 0.8 averaged over five CheXpert classes, 0.5 on DDSM, and 0.7 on PatchCamelyon.Source: Appendix I · p. 27
  • The AUC-M loss ablation studies use m = 0.3.Source: Appendix I · p. 27

Appendix J. A Two-stage Training Framework for DAM

  • The two-stage framework first learns representations with standard cross-entropy, then fine-tunes the classifier’s decision boundary with AUC margin loss.Source: Figure 8 · p. 27

Appendix K. Network Architecture for Melanoma Classification

The melanoma classifier combines an EfficientNet CNN with a 2-layer neural network to use image and patient contextual data. Training proceeds in two stages, with CNN parameter updates frozen during DNN training.

Appendix K. Network Architecture for Melanoma Classification

  • The melanoma model combines an EfficientNet CNN and a 2-layer neural network to predict from image and patient contextual data.The architecture is shown in Figure 9.Source: Figure 9 · p. 28
  • Training first fits the CNN, then trains the DNN with the same configurations while freezing CNN parameter updates.The training configurations are described in the main section.Source: Appendix K · p. 28

Related papers

Loading 2012.03173v2…