PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Do Better Loss Functions Lead to Less Transferable Features?

Simon Kornblith, Ting Chen, Honglak Lee, Mohammad Norouzi

arXiv:2010.16402v2cs.CVcs.LG

TL;DR

Whether objectives that improve image-classification accuracy also learn useful downstream representations remains unclear. The paper compares training objectives through their learned representations and transferability. It finds that better ImageNet accuracy often accompanies worse fixed-feature transfer, while objective choice has little effect after full fine-tuning.

  • Problem

    Although alternative objectives can improve ImageNet accuracy, their effects on downstream representations and transfer are less established.

    Source: §1 · p. 1

  • Method

    The paper compares nine training objectives on ImageNet and studies their effects through network representations and transferability.

    Source: §1 · p. 1

  • Results

    Objectives that improve ImageNet accuracy over softmax cross-entropy produce fixed features that transfer substantially worse, with little objective effect after full fine-tuning.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Pretraining accuracy and fixed-feature quality for transfer trade off, so objectives should be evaluated in transfer settings as well as on the pretraining task.

    Source: §6 · p. 10

    The experiments use moderately sized datasets and models and are limited to supervised classification settings.

    Source: §5 · p. 10

Abstract

from arXiv · show

Previous work has proposed many new loss functions and regularizers that improve test accuracy on image classification tasks. However, it is not clear whether these loss functions learn better representations for downstream tasks. This paper studies how the choice of training objective affects the transferability of the hidden representations of convolutional neural networks trained on ImageNet. We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks, and the choice of loss has little effect when networks are fully fine-tuned on the new tasks. Using centered kernel alignment to measure similarity between hidden representations of networks, we find that differences among loss functions are apparent only in the last few layers of the network. We delve deeper into representations of the penultimate layer, finding that different objectives and hyperparameter combinations lead to dramatically different levels of class separation. Representations with higher class separation obtain higher accuracy on the original task, but their features are less useful for downstream tasks. Our results suggest there exists a trade-off between learning invariant features for the original task and features relevant for transfer tasks.

1 Introduction

Prior work often judged alternative training objectives by ImageNet accuracy, leaving their value for downstream transfer less studied. This paper compares nine objectives and finds that better ImageNet accuracy often comes with worse fixed-feature transfer.

1 Introduction

  • The study addresses a gap: most evaluations of alternatives to softmax cross-entropy focused on original-dataset accuracy, although transfer accuracy can matter more for many applications.A previous investigation suggested label smoothing and dropout can hurt transfer learning.Source: §1 · p. 1
  • Several objectives significantly improve ImageNet accuracy over softmax cross-entropy, yet their fixed feature extractors transfer substantially worse and provide no benefit after full fine-tuning.The comparison covers nine objectives and evaluates fixed features with linear and k-nearest neighbors classification.Source: §1 · pp. 1–2
  • Different objectives mainly change representations in the network’s last few layers, while earlier layers remain similar.Centered kernel alignment reveals this pattern, which helps explain the small effect of objective choice on fine-tuning transfer accuracy.Source: §1 · p. 2
  • Objectives that improve ImageNet accuracy also increase penultimate-layer class separation, apparently collapsing within-class variability and reducing downstream feature quality.Greater separation is associated with improved accuracy on the original task but poorer transfer features.Source: §1 · p. 2

2 Loss Functions and Output Layer Regularizers

The paper compares nine loss functions and output-layer regularizers, treating objectives as combinations of terms that encourage target alignment and constrain logit scale. The methods include changes to targets, activations, logits, and final-layer weights.

2 Loss Functions and Output Layer Regularizers

  • The study compares nine objectives; each encourages logits to align with targets while contracting or normalizing logits to prevent scale alone from increasing that alignment.The authors do not draw a firm distinction between loss functions and regularizers because many regularizers add contractive terms.Source: §2 · p. 2
  • Softmax cross-entropy combines target-logit alignment with a contractive LogSumExp term, while label smoothing [76] mixes one-hot targets with a uniform distribution.The smoothing loss is scaled by 1/(1 − α) to maintain the positive-logit gradient scale; prior work found improved calibration and simplex-vertex class centroids.Source: §2 · p. 2
  • Penultimate-layer dropout randomly masks activations and yields both implicit noise-based and explicit parameter regularization, while stronger final-layer L2 regularization can improve performance.The dropout activations are scaled by the keep probability; the final layer is commonly regularized like the rest of the network.Source: §2 · pp. 2–3
  • An explicit logit L2 penalty constrains logit magnitude, while additional L2 regularization with unregularized batch-normalization scales can alter optimization implicitly.The logit penalty was reported to yield accuracy improvements comparable to dropout.Source: §2 · p. 3
  • Logit normalization uses temperature τ to control magnitude and output entropy, while cosine softmax normalizes features and class weights before computing cosine-similarity logits.Even without an explicit margin, changing τ substantially affects observed class separation.Source: §2 · p. 3
  • The study also evaluates sigmoid cross-entropy as a one-versus-rest-style classifier and squared error, where κ weights the correct class and M sets its target magnitude.The sigmoid loss replaces LogSumExp with summed softplus-transformed logits and initializes biases to −log(K); greater correct-class weighting slightly improves ImageNet accuracy [36].Source: §2 · p. 3

3 Results

Across nine objectives, many improve ImageNet accuracy over softmax cross-entropy, but fixed features transfer worse and full fine-tuning largely removes the objective’s effect. The paper relates these outcomes to late-layer representation differences and class separation.

3.1 Better objectives improve accuracy, but do not transfer better

  • 0.9% higher top-1 accuracy than softmax cross-entropy is achieved by sigmoid cross-entropy and cosine softmax, while fixed softmax features transfer best on most of eight datasets.All investigated objectives except squared error significantly improve ImageNet accuracy when properly tuned; alternative objectives transfer worse in linear and kNN evaluation.Source: §3.1 · p. 4, Table 1 · p. 4
  • Only a 0.2% average improvement over softmax cross-entropy is obtained by the best objective after fine-tuning, and the difference is not statistically significant.This is smaller than the 0.9% ImageNet top-1 accuracy difference.Source: §3.1 · p. 5, Table 2 · p. 5

3.2 The choice of objective primarily affects hidden representations close to the output

  • Representations are similar through most network layers, but loss-dependent differences emerge in the final third, starting around block 13.The CKA comparisons cover ResNet-50 representations on ImageNet validation data and 18 networks; corresponding layers are more similar than non-corresponding layers.Source: §3.2 · p. 5, Figure 2 · p. 6
  • Penultimate-layer sparsity varies substantially by objective: softmax has 92.8% non-zero activations, while logit normalization, cosine softmax, and squared error each yield fewer than 25%.Sparsity is broadly similar across objectives in the first three ResNet stages but differs in the final stage and penultimate average-pooling layer.Source: §3.2 · pp. 6–7, Figure 3 · p. 6

3.3 Regularization and alternative losses increase class separation

  • All investigated alternative losses and regularizers increase penultimate-layer class separation over vanilla softmax, particularly when regularization affects final layers.Data augmentation improves accuracy without substantially changing class separation; class separation peaks early with softmax but saturates or rises with other objectives.Source: §3.3 · pp. 7–8, Table 3 · p. 7

3.4 Greater class separation is associated with less transferable features

  • Class separation negatively correlates with mean linear transfer accuracy across losses (Spearman’s ρ = −0.93, p = 0.002), and higher cosine-softmax temperatures continue to increase separation while transfer declines.At τ = 0.01, ImageNet accuracy is 2.7% below the best temperature, yet transfer features are best for nearly all datasets; for τ ≥ 0.04, ImageNet accuracy saturates while separation rises and transfer falls.Source: §3.4 · p. 8, Figure 5 · p. 8, Table 4 · p. 8
  • Greater class separation benefits relearning the original ImageNet classes, while overlap with pretraining classes helps explain Oxford-IIIT Pets’ exception to the usual transfer pattern.Oxford-IIIT Pets shares 25 of its 37 cat and dog breeds with ImageNet, and mapped ImageNet top-1 predictions reach 71.2% accuracy on Pets but much lower accuracy on other datasets.Source: §3.4 · pp. 8–9, Figure 6 · p. 9

4 Related work

Related work studies why objectives affect classification, how objectives shape transfer, and how class separation relates to performance. This paper extends prior transfer findings while noting differing results across downstream settings.

Understanding training objectives

  • Theoretical work analyzes convergence of logistic objectives to minimum-norm solutions on linearly separable data, while empirical studies examine label smoothing and dropout mechanisms.Prior explanations for label smoothing include label-noise mitigation, entropy regularization, and accelerated convergence.Source: §4 · p. 9

Training objectives for transfer

  • Kornblith et al. found that label smoothing and dropout hurt transfer, while this study reports the pattern across objectives and connects it to hidden representations.Self-supervised studies also report that parameters maximizing contrastive-task accuracy need not yield the best linear-classification features [14].Source: §4 · p. 9
  • Prior few-shot work finds softmax features can match more sophisticated embeddings, and proposed “supervision collapse” as related to class separation [23].Studies of final-layer L2 regularization report few-shot transfer advantages on some, but not all, downstream tasks; shared classes are suggested as a possible explanation, requiring further study [91] [1].Source: §4 · pp. 9–10

Class separation

  • Class-separation findings vary by setting: greater separation can improve deep metric-learning recall and some few-shot results, while a negative cosine-softmax margin can lower separation and improve accuracy.The few-shot literature reports differing conclusions across mini-ImageNet, CIFAR-FS, and other setups [26] [44].Source: §4 · p. 10

5 Limitations

The evidence is bounded to moderately sized datasets and models in supervised classification. The authors also note that class separation and pretraining accuracy do not have a simple monotonic relationship.

5 Limitations

  • The experiments cover moderately sized datasets and models, and the conclusions are limited to supervised classification.The authors suggest larger datasets with richer label spaces could mitigate the trade-off and make greater class separation helpful when labels overlap downstream classes.Source: §5 · p. 10
  • Squared error achieves the highest class separation but does not significantly outperform softmax, so class separation alone does not specify a recipe for higher pretraining accuracy.The relationship between class separation and pretraining accuracy is nonmonotonic.Source: §5 · p. 10

6 Conclusion

Objectives that improve pretraining performance learn more invariant, class-separated representations, but those features are less useful when transferred in fixed form. The authors call for evaluating objectives directly in transfer settings.

6 Conclusion

  • Objectives that improve pretraining performance learn more invariant representations with greater class separation, but those properties hinder fixed-feature transfer.The paper describes a trade-off between features suited to the original task and those useful for transfer.Source: §6 · p. 10
  • Because objectives mainly change later layers, which change substantially during fine-tuning, higher pretraining accuracy does not yield downstream fine-tuning gains.Source: §6 · p. 10
  • The authors recommend that future loss-function research explicitly target and evaluate transfer performance rather than optimizing only pretraining accuracy.Source: §6 · p. 10

A.1 Training and tuning neural networks

The appendices describe ImageNet model training and hyperparameter selection, evaluation of fixed features with logistic regression, and downstream fine-tuning.

A.1 Training and tuning neural networks

  • ResNet-50 ImageNet models used SGD with Nesterov momentum, large-batch training, exponential moving average evaluation, and validation-based hyperparameter and stopping-epoch selection.The training runs used 72 main-text configurations plus approximately as many tuning runs; early stopping affected maximal performance but not the transferability and class-separation conclusions.Source: §A.1 · p. 17, Table A.1 · p. 17

A.2 Training and tuning multinomial logistic regression classifiers

  • Fixed-feature transfer used L-BFGS logistic regression, with L2 regularization selected on a held-out validation set before test evaluation.The datasets and their evaluation metrics are listed in Table A.2.Source: §A.2 · p. 17, Table A.2 · p. 18

A.3 Fine-tuning

  • Fine-tuning used 20,000 SGD steps with cosine annealing; learning rate and weight decay were tuned on validation data across three pretrained-model initializations.The best configuration was retrained on combined training and validation data before test evaluation.Source: §A.3 · p. 18

B Confirmation of main findings with Inception v3

Inception v3 experiments reproduce the paper’s findings on class separation, transferability, and late-layer representation differences. Supplementary learning-curve analyses compare validation and training accuracy across objectives.

B Confirmation of main findings with Inception v3

  • Inception v3 confirms that softmax features are more transferable, lower class separation is associated with greater transferability, and loss choice affects later-layer representations.Table B.1 reports ImageNet validation accuracy and fixed-feature transfer accuracy; Figure B.1 shows CKA between representations on ImageNet validation data.Source: B Confirmation of main findings with… · p. 18, Table B.1 · p. 18, Figure B.1 · p. 19

C.1 Training accuracy and learning curves

  • Regularizers and modified losses reduce ImageNet training accuracy overall, while label smoothing ties softmax on top-1 and is slightly better on top-5.These training-accuracy comparisons are reported in Table C.1.Source: Table C.1 · p. 20
  • Logit normalization appears to provide higher ImageNet validation accuracy than vanilla softmax across the training run.Figure C.1 shows validation-accuracy trajectories, with curves ending at the epoch of highest holdout accuracy.Source: Figure C.1 · p. 20
  • Regularized models achieve higher validation accuracy than softmax at a given training accuracy.Figure C.3 compares validation and training accuracy curves, terminating each curve at the training accuracy associated with its highest holdout accuracy.Source: Figure C.3 · p. 21

C.2 Robustness and calibration

Alternative losses generally improve out-of-distribution accuracy, but calibration effects are mixed. Across prediction similarity, regularizer combinations, and transfer-training dynamics, the objectives show distinct effects beyond ImageNet accuracy.

C.2 Robustness and calibration

  • Alternative losses outperform softmax cross-entropy on almost all tested out-of-distribution sets, with logit normalization and cosine softmax typically slightly better, while calibration effects are mixed.Label smoothing reduces expected calibration error, cosine softmax has lower negative log likelihood, and logit penalty has the worst calibration; calibration does not track accuracy clearly.Source: §C.2 · p. 22, Table C.2 · p. 22, Table C.3 · p. 22

C.3 Similarity of model predictions

  • Models cluster by training objective in prediction similarity, although same-objective models still disagree and mixed-objective ensembles offer little accuracy benefit over same-objective ensembles.The most similar pair disagrees on 13.9% of validation examples; even softmax and squared-error ensembles disagree on 11.5%.Source: §C.3 · pp. 22–23, Figure C.4 · p. 23, Figure C.6 · p. 24

C.4 Combining regularizers does not improve accuracy

  • Combining losses or final-layer regularizers does not improve accuracy, whereas AutoAugment adds gains through a complementary mechanism; cosine softmax with AutoAugment and Mixup reaches 79.1% top-1 and 94.5% top-5 accuracy.The paper reports these as the best 224 × 224 single-crop accuracies for an unmodified ResNet-50 trained from scratch at the time of first posting.Source: §C.4 · p. 24, Table C.4 · p. 25, Table C.5 · p. 25

D.1 Scatterplots of ImageNet vs. transfer accuracy by dataset

  • Higher ImageNet accuracy is not associated with higher linear transfer accuracy across the individual training runs shown.The figure presents results separately by dataset.Source: Figure D.1 · p. 26

D.2 Training dynamics of transfer accuracy

  • Softmax cross-entropy has the highest linear transfer accuracy on most datasets over training, tying sigmoid cross-entropy on SUN397 while other losses do better on Pets.Squared-error transfer results are off the plot scale except on Pets.Source: Figure D.2 · p. 26

D.3 Results on Chexpert

On CheXpert, fixed ImageNet features from softmax cross-entropy perform at least as well as those from alternative losses in the tested settings. The accompanying analyses relate transfer changes to representation shifts, class-separation measures, and data augmentation.

D.3 Results on Chexpert

  • On a 22,431-image CheXpert split, softmax cross-entropy significantly outperforms all other losses on four of five pathologies and ties for best AUC on the fifth.The official validation set has only 234 images and high variance; softmax has the best numerical results on all but one pathology there, though many losses are statistically tied.Source: §D.3 · p. 27, Table D.1 · p. 27

D.4 CKA between models before and after transfer

  • Transfer changes representations more in later network layers than earlier ones, and the extent of those changes varies substantially by dataset.The comparison uses CKA between softmax-trained ResNet-50 models before and after transfer.Source: Figure D.3 · p. 28

E.1 Relation of class separation index to variance ratios

  • The class-separation index generalizes η2 in ANOVA or R2 for categorical predictors; for normalized embeddings, it measures separation through within-class and between-class variance relationships.The appendix relates the index to variance ratios and cosine-distance measures.Source: §E.1 · pp. 28–29

E.2 Other class separation indexes and measurements

  • Class-centroid singular-value spectra distinguish softmax from sigmoid cross-entropy, whereas the spectra of penultimate activations and final-layer weights do not.Other class-mean direction measurements also distinguish softmax from alternative losses.Source: §E.2 · p. 30, Figure E.2 · p. 31

E.3 Augmentation can improve accuracy without increasing class separation

  • AutoAugment increases ImageNet accuracy without increasing class separation, supporting the conclusion that augmentation and final-layer modifications act through complementary mechanisms.The table compares top-1 validation accuracy with class separation measured on the training set.Source: §E.3 · p. 32, Table E.3 · p. 32

E.4 Training dynamics of class separation

Differences in class separation across objectives emerge early in training and their relative ordering changes little, rather than being explained by differing training durations.

E.4 Training dynamics of class separation

  • Loss-specific class-separation differences emerge early and retain their relative ordering; softmax separation peaks at epoch 32, while most losses peak on validation before optimal accuracy.Alternative-objective models reach their maximum training-set class separation at the last checkpoint.Source: §E.4 · p. 32, Figure E.4 · p. 32

F Class overlap between ImageNet and transfer datasets

Semantic class overlap with ImageNet is ambiguous and does not capture shifts in class-conditional image distributions. The paper therefore estimates overlap using an ImageNet classifier and a mapped-label accuracy measure.

F Class overlap between ImageNet and transfer datasets

  • Semantic overlap counts downstream classes whose names directly or nearly match ImageNet classes, or are superclasses of ImageNet classes, but this criterion misses distribution shifts such as CIFAR-100's lower image resolution.The authors note that 61 of CIFAR-100's 100 classes are ImageNet superclasses even though an ImageNet classifier performs poorly on CIFAR images.Source: F Class overlap between ImageNet and… · p. 33
  • The assignment-accuracy measure maps ImageNet classes to downstream labels using an ImageNet classifier; most datasets score below half their linear transfer accuracy, with CIFAR-10 and Pets as exceptions.Pets has higher assignment accuracy than CIFAR-10 despite comparable linear transfer accuracy, suggesting the greatest class overlap with ImageNet.Source: F Class overlap between ImageNet and… · p. 33, Table F.1 · p. 33
Loading 2010.16402v2…