PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks

Like Hui, Mikhail Belkin

arXiv:2006.07322v5cs.LGstat.ML

TL;DR

Cross-entropy is the near-universal default for neural classification despite limited comparative evidence supporting its superiority over square loss. Across modern architectures and benchmark tasks, this study finds square loss comparable or better overall, with the strongest results outside computer vision.

  • Problem

    Limited empirical evaluations and theoretical analyses justify cross-entropy’s predominance over square loss in modern neural classification.

  • Method

    The study compares square loss and cross-entropy across modern architectures and benchmark datasets in NLP, ASR, and computer vision, including compute-equalized training.

  • Results

    Square loss achieves accuracy better than or equal to cross-entropy in 22 of 28 learning tasks.

  • Takeaways & Limitations

    Loss selection should be an important model-selection consideration, with square loss evaluated on equal footing with cross-entropy for classification.

  • Takeaways & Limitations

    Domain-specific superiority remains unsettled because most vision results are similar and only one vision architecture shows a significant cross-entropy advantage.

Abstract

from arXiv · show

Modern neural architectures for classification tasks are trained using the cross-entropy loss, which is widely believed to be empirically superior to the square loss. In this work we provide evidence indicating that this belief may not be well-founded. We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizing computational resources. Indeed, we observe that the square loss produces better results in the dominant majority of NLP and ASR experiments. Cross-entropy appears to have a slight edge on computer vision tasks. We argue that there is little compelling empirical or theoretical evidence indicating a clear-cut advantage to the cross-entropy loss. Indeed, in our experiments, performance on nearly all non-vision tasks can be improved, sometimes significantly, by switching to the square loss. Furthermore, training with square loss appears to be less sensitive to the randomness in initialization. We posit that training using the square loss for classification needs to be a part of best practices of modern deep learning on equal footing with cross-entropy.

1 INTRODUCTION

Across 28 classification tasks, square-loss training matched or outperformed cross-entropy in 22 tasks, with lower initialization variance in most experiments. The authors therefore argue that loss choice deserves equal consideration in model selection, while noting domain-specific differences and limits on probability-calibration claims.

  • Motivation and contribution: Because cross-entropy is nearly universal in modern classification practice, the authors conclude that square loss should be considered alongside optimization methods and hyper-parameter tuning.Their experiments used literature-reported hyper-parameters except for learning rate, leaving potential room for further square-loss improvements.
  • Empirical results: Square loss achieved accuracy at least as good as cross-entropy in 22 of 28 evaluated learning tasks.Tasks were assessed using error rate or accuracy, with additional F1 and ImageNet Top-5 accuracy metrics.
  • Empirical results: Square-loss performance was less sensitive to initialization randomness, showing smaller variance in the majority of experiments.The reported averages over multiple random initializations were consistent with individual-initialization results.
  • Empirical results: Square-loss models remained competitive or better across nearly all settings, including after equalizing computation using cross-entropy’s validation-selected training duration.The supplied passage states that this performance advantage persists under matched computational resources.
  • Scope and limitations: Accurate classification, the study’s primary metric, may take precedence over probability estimation, so calibration is not treated as an unambiguous cross-entropy advantage.The paper explicitly places calibration analysis beyond its scope.
  • Domain applicability: Square loss generally performed better on NLP and ASR, whereas cross-entropy had a slight edge on computer vision.The authors caution against treating this pattern as definitive evidence that each loss is universally suited to a particular domain.

2 EXPERIMENTS

Experiments across NLP, ASR, and computer vision benchmark tasks compare square-loss and cross-entropy training under matched architectures and hyperparameters. Square loss generally matches or outperforms cross-entropy in NLP and ASR, while vision results are comparable overall with architecture-dependent differences.

  • Experimental setup: The study reimplements literature models with unchanged architectures and hyperparameters apart from learning-rate adjustment, then trains them with square loss under two protocols.One protocol matches cross-entropy training, while the other uses cross-entropy-selected epochs to equalize computational resources and favor cross-entropy.
  • NLP experiments: Square loss has better or equal accuracy in 12 of 14 NLP tasks and higher F1 in 5 of 6, including under equal computation cost.With the same epochs, square loss has equal or better accuracy in 11 of 14 tasks and higher F1 in 5 of 6 tasks.
  • ASR experiments: Square loss performs better or equally in 7 of 8 ASR tasks, with its largest reported advantage on the largest dataset, Librispeech.For Librispeech with VGG+BLSTM, square loss improves CER by 9.3% and WER by 7.5% relatively.
  • ASR experiments: For TIMIT and WSJ, both losses converge in the same number of epochs, whereas Librispeech requires more square-loss epochs but achieves better performance.The two square-loss protocols perform identically on TIMIT and WSJ and are comparable or better than cross-entropy.
  • Vision experiments: Vision performance is comparable between losses on MNIST, CIFAR-10, and ResNet-50 ImageNet, while cross-entropy performs better with EfficientNet.Across all three datasets, square loss converges as fast as cross-entropy; the two square-loss protocols produce the same accuracy except for ImageNet with ResNet-50.

3 PERFORMANCE ACROSS DIFFERENT INITIALIZATIONS

The analysis evaluates how initialization randomness affects square-loss and cross-entropy training by comparing per-seed accuracy or error differences. Square loss shows lower variance in 21 of 28 tasks, indicating reduced sensitivity to randomness in training.

  • 3 PERFORMANCE ACROSS DIFFERENT INITIALIZATIONS: The authors analyze results separately for each random-seed initialization to assess stability with respect to model-initialization randomness.Absolute per-run error and accuracy results, together with standard deviations, are provided in Appendix F.
  • 3 PERFORMANCE ACROSS DIFFERENT INITIALIZATIONS: Per random seed, the study computes square-loss versus cross-entropy differences in accuracy or error and reports one-standard-deviation error bars in Figure 1.For accuracy, the difference is square-loss accuracy minus cross-entropy accuracy; for error rate, it is cross-entropy error minus square-loss error.
  • 3 PERFORMANCE ACROSS DIFFERENT INITIALIZATIONS: 21 of 28 tasks show smaller standard deviation with square loss than cross-entropy, indicating lower sensitivity to randomness in the training process.Table 8 reports standard deviations of test accuracy/error for Librispeech and ImageNet tasks; smaller values are bolded.

4 OBSERVATIONS DURING TRAINING

Training with square loss converges as fast as cross-entropy on two-class NLP tasks, while on 1000-class Librispeech and ImageNet it may require more epochs but reaches similar or better accuracy.

  • Experimental scope: The optimization-speed comparison covers two-class NLP tasks and the relatively large class settings of Librispeech and ImageNet, both with 1000 classes.The experiments examine how convergence changes with the number of classes.
  • Convergence speed when class number is small: For two-class NLP classification, square-loss and cross-entropy training curves are quite similar, with square loss converging as fast as cross-entropy across BERT, LSTM+Attention, and LSTM+CNN on QNLI.The comparison is based on accuracy across epochs.

5 IMPLEMENTATION

The square-loss implementation removes the final softmax layer and applies loss rescaling for datasets with many output classes. Rescaling uses parameters k and M to emphasize the true class and improve optimization or performance.

  • No softmax: Removing the final softmax layer is necessary for square-loss training because it appears to impede optimization.
  • Loss rescaling mechanism: For datasets with at least 42 output classes, loss rescaling accelerates training, whereas datasets with fewer classes use no additional mechanism.
  • Loss rescaling mechanism: The rescaled square loss uses k to scale the true-label loss and M to scale the one-hot encoding; k = M = 1 recovers the standard square loss.
  • Loss rescaling mechanism: The parameter k emphasizes the correct class and can simplify the multiclass optimization landscape, while M further improves performance for very large class counts.

6 SUMMARY AND DISCUSSION

Across a broad empirical comparison, square loss outperformed cross-entropy on most datasets and architectures, sometimes substantially, while generally requiring only learning-rate adjustment. The authors therefore suggest including square-loss training for classification alongside cross-entropy in deep-learning best practices.

  • Square loss outperformed cross-entropy across most evaluated datasets and architectures, sometimes by a significant margin.The comparison covered classification tasks across multiple datasets and architectures.
  • The experiments spanned ten neural architectures and ten datasets across three major application domains.The authors acknowledge that these models and tasks represent only a sample of modern deep learning.
  • The results support placing square-loss training for classification on equal footing with cross-entropy in deep-learning best practices.For most datasets, no parameter changes beyond learning-rate adjustment were necessary.

APPENDICES … B.1 HYPER-PARAMETERS FOR NLP TASKS

The appendices document the datasets, standard preprocessing, implementation toolkits, and reproducibility settings used across NLP, ASR, and vision experiments. NLP-specific configurations are summarized through BERT and recurrent-model implementations plus key hyperparameters in Table 10.

  • A DATASETS AND TASKS: The benchmark suite includes NLP datasets such as MRPC and SST-2, with MRPC evaluated using accuracy and F1 score.MRPC contains sentence pairs annotated for semantic equivalence, while SST-2 concerns sentiment classification of movie-review sentences.
  • A DATASETS AND TASKS: The experiments also use TIMIT and WSJ for speech recognition, plus MNIST, CIFAR-10, and ImageNet for vision classification.The passages specify dataset sizes and task characteristics, including ImageNet’s 1,000 classes and approximately 1.28 million training images.
  • APPENDICES: The appendices provide dataset descriptions, implementation details, hyperparameter settings, and training epochs for the square-loss and cross-entropy experiments.These materials are intended to support reproducibility across the reported tasks.
  • B HYPER-PARAMETER SETTINGS: Data processing follows standard methods: Wang et al. (2018) for NLP, Watanabe et al. (2018) for ASR, and the corresponding papers’ defaults for vision.The appendix explicitly identifies these preprocessing conventions for reproducing the experiments.
  • B.1 HYPER-PARAMETERS FOR NLP TASKS: BERT uses PyTorch and the bert-base-cased model with the Hugging Face GLUE fine-tuning script, while LSTM+Attention and LSTM+CNN use Lan & Xu (2018)’s toolkit.The specific script path is given in the appendix, and the recurrent models are based on the released toolkit.
  • B.1 HYPER-PARAMETERS FOR NLP TASKS: Table 10 lists the key hyperparameters for NLP experiments, while the maximum sequence length equals the training set’s maximum sentence length.The appendix notes that not all hyperparameters are listed because there are many; training steps are also marked separately.

B.2 HYPER-PARAMETERS FOR ASR TASKS · B.3 HYPER-PARAMETERS FOR VISION TASKS

The ASR experiments follow ESPnet task configurations and documented task-specific exceptions, while the vision experiments use established open-source implementations and original training-step schedules. Hyper-parameter settings for both domains are tabulated in Tables 11 and 12.

  • B.2 HYPER-PARAMETERS FOR ASR TASKS: ASR implementations use ESPnet’s task-specific run.sh scripts and configuration files for TIMIT, WSJ, and LibriSpeech.The referenced scripts and configuration files provide the acoustic-model training hyper-parameters.
  • B.2 HYPER-PARAMETERS FOR ASR TASKS: Table 11 lists the hyper-parameters used across the ASR tasks.The table documents the task configurations underlying the experiments.
  • B.2 HYPER-PARAMETERS FOR ASR TASKS: For WSJ, the experiments use a specified language model, while selected settings set mtlalpha=0.3, batch-size=30, or elayers=4 for particular conditions.The elayers=4 setting corresponds to using 100 hours of training data.
  • B.3 HYPER-PARAMETERS FOR VISION TASKS: Vision models rely on established open-source implementations: TCNN, EfficientNet, Wide ResNet, and TensorFlow’s TPU ResNet-50 implementation.The cited implementations originate from Bai et al. (2018) and Tan & Le (2019), the WideResNet PyTorch repository, and TensorFlow’s official ResNet toolkit.
  • B.3 HYPER-PARAMETERS FOR VISION TASKS: Table 12 lists the hyper-parameters used for the vision tasks.The table provides the documented vision-task training settings.
  • B.3 HYPER-PARAMETERS FOR VISION TASKS: The vision setup follows Bai et al. (2018) for permuted MNIST and uses training-step counts from the original implementations.These notes specify how the permuted-MNIST task and training schedules were aligned with prior implementations.

C EXPERIMENTAL RESULTS ON VALIDATION AND TRAINING SETS

This section reports validation results for NLP and ASR tasks, along with training results for NLP, ASR, and vision tasks across Tables 13–19.

  • Validation results: Validation results for NLP tasks are reported in Tables 13 and 14, covering accuracy and F1 scores.
  • Validation results: Validation results for ASR tasks are reported in Table 15.
  • Training results: Training results are reported for NLP tasks in Tables 16 and 17, and for ASR and vision tasks in Tables 18 and 19, respectively.

D OUR RESULTS COMPARED WITH THE ORIGINAL WORK · E REGULARIZATION TERMS

The cross-entropy results are comparable to those originally reported, although several comparisons are unavailable or confounded by differing datasets, toolkits, and computational resources. The section also documents task-specific regularization terms, including zero where none was added.

  • D OUR RESULTS COMPARED WITH THE ORIGINAL WORK: Our cross-entropy results are comparable to the original results reported in the literature or toolkits.The comparison is presented in Table 20.
  • D OUR RESULTS COMPARED WITH THE ORIGINAL WORK: Several comparisons are unavailable because LSTM+Attention and LSTM+CNN implementations lacked reported MRPC and QNLI results, while some models are marked N/A.
  • D OUR RESULTS COMPARED WITH THE ORIGINAL WORK: QQP results are not comparable because the prior work used a different test set, whereas this study uses the standard test set from Wang et al. (2018).
  • D OUR RESULTS COMPARED WITH THE ORIGINAL WORK: NLP validation performance is reported using accuracy and F1 scores in Tables 13 and 14.
  • D OUR RESULTS COMPARED WITH THE ORIGINAL WORK: ASR validation performance is reported using error rate in Table 15.
  • D OUR RESULTS COMPARED WITH THE ORIGINAL WORK: The Librispeech VGG+BLSTM comparison is limited by using 100 hours of train-clean-100 data and a 1000-unigram dictionary instead of 1000 hours and at least 2000 unigrams.
  • E REGULARIZATION TERMS: Regularization terms for each task are listed in Table 21, with 0 indicating that no regularization term was added.WSJ regularization details are referenced in the ESPnet decoder implementation.

F VARIANCE OF ACCURACY AMONG DIFFERENT RANDOM SEEDS

Across five random seeds, square-loss training shows better accuracy/error rates and lower variance than cross-entropy on NLP and ASR tasks, indicating greater statistical robustness.

  • F Variance of Accuracy Among Different Random Seeds: The reported comparison concerns accuracy/error-rate variance across random seeds rather than a single deterministic run.Figure 3 summarizes results from five runs corresponding to five different random seeds.
  • F Variance of Accuracy Among Different Random Seeds: Square loss achieves better accuracy/error rates and smaller variance than cross-entropy in NLP and ASR, indicating statistically better training for those tasks.The comparison uses results across five different random seeds.
  • F Variance of Accuracy Among Different Random Seeds: Figure 3 compares five runs for square-loss training and cross-entropy, showing error bars and individual-run results for each random seed.Square-loss results appear on the left of each subfigure and cross-entropy results on the right.
Loading 2006.07322v5…