PaperlayerView exact revision on arXiv

Source-linked AI summary

PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions

Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Xiaojie Shi, Shuyang Cheng, Dragomir Anguelov

arXiv:2204.12511v2cs.CV

TL;DR

Classification losses have a large design space, and cross-entropy or focal loss may not be optimal across tasks and datasets. PolyLoss represents losses as weighted polynomial bases, subsuming both as special cases, and Poly-1 improves multiple tasks with one extra hyperparameter and one line of code.

  • Problem

    Designing a classification loss that works well across different tasks and datasets is challenging because the loss-function design space is large.

  • Method

    PolyLoss represents classification losses as weighted polynomial bases whose coefficients can be adjusted, with cross-entropy and focal loss as special cases.

  • Results

    Poly-1 improves models across multiple tasks and datasets by adjusting the leading polynomial coefficient with one extra hyperparameter.

  • Takeaways & Limitations

    Loss-function coefficients can be tailored to tasks and datasets, and simple modifications can improve established classification models.

  • Takeaways & Limitations

    The paper focuses on integer polynomial powers and leaves collectively tuning multiple coefficients as an important topic for future work.

Abstract

from arXiv · show

Cross-entropy loss and focal loss are the most common choices when training deep neural networks for classification problems. Generally speaking, however, a good loss function can take on much more flexible forms, and should be tailored for different tasks and datasets. Motivated by how functions can be approximated via Taylor expansion, we propose a simple framework, named PolyLoss, to view and design loss functions as a linear combination of polynomial functions. Our PolyLoss allows the importance of different polynomial bases to be easily adjusted depending on the targeting tasks and datasets, while naturally subsuming the aforementioned cross-entropy loss and focal loss as special cases. Extensive experimental results show that the optimal choice within the PolyLoss is indeed dependent on the task and dataset. Simply by introducing one extra hyperparameter and adding one line of code, our Poly-1 formulation outperforms the cross-entropy loss and focal loss on 2D image classification, instance segmentation, object detection, and 3D object detection tasks, sometimes by a large margin.

1 INTRODUCTION

PolyLoss frames classification-loss design as weighted polynomial expansion, unifying cross-entropy and focal loss while allowing task- and dataset-specific coefficient adjustments. The paper finds that changing only the leading coefficient can substantially improve performance with minimal complexity.

  • Framework: PolyLoss decomposes cross-entropy and focal loss into weighted polynomial bases for systematically designing classification losses.The framework represents each basis (1 − Pt)^j with a coefficient αj.
  • Framework: When αj = 1/j for all j, PolyLoss is equivalent to cross-entropy loss.
  • Design motivation: The optimal polynomial coefficients depend on the task and dataset, so a universal coefficient assignment may not be optimal.
  • Poly-1: Adjusting only the leading polynomial coefficient, called LPoly-1, achieves significant improvements over cross-entropy and focal loss.This strategy uses one extra hyperparameter and a minimal code change.
  • Contributions: PolyLoss explains cross-entropy and focal loss as special cases and motivates vertical coefficient adjustments beyond focal loss’s horizontal shifts.

2 RELATED WORK

The paper positions PolyLoss as a more general alternative for designing classification losses than commonly used cross-entropy and focal loss. It addresses the difficulty of searching large polynomial-loss spaces by simplifying the search to one hyperparameter.

  • Motivation: Cross-entropy is widely used across perception tasks, while focal loss is designed to address class imbalance in detection.
  • PolyLoss view: PolyLoss represents loss functions through polynomial coefficients, allowing vertical adjustments that focal loss’s horizontal coefficient shift does not provide.
  • Search complexity: Hundreds of polynomial terms may be needed for 1000-class classification tasks, creating a prohibitively large search space.
  • Search complexity: Poly-1 mitigates this search challenge through a simple grid search over one hyperparameter without relying on advanced black-box optimization.

3 POLYLOSS

PolyLoss expresses classification losses as polynomial expansions whose coefficients can be adjusted to change loss behavior. The framework connects cross-entropy and focal loss to these bases and motivates Poly-1 as a simple coefficient modification.

  • General form: In the expansion, Pt denotes the model’s prediction probability for the target ground-truth class.
  • Cross-entropy: Cross-entropy uses coefficients 1/j, causing its gradient expansion to contain polynomial terms (1 − Pt)^j.
  • Gradient behavior: The leading gradient term is constant, whereas higher-order gradient terms become strongly suppressed as Pt approaches 1.
  • Focal loss: Focal loss horizontally shifts polynomial powers by γ, replacing cross-entropy’s constant leading gradient with a term suppressed by (1 − Pt)^γ.
  • Poly-1: Poly-1 adjusts the leading polynomial coefficient, simplifying the broader strategies for vertically modifying coefficients to one hyperparameter.
  • Regression connection: The polynomial representation interprets cross-entropy and focal loss as weighted ensembles of prediction-label distances raised to different powers.
  • General form: PolyLoss is a monotone decreasing function on [0, 1] expressed as a weighted sum of (1 − Pt)^j terms.The paper focuses on integer powers j ∈ Z+ while noting that non-integer powers are possible.

4 UNDERSTANDING THE EFFECT OF POLYNOMIAL COEFFICIENTS

The paper finds that many higher-order terms may be needed to match cross-entropy, motivating Poly-N, which tunes only leading polynomial coefficients. Across classification, pretraining, detection, segmentation, and 3D detection, task-specific PolyLoss settings improve reported results.

  • 4.1 LDrop: Revisiting Dropping Higher-Order Polynomial Terms: More than 600 polynomial terms are needed to match cross-entropy accuracy on ImageNet-1K, and higher-order terms matter early when Pt is small.At Pt ∼0.001, the 500th term’s gradient coefficient is approximately 0.6.
  • 4.2 LPoly-N: Perturbing Leading Polynomial Coefficients: Tuning many coefficients creates a prohibitively large search space, while collectively tuning them does not outperform cross-entropy.The paper therefore perturbs only the leading coefficients while leaving the rest unchanged.
  • 4.2 LPoly-N: Perturbing Leading Polynomial Coefficients: Adjusting only the first polynomial coefficient improves ImageNet-1K accuracy, while tuning two coefficients adds only +0.1 beyond the first coefficient’s +0.4 gain.The experiments compare grid-search strategies through N = 3.
  • 5.1 LPoly-1 Improves 2D Image Classification on ImageNet: A simple grid search over the first coefficient improves EfficientNetV2 pretraining accuracy by around 1% across model sizes and improves ImageNet-1K finetuning from 86.8% to 87.2%.The pretraining gain nearly matches scaling the model from S to M or M to L.
  • 5.1 LPoly-1 Improves 2D Image Classification on ImageNet: The preferred coefficient direction depends on the task: positive ϵ1 improves imbalanced ImageNet-21K pretraining, whereas negative ϵ1 improves Mask R-CNN and 3D detection metrics.For Mask R-CNN, reducing the leading coefficient systematically improves box and mask AP and AR; Poly-1 with ϵ = −1 improves all reported 3D metrics for vehicle and pedestrian models.

6 CONCLUSION

PolyLoss unifies common classification losses through polynomial coefficients and proposes Poly-1 as a simple, effective modification across tasks and datasets.

  • PolyLoss views cross-entropy and focal loss as members of a unified polynomial-coefficient framework.
  • Focal loss is characterized as a horizontal coefficient shift relative to cross-entropy loss, motivating vertical coefficient modifications.
  • Poly-1 adjusts the leading polynomial coefficient using one extra hyperparameter and one line of code.
  • Poly-1 improves a variety of models across multiple tasks and datasets while retaining a simple formulation.
  • The results highlight limitations of common losses and motivate continued exploration beyond cross-entropy, focal loss, and Poly-1.

REPRODUCIBILITY STATEMENT

The reproducibility statement describes implementations of Poly-1 for focal and cross-entropy losses, including sigmoid, softmax, α-balance, and label-smoothing variants.

  • The experiments modify only loss functions while using public datasets and open-source code repositories.
  • The softmax cross-entropy implementation adds ε(1 − pt) to cross-entropy.
  • The cross-entropy implementation also supports α label smoothing.
  • The Poly-1 focal-loss implementation adds ε(1 − pt)^(γ + 1) to focal loss.
  • The α-balanced focal variant multiplies the added polynomial term by class-dependent weights.

SUPPLEMENTARY MATERIAL

Supplementary experiments examine training hyperparameters and show that tuning settings beyond the baseline can increase PolyLoss gains in ImageNet and COCO detection.

  • The ImageNet-1K weight-decay comparison uses 1e-4 as the default value.
  • RetinaNet experiments vary focal-loss γ and α, with (2.0, 0.25) as the default pair.
  • Tuning the leading polynomial term alone produces no COCO detection improvement under hyperparameters optimized for focal loss.

9 LDROP WITH MORE HYPERPARAMETER TUNING

Additional hyperparameter tuning shows that adjusting polynomial coefficients can substantially improve LDrop and that Poly-1 remains more effective with fewer hyperparameters.

  • For LDrop with N = 2, the study tunes the second polynomial coefficient, learning rate, and weight decay.
  • α = 8 raises ImageNet-1K ResNet-50 accuracy to 70.9, while the optimal learning rate remains 0.1.
  • The result indicates that dropping polynomial terms alone is insufficient and coefficient adjustment is critical.
  • Poly-1 is reported as more effective than prior LDrop methods while using only one hyperparameter.
  • Further weight-decay tuning improves Poly-1 accuracy while retaining fewer hyperparameters than LDrop*.

10 COLLECTIVELY TUNING MULTIPLE POLYNOMIAL COEFFICIENTS

The paper explores collectively tuning multiple PolyLoss polynomial coefficients by replacing cross-entropy’s 1/j coefficients with exponential decay. Poly-1 outperforms this exponential-decay alternative, while broader coefficient strategies remain open.

  • 10 COLLECTIVELY TUNING MULTIPLE POLYNOMIAL COEFFICIENTS: The study replaces cross-entropy coefficients 1/j with exponential decay to collectively tune multiple PolyLoss polynomial coefficients.The infinite sum is truncated at twice the decay factor N.
  • 10 COLLECTIVELY TUNING MULTIPLE POLYNOMIAL COEFFICIENTS: 72.3 is the best reported accuracy, achieved with N = 80 and learning rate = 1.6.The result comes from a 2D grid search over N and learning rate.
  • 10 COLLECTIVELY TUNING MULTIPLE POLYNOMIAL COEFFICIENTS: Poly-1 performs better than the exponential-decay formulation Lexp.The comparison motivates examining additional ways to collectively tune multiple coefficients.
  • 10 COLLECTIVELY TUNING MULTIPLE POLYNOMIAL COEFFICIENTS: The paper identifies understanding collective tuning of multiple coefficients as an important topic for future study.

11 COMPARING TO OTHER TRAINING TECHNIQUES

The paper places Poly-1 alongside established training techniques and reports comparable improvements to label smoothing and dropout on fully connected layers.

  • 11 COMPARING TO OTHER TRAINING TECHNIQUES: Poly-1 achieves improvements similar to commonly used training techniques such as label smoothing and dropout on FC.The comparison is reported for ResNet50 accuracy on ImageNet-1K.
  • 11 COMPARING TO OTHER TRAINING TECHNIQUES: The paper notes that combining independently novel training techniques can produce significant overall improvements despite individual gains often being below 1%.

12 REDISCOVERING FOCAL LOSS FROM POLYLOSS

This section uses PolyLoss to rediscover focal loss and studies how removing leading cross-entropy polynomial terms affects class-imbalanced detection. Dropping the first two terms improves RetinaNet performance and reduces majority-class overfitting.

  • 12 REDISCOVERING FOCAL LOSS FROM POLYLOSS: Focal loss was developed for RetinaNet to address strong class imbalance in single-stage object detection.
  • 12 REDISCOVERING FOCAL LOSS FROM POLYLOSS: The paper systematically investigates focal loss through the PolyLoss representation of cross-entropy and its gradient.
  • 12 REDISCOVERING FOCAL LOSS FROM POLYLOSS: Dropping the first two cross-entropy polynomial terms significantly improves RetinaNet detection AP and AR.The paper reports the best RetinaNet performance at N = 2.
  • 12 REDISCOVERING FOCAL LOSS FROM POLYLOSS: N = 2 produces RetinaNet performance similar to focal loss with γ = 2.The paper relates this to shifting polynomial coefficients to the right by two positions.
  • 12 REDISCOVERING FOCAL LOSS FROM POLYLOSS: Leading cross-entropy polynomials can overfit the majority class because its gradient contribution dominates under extreme class imbalance.For binary classification, the leading gradient is described as N_background − N_object.
  • 12 REDISCOVERING FOCAL LOSS FROM POLYLOSS: Dropping polynomials reduces overall prediction confidence primarily by reducing background confidence, while reducing majority-background overfitting yields more confident object predictions.
Loading 2204.12511v2…