PaperlayerView exact revision on arXiv

Source-linked AI summary

Tilted Empirical Risk Minimization

Tian Li, Ahmad Beirami, Maziar Sanjabi, Virginia Smith

arXiv:2007.01162v2cs.LGcs.ITstat.ML

TL;DR

ERM’s focus on average loss can be inadequate for outliers, imbalance, fairness, and generalization. TERM adds a tunable tilt to reweight losses, and the paper finds it competitive across these applications, including combined noisy-outlier and fairness settings. Its robustness evidence is scoped to random additive or unstructured noise rather than adversarial corruption.

  • Problem

    ERM optimizes average loss, which can be inadequate for noisy data, class imbalance, subgroup fairness, and out-of-sample performance.

  • Method

    TERM extends ERM with a real-valued tilt that adjusts individual-loss influence and supports hierarchical multi-objective tilting.

  • Results

    TERM consistently outperforms ERM and is competitive with state-of-the-art problem-specific methods across applications, including simultaneous handling of noisy outliers and class imbalance.

  • Takeaways & Limitations

    A single tilted-objective framework can address robustness, class imbalance, fairness, and generalization, including combined objectives that existing approaches often treat separately.

  • Takeaways & Limitations

    The robustness evaluation focuses on random additive or unstructured noise, not adversarial noise, and structured large noise can cause TERM to overfit noisy samples.

Abstract

from arXiv · show

Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly. While many methods aim to address these problems individually, in this work, we explore them through a unified framework -- tilted empirical risk minimization (TERM). In particular, we show that it is possible to flexibly tune the impact of individual losses through a straightforward extension to ERM using a hyperparameter called the tilt. We provide several interpretations of the resulting framework: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to a superquantile method. We develop batch and stochastic first-order optimization methods for solving TERM, and show that the problem can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. TERM is not only competitive with existing solutions tailored to these individual problems, but can also enable entirely new applications, such as simultaneously addressing outliers and promoting fairness.

1 INTRODUCTION

TERM extends ERM with a tilt that adjusts how individual losses influence the objective, addressing settings where average loss is inadequate. The framework connects this simple modification to robustness, fairness, class imbalance, and other applications.

  • 1 INTRODUCTION: ERM can be inadequate when average performance poorly represents goals involving noisy data, class imbalance, subgroup fairness, or out-of-sample performance.
  • 1 INTRODUCTION: TERM introduces a real-valued tilt that flexibly adjusts individual-loss influence, with t=0 recovering average-loss ERM and extreme tilts recovering max- and min-loss objectives.Positive tilts emphasize larger losses, while negative tilts suppress them.
  • 1 INTRODUCTION: Negative tilts can suppress outliers in point estimation and regression, whereas positive tilts can emphasize them or approach worst-case fitting for fairness-oriented objectives.
  • 1 INTRODUCTION: In logistic regression, changing the tilt alters the classifier’s slope and emphasizes misclassified data, unlike changing the decision threshold, which shifts the intercept.
  • 1 INTRODUCTION: TERM combines theoretical analysis, efficient optimization, and case studies, including compound settings with noisy samples and imbalanced classes.

2 TERM: PROPERTIES & INTERPRETATIONS

TERM provides a smooth, tilt-controlled family of objectives spanning minimum, average, and maximum loss, with interpretations based on reweighting, fairness-related tradeoffs, variance reduction, and superquantiles.

  • 2 TERM: PROPERTIES & INTERPRETATIONS: TERM is smooth for finite tilts, convex for positive tilts under strong convexity, and recovers min-, average-, and max-loss solutions across its tilt range.
  • 2 TERM: PROPERTIES & INTERPRETATIONS: The tilted gradient exponentially reweights individual-loss gradients: positive t magnifies high-loss outliers, negative t suppresses them, and t=0 gives uniform ERM weights.
  • 2 TERM: PROPERTIES & INTERPRETATIONS: For positive t, increasing the tilt decreases max-loss while increasing average loss, creating a smooth tradeoff that can promote uniformity or fairness.
  • 2 TERM: PROPERTIES & INTERPRETATIONS: As t increases, empirical loss variance is non-increasing while average loss is non-decreasing, enabling a potential bias-variance tradeoff for generalization.
  • 2 TERM: PROPERTIES & INTERPRETATIONS: TERM smoothly approximates superquantile objectives, whose nonsmooth and generally non-convex k-loss formulations are difficult to optimize at scale.

3 TERM EXTENDED: HIERARCHICAL MULTI-OBJECTIVE TILTING

Hierarchical multi-objective TERM extends tilting from individual samples to groups, enabling simultaneous control of objectives such as robustness to noisy data and fairness across subgroups.

  • 3 TERM EXTENDED: HIERARCHICAL MULTI-OBJECTIVE TILTING: Multi-objective TERM supports simultaneous robustness to noisy data and fair performance across groups through sample- and group-level tilting parameters.
  • 3 TERM EXTENDED: HIERARCHICAL MULTI-OBJECTIVE TILTING: The extension recovers sample-level TERM when τ=t and group-level TERM as τ approaches zero, while retaining TERM’s relevant properties.
  • 3 TERM EXTENDED: HIERARCHICAL MULTI-OBJECTIVE TILTING: Hierarchical tilting uses a weighted sum of gradients and significantly outperforms baselines for simultaneous class imbalance and noisy-outlier handling.

4 SOLVING TERM

TERM is solved with batch and stochastic first-order gradient methods that compute tilted gradients, including for the hierarchical multi-objective formulation. The objective’s smoothness and convexity depend on the tilt and affect convergence.

  • 4 SOLVING TERM: Batch TERM and its stochastic variant apply first-order gradient-based optimization to compute tilted gradients and solve sample- or multi-objective TERM.
  • 4 SOLVING TERM: Batch TERM reduces to the sample-level tilted objective when t=τ, while the hierarchical algorithm computes losses and gradients within each group.
  • 4 SOLVING TERM: The smoothness and convexity properties governing gradient-method convergence vary with the tilt.

5 TERM IN PRACTICE: USE CASES

TERM is evaluated across robustness, fairness, generalization, and compound objectives, where tilted losses are competitive with or outperform specialized baselines. The experiments also show that TERM can address noisy samples and class imbalance jointly through hierarchical tilting.

  • 5.1 MITIGATING NOISY OUTLIERS: TERM is competitive with robust regression baselines at moderate noise and achieves better robustness under moderate-to-extreme noise.The evaluation reports RMSE on clean test data across noise levels, with Genie ERM serving only as an impractical performance limit.
  • 5.1 MITIGATING NOISY OUTLIERS: TERM performs competitively with 20% label noise and outperforms all compared baselines in high-noise CIFAR-10 classification.The comparison excludes methods requiring clean validation data from the main baseline set, while TERM matches MentorNet-DD in an appendix experiment.
  • 5.2 FAIRNESS AND GENERALIZATION: TERM with t=100 is competitive with state-of-the-art methods for classification with imbalanced classes.The class-imbalance experiment compares TERM against several sample-reweighting baselines on imbalanced MNIST datasets.
  • 5.2 FAIRNESS AND GENERALIZATION: Positive tilts support fairness and generalization, while TERM-PCA trades performance between high- and low-education groups as the tilt varies.The framework uses positive tilting to promote fairness and variance reduction, and the PCA experiment varies the aggregate losses of the two groups.
  • 5.3 SOLVING COMPOUND ISSUES: HIERARCHICAL MULTI-OBJECTIVE TILTING: Hierarchical TERM is far superior to all baselines when noisy samples and class imbalance occur simultaneously.The method applies tilting at multiple levels, including sample and class levels, and also considers class and annotator levels for noisy annotator settings.

6 RELATED WORK

TERM relates to alternative ERM strategies by changing how losses are aggregated, modified, or weighted. Unlike approaches focused on one mechanism, it provides a unified tilted objective spanning these concerns.

  • Alternate aggregation schemes: TERM changes loss aggregation through exponential smoothing, connecting average-loss ERM with min-max and superquantile-style objectives.Positive tilting emphasizes large losses, while related superquantile methods directly target upper-tail performance.
  • Alternate loss functions: Unlike methods that modify losses with L1, Huber, M-estimator, or other task-specific forms, TERM uses a general tilted aggregation mechanism.Prior exponentially tilted robust regression was limited to squared loss and negative tilt.
  • Sample re-weighting schemes: TERM also relates to sample re-weighting because its objective changes the influence assigned to individual examples according to their losses.This connects TERM to methods emphasizing hard examples or rebalancing classes through altered sampling weights.

7 CONCLUSION

The paper concludes that TERM is a flexible ERM extension that performs competitively across robustness, imbalance, fairness, generalization, and compound settings. Its theory and experiments support broad utility, while future work remains on generalization bounds and stochastic-solver convergence.

  • Conclusion: TERM consistently outperforms ERM and remains competitive with state-of-the-art problem-specific methods across diverse applications.The reported applications include noisy outliers, class imbalance, fairness, generalization, and simultaneous imbalance with noisy outliers.
  • Conclusion: The paper develops theoretical analyses, efficient TERM solvers, and empirical studies spanning generalized linear models and broader application settings.The appendices provide proofs, solver details, complete empirical results, and broader-impact discussion.
  • Conclusion: Future work includes deriving generalization bounds as a function of tilt and theoretical convergence guarantees for the proposed stochastic solvers.These are explicitly identified as open directions in the conclusion.
  • Assumptions: The theoretical results rely on smoothness, strong convexity, generalized-linear-model, and strict-saddle assumptions, with the strongest condition tied to exponential-family negative log-likelihoods.The paper notes that L2 regression and logistic classification satisfy the generalized-linear-model assumption.

B BASIC PROPERTIES OF THE TERM OBJECTIVE

The TERM objective has tractable smoothness and convexity properties under stated assumptions, recovers familiar ERM and extreme-loss objectives, and extends naturally to hierarchical multi-objective tilting.

  • Basic properties: At t=0 TERM recovers ERM, while t approaching positive or negative infinity recovers max-loss or min-loss objectives.These limits establish TERM as a continuous family spanning average, worst-case, and best-case loss criteria.
  • Convexity and smoothness: For positive tilt, the TERM objective is strongly convex under the strong-convexity assumption, supporting favorable optimization structure.The proof combines the tilted Hessian decomposition with positive-definite curvature from the individual losses.
  • Convexity and smoothness: For every finite tilt, TERM is smooth under smooth individual losses, with local smoothness characterized near the final solution.The analysis distinguishes smoothness behavior for negative and positive tilts.
  • Optimization: Under the strict-saddle property, gradient descent converges to a local minimum, with its rate governed by the objective’s smoothness parameter.For positive tilt, strong convexity also readily verifies the strict-saddle assumption.
  • Hierarchical tilting: Hierarchical multi-objective TERM recovers sample-level TERM as the special case t=τ, linking group-level and sample-level tilting.The hierarchy is used to clarify experiments that tilt at class or annotator level.

E GENERAL PROPERTIES OF TERM SOLUTIONS FOR GLMS

For generalized linear models, TERM solutions vary systematically with tilt: increasing tilt shifts emphasis toward larger losses, while the framework connects this behavior to loss quantiles, tail probabilities, and variance reduction.

  • Solution behavior: TERM approximately minimizes progressively higher loss quantiles as tilt increases, moving from the smallest toward the largest losses.The tilt parameter determines the approximate loss-rank target across the full range from negative to positive infinity.
  • Variance and weighting: TERM has variance-reduction properties, and its loss-vector alignment with the all-ones vector increases with tilt.The paper separately characterizes variance reduction, cosine similarity, and increasingly uniform gradient weights.
  • Objective properties: The tilted objective and its optimum are increasing functions of tilt, while the gradient weights become more uniform as tilt increases.These results formalize how changing tilt alters the objective landscape and sample influence.
  • Exponential-tilting interpretation: Under an i.i.d. intuition, TERM approximates a cumulant generating function and therefore relates optimization to minimizing a loss tail probability or complementary CDF.The distributional assumption is used for intuition rather than for the paper’s proofs.

G TERM AS AN APPROXIMATE SUPERQUANTILE METHOD

TERM provides a computationally feasible approximation to superquantile optimization, with tilted solutions serving as approximate solutions under the paper’s stated bounds.

  • Superquantile connection: TERM offers a computationally feasible approximation to a non-smooth, non-convex superquantile problem that is challenging to solve globally.The paper frames TERM as a practical alternative for minimizing the fraction of losses exceeding a threshold.
  • Theoretical guarantee: Theorem 10 bounds the quantile of losses exceeding a threshold by a smooth function of the TERM objective.
  • Approximation: TERM’s tilted solutions are approximate solutions to the corresponding superquantile optimization problem.
  • Empirical validation: Figure 6 shows Q0(a) approximately equals Q2(a), indicating that solving Q3(a), which equals Q2(a), tightly approximates the globally optimal Q0(a) solution.
  • Optimization: The stochastic TERM solver has roughly the same time and space complexity as minibatch SGD, while its solution can approach the full-batch optimum.

H.1 CONVERGENCE WITH t

TERM’s tilt controls optimization smoothness and loss aggregation: moderate tilts remain efficiently solvable, while increasing positive tilt can slow convergence as the objective becomes less smooth.

  • Convergence: TERM is as efficient as traditional ERM for small-to-moderate tilts, contrasting with the slower optimization associated with the min-max limit.
  • Convergence: As t approaches positive infinity, the objective becomes less smooth near the final solution and convergence slows, whereas negative tilts converge quickly because solutions remain locally smooth.
  • Negative tilts: For sufficiently large negative t, TERM becomes non-convex and may have multiple local minima, so the solver decreases t smoothly from zero to follow a continuous solution path.
  • Loss tradeoffs: Increasing t from negative to positive infinity reduces loss variance while trading average loss against max- or min-loss objectives.

I.2 COMPLETE CASE STUDIES

Across case studies, TERM addresses noisy data, robustness, fairness, and generalization, with reported advantages over or competitiveness with specialized baselines in several settings.

  • Robust regression: TERM with t = -2 effectively removes outliers under random noise, including synthetic regression settings with varying noise levels.
  • Robust classification: TERM matches MentorNet-DD while avoiding that baseline’s need for 5,000 clean validation images.
  • Robust classification: TERM with t = -2 converges to the correct classifier under 20%, 40%, and 80% random label noise in the synthetic classification example.
  • Robustness caveat: Negative tilts may fail under adversarial or structured noise, producing high error on clean data at 40% and 80% noise.
  • Fairness and federated learning: TERM achieves higher test accuracy than baselines and can match Genie ERM, while TERM-FL improves worst-device accuracy with similar average accuracy.
  • Variance reduction: For HIV-1 generalization, TERM is competitive with strong baselines and can achieve similar accuracy across classes without an extra tuned decision-threshold hyperparameter.

J.1 DATASETS AND MODELS

The experiments apply TERM across diverse datasets, models, and use cases, using validation-based hyperparameter selection and fixed negative tilts for noisy-data settings.

  • Datasets and applications: TERM is evaluated across regression, classification, fair PCA, class imbalance, federated learning, annotator noise, and generalization applications.
  • Datasets and models: The study uses datasets including Drug Discovery, CIFAR-10, Default Credit, and HIV-1, with task-specific models and preprocessing procedures.
  • Hyperparameters: Positive tilts are selected from a limited validation candidate set, while noisy-training experiments generally use t = -2.
  • Hyperparameters: Other hyperparameters are tuned on validation data, while baseline methods use original-paper or released-code defaults.
  • Scope and limitations: The discussion cautions that incorrectly tuning t can magnify biased or corrupted data relative to ERM.
  • Scope and limitations: The experiments rely on common benchmark datasets, some containing potentially sensitive information, and do not provide a comprehensive treatment of membership bias.
Loading 2007.01162v2…