PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributionally Robust Logistic Regression

Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani, Daniel Kuhn

arXiv:1509.09259v3math.OCstat.ML

TL;DR

Logistic regression can overfit sparse training data, motivating a probabilistic robustness interpretation of regularization. The paper uses Wasserstein ambiguity sets around the empirical distribution to minimize worst-case logloss, yielding tractable reformulations, regularized logistic regression as a special case, confidence bounds, and out-of-sample guarantees.

  • Problem

    Sparse training data can cause logistic regression to overfit and perform poorly out of sample, while existing regularization lacks a distributionally robust interpretation for logistic regression.

  • Method

    The paper constructs Wasserstein ambiguity sets around the empirical training distribution and minimizes worst-case expected logloss over each set.

  • Results

    The framework admits a tractable convex reformulation, includes regularized logistic regression as a special case, and provides tractable confidence bounds and finite-sample out-of-sample guarantees.

  • Takeaways & Limitations

    Distributionally robust optimization gives regularized logistic regression a probabilistic interpretation and supports guarantees for the resulting classifiers.

  • Takeaways & Limitations

    The guarantees assume a light-tailed data-generating distribution, and finite Wasserstein metric parameters are preferable when labels are not deterministic functions of features.

Abstract

from arXiv · show

This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distributionally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

1 Introduction

The paper addresses poor out-of-sample performance and overfitting in logistic regression by developing a distributionally robust regularization perspective based on Wasserstein ambiguity sets.

  • Sparse training data can cause logistic regression to perform poorly out of sample, motivating regularization to combat overfitting.
  • The paper connects regularization in logistic regression to distributionally robust optimization, addressing a gap previously identified for support vector machines and Lasso.
  • The proposed framework derives regularized logistic regression models with intuitive probabilistic interpretations and includes classical regularized logistic regression as a special case.
  • The approach uses measure concentration results to establish finite-sample probabilistic out-of-sample guarantees.
  • The framework provides a tractable convex reformulation of the distributionally robust logistic regression model.
  • Two tractable linear programs provide confidence bounds on the misclassification probability of the resulting classifiers.

2 A distributionally robust perspective on statistical learning

The paper treats the unknown data-generating distribution as uncertain and constructs Wasserstein ambiguity sets around the empirical training distribution. It then minimizes worst-case expected logloss while preserving tractability and seeking reliability and asymptotic consistency.

  • Because the true distribution is only indirectly observed through independent training samples, the paper constructs an ambiguity set intended to contain it with high confidence.
  • The distributionally robust model minimizes worst-case expected logloss over distributions in the ambiguity set.
  • The Wasserstein construction is designed to support tractability, reliability of the robust optimizer, and asymptotic consistency.
  • The Wasserstein distance measures the minimum cost of transporting one distribution to another under a feature-label metric.
  • The ambiguity set is a Wasserstein ball centered on the empirical distribution of the training samples.
  • Setting the Wasserstein radius to zero recovers the sample-average logloss minimization problem of classical logistic regression.

3 Tractable reformulation and probabilistic guarantees

The paper reformulates Wasserstein-distributionally robust logistic regression as a tractable convex program and derives probabilistic out-of-sample guarantees under a light-tail assumption.

  • 3.1 Tractable reformulation: The Wasserstein-based robust logistic regression problem admits an equivalent tractable convex reformulation.The reformulation uses a feature-label metric and auxiliary variables, and is tractable for most commonly used norms.
  • 3.1 Tractable reformulation: As κ tends to infinity, the robust model reduces to regularized logistic regression, with the dual norm determining regularization and the Wasserstein radius determining its coefficient.In this limit, label uncertainty is excluded and the complementary label-uncertainty regularizer vanishes.
  • 3.2 Out-of-sample performance guarantees: The out-of-sample guarantee relies on choosing the Wasserstein radius using a measure-concentration bound under a light-tailed data-generating distribution.The theorem assumes finite exponential moments of the feature norm and provides constants depending on tail parameters, dimension, and the metric.
  • 3.2 Out-of-sample performance guarantees: With probability at least 1 − η, the expected population logloss of the robust solution is no greater than its worst-case objective value.The guarantee holds for all sample sizes N ≥ 1 and confidence levels η ∈ (0, 1].
  • 3.1 Tractable reformulation: The metric parameter κ controls the relative emphasis between feature mismatch and label uncertainty.It can therefore be interpreted as the model’s trust in the observed labels.

I. Light-Tail Assumption:

The paper’s guarantees use Wasserstein ambiguity sets whose radius shrinks with sample size, and extend to confidence bounds on classifier misclassification risk.

  • I. Light-Tail Assumption:: For fixed confidence level η, the prescribed Wasserstein radius ε_N(η) decreases to zero as the sample size N increases.Consequently, the ambiguity set converges to a singleton and the robust solution converges to the nominal solution.
  • I. Light-Tail Assumption:: The radius bound has different finite-sample growth regimes, with rates depending on the feature-space dimension and tail parameter.For bounded-support or Gaussian distributions, the parameter a can effectively be set to 1.
  • III. Risk Estimation: The classifier’s risk is its misclassification probability, estimated through worst- and best-case expectations over Wasserstein balls.The classifier predicts +1 when the modeled conditional probability exceeds 0.5 and −1 otherwise.
  • III. Risk Estimation: The worst-case and best-case risk values provide upper and lower confidence bounds on the true risk with probability 1 − η.Both bounds are obtained from highly tractable linear programs, and jointly satisfy R_min(β̂) ≤ R(β̂) ≤ R_max(β̂) with probability 1 − 2η.

4 Numerical Results

Experiments evaluate distributionally robust logistic regression through out-of-sample performance, logloss risk, real-world datasets, and confidence-bound estimation. The results show that Wasserstein-radius choices affect classification, tail risk, and the width and coverage of risk intervals.

  • Out-of-Sample Performance: Distributional robustness increases average CCR with ε until the Wasserstein ball contains the true distribution with high confidence.Further increasing ε then reduces average CCR.
  • Out-of-Sample Performance: For N = 10, 10^2, 10^3, the 95% confidence Wasserstein radii are ε ≈0.2, 0.02, 0.003, respectively.The implied radius scales inversely with the number of training samples.
  • Effect of the Wasserstein Ball: For ε ≥0.1, the learned weight vector approaches zero and the logloss approaches the deterministic value log(2) = 0.69.The CDF converges toward a step function for large ε.
  • Real-World Case Studies: DRLR outperforms RLR in out-of-sample CVaR by 4%–43% across the evaluated experiments.The comparison is reported for the bottom portion of Table 1.
  • Real-World Case Studies: On Ionosphere, DRLR outperforms RLR for sufficiently small ε, producing a wider range of radii with attractive out-of-sample logloss and CCR.The paper attributes this to DRLR accounting for label uncertainty.
  • Risk Estimation: At ε ≈0.05, the risk confidence interval is [0, 0.19] and contains the true risk with probability 95%.The interval widens as ε increases; at ε = 0 its bounds coincide with empirical training risk.

5 Appendix

The appendix develops the analytical reformulations and confidence-bound results underlying the distributionally robust logistic regression method. It uses convex conjugacy, dual norms, and Wasserstein duality to reduce distributional problems to tractable optimization programs.

  • Lemma 1: Convex conjugacy represents the logistic logloss as an upper envelope of infinitely many affine functions.This representation enables evaluation of the Wasserstein worst-case expression using dual norms.
  • Theorem 1: The proof handles a hybrid uncertainty variable containing continuous features and discrete labels.It represents ξ as (x, y) with Ξ = R^n × {−1, +1}.
  • Duality: Strong duality holds for every ε > 0, supporting the equivalence between the measure optimization and its tractable dual reformulation.The proof invokes Proposition 3.4 and Lemma 1.
  • Tractable Reformulation: The resulting reformulation imposes lβ(ˆx_i, ˆy_i) ≤ s_i, lβ(ˆx_i, −ˆy_i) − λκ ≤ s_i, and ∥β∥_* ≤ λ.These constraints are the finite reformulation used in the proof.
  • Risk Estimation: The risk-bound derivation represents indicator losses as finite maxima of concave functions and reformulates the worst-case problems using linear-program constraints.The resulting optimal values provide upper and lower risk bounds.
  • Risk Estimation: The upper and lower worst-case risks are interpreted as confidence bounds through Theorem 2.This connects the linear-program values to statistical guarantees.
Loading 1509.09259v3…