Distributionally Robust Learning

Introduction

• The learning framework is motivated by medical data outliers and the sensitivity of OLS, while the monograph mainly uses interpretable, analyzable linear models with some nonlinear extensions.The introduction also identifies overfitting and high-dimensional predictors as reasons to seek sparse models, and notes that nearest-neighbor ideas can yield piecewise linear predictors.Source: Introduction · pp. 6–8
• Most studied problems are static, single-period, and assume independently and identically distributed data; the monograph mainly uses linear predictive models, except for its nonparametric K-NN model in Section 7.The stated scope excludes most dynamic settings and focuses on the intersection of statistical learning and DRO under the Wasserstein metric.Source: Introduction · pp. 9–10
• The monograph formulates robust learning as minimizing worst-case loss over an ambiguity set, addressing noisy training data or test data contaminated with outliers.The stated goal is to make learning insensitive to data uncertainty characterized by distributions within a prespecified distance of a nominal distribution.Source: Introduction · p. 9, §1.3 · p. 17

1.1 Robust Optimization

• Robust optimization models data uncertainty with a deterministic uncertainty set and chooses a solution optimal for the worst-case realization.The introduction describes robust counterparts as computationally tractable for many classes of uncertainty sets.Source: §1.1 · p. 11

1.2 Distributionally Robust Optimization

• Wasserstein DRO minimizes worst-case expected loss over a probabilistic ambiguity set, and its ambiguity sets can include continuous and discrete distributions.The monograph adopts a data-driven Wasserstein ambiguity set; unlike KL-based sets centered on discrete empirical data, it can allow probability mass beyond the empirical support and accounts for distances between support points.Source: §1.2 · pp. 12, 15, 16

1.3 Outline

• The monograph develops Wasserstein-based DRO learning methods and covers predictive and prescriptive models, problem-tailored theory, and real-world applications.Its stated scope includes background on DRO and the Wasserstein metric, synthetic and real-world case studies, and models ranging from robust regression to multi-output learning and K-NN-informed decisions.Source: §1.3 · pp. 16–17
• The monograph studies DRLR with worst-case expected absolute loss, GWGL for group-level sparsity, and multi-output DRO as an extension of single-output learning.It also introduces decision making informed by DRLR-based K-NN estimation; the supplied passage does not specify the remainder of that model’s description.Source: §1.3 · p. 17

1.3. Outline

• The decision framework selects the best action by predicting outcomes under each action with K-NN and a distance metric weighted by the DRLR solution.Source: §1.3 · p. 17
• The semi-supervised method estimates a robust classifier from partially labeled data by constraining the marginal to match unlabeled data or varying the ambiguity-set center to reflect label uncertainty.Source: §1.3 · p. 18
• The reinforcement-learning framework injects robustness into an MDP’s probabilistic transition model and derives a regularized lower bound on its distributionally robust value function.Source: §1.3 · p. 18
• The monograph presents Wasserstein metric properties and general DRO formulations before covering regression, grouped-variable selection, multi-output learning, multiclass classification, decision-making, semi-supervised learning, and reinforcement learning.Source: §1.3 · p. 18

2.1 Basics

• Wasserstein distance between discrete distributions is the minimum cost of transporting probability mass between their support points subject to supply and demand constraints.The order-1 distance is the LP optimum; order-t distance uses the powered cost matrix and takes the t-th root.Source: §2.1 · p. 26

2.2 A Distance Metric

• When the underlying cost is a proper metric, every order-t Wasserstein distance for t ≥1 is a metric, and the order-1 distance is convex in the distribution vectors.The metric properties include non-negativity with equality only for identical distributions, symmetry, and the triangle inequality.Source: §2.2 · pp. 27, 28, 30

2.3 The Dual Problem

• LP duality gives an equivalent maximization formulation with dual feasibility constraints, while Kantorovich duality extends the transport formulation to arbitrary measures on Polish spaces.Complementary slackness requires f_j + g_i = s(i, j) wherever transport flow is positive.Source: §2.3 · p. 30, §2.3.1 · p. 32

2.4 Some Special Cases

• For one-dimensional distributions with absolute-distance cost, Wasserstein distances can be computed from ordered support points or CDFs, and Gaussian order-2 distance has a closed form.The Gaussian result uses the Euclidean distance cost.Source: §2.4.1 · p. 32, §2.4 · p. 33, §2.4.3 · p. 34

2.4.2 Sliced Wasserstein Distance

• Sliced Wasserstein distance averages one-dimensional Wasserstein distances over projections, with Monte Carlo integration providing a computational approximation in higher dimensions.The projections push distributions on R^d onto R.Source: §2.4.2 · pp. 33–34

2.5 The Transport Cost Function

• Transport cost should reflect data knowledge: ℓ∞ suits sparse models, ℓ2 or weighted ℓp suits dense models, and group norms can encode grouped predictors and group sparsity.The described (2, ∞)-norm treats variables within a group with an ℓ2 norm and compares groups with an ℓ∞ norm.Source: §2.5 · pp. 34–36

2.5.1 Transport Cost Function via Metric Learning

• A metric-learning approach selects a weighted ℓ2 transport cost in a data-driven way using nearby same-label samples and pairs of samples that are far apart.The weight matrix is symmetric and positive semidefinite; closeness can be defined by an ℓ2-based k-nearest-neighbor relation.Source: §2.5.1 · p. 36

2.5. The Transport Cost Function

• The transport-cost subsection presents absolute and relative metric-learning formulations and robustifies them against inaccurate labels or triplets [Blanchet et al., 2019c].It refers to a sequential iterative algorithm alternating between the weight matrix and uncertain parameters.Source: §2.5 · p. 37

2.6 Robustness of the Wasserstein Ambiguity Set

• The Wasserstein ambiguity set can retain the clean distribution while excluding the outlying one, enabling an estimator robust to outliers.The theorem analyzes a mixture Pmix = qPout + (1 − q)P and establishes W1(Pout, Pmix) + W1(P, Pmix) = W1(Pout, P).Source: §2.6 · pp. 39, 41

2.7 Setting the Radius of the Wasserstein Ball

• When q < 0.5, a Wasserstein ball centered at the contaminated mixture can include the clean distribution and exclude the outlier distribution, protecting against outliers.For sufficiently large samples, the empirical distribution approximates the mixture; the radius must satisfy W1(P, Pmix) ≤ ϵ < W1(Pout, Pmix).Source: §2.7 · p. 42
• As contamination decreases, the distance gap between clean and outlying distributions grows, allowing a smaller radius; for contaminated test data, qW1(Pout, P) < ϵ < W1(Pout, P) can include the mixture but exclude outliers.The discussion treats the outlying distribution as the minority, so q is assumed below 0.5.Source: §2.7 · pp. 43–44

2.7.1 Measure Concentration

• Measure concentration guides radius selection by bounding empirical-to-true Wasserstein distance under a norm metric and light-tail assumption, but unknown constants can make the resulting radius conservative and impractical.Other proposed radius formulas rely on finite bounded support and a discrete true distribution, with restrictions arising because KL divergence diverges when the true distribution is continuous and the empirical measure is discrete.Source: §2.7.1 · pp. 44–46

2.7.2 Robust Wasserstein Profile Inference

• Robust Wasserstein profile inference selects the smallest radius making the true parameter plausible with 1 − α confidence, and its profile-based set is a confidence region.The asymptotic result depends on specified assumptions, and its limiting variable involves unknown parameters and the true distribution, requiring further relaxation for practical use.Source: §2.7.2 · pp. 47–50

Example: Optimal Radius Selection for Wasserstein Distributionally Robust Logistic Regression Using RWP Inference

• The logistic-regression example demonstrates using the Robust Wasserstein Profile function and its limiting variable to select the Wasserstein ambiguity-set radius.The example concerns a binary-label classifier with predictor x and coefficient vector β.Source: Example: Optimal Radius Selection for… · p. 50
• The procedure seeks a conservative stochastic upper bound for R̄(1) that is independent of the unknown β* and P*.The passage also notes that the bound is obtained using a positive-definite condition and stochastic dominance.Source: Example: Optimal Radius Selection for… · pp. 51–52
• The ambiguity-set size for distributionally robust logistic regression is selected using a procedure based on stochastic dominance.The order-1 Wasserstein metric defines the set, and Theorem 2.7.4 is applied with t = 1 to derive the optimal radius.Source: Example: Optimal Radius Selection for… · pp. 51–52
• The first procedure step estimates the (1 − α) quantile of ∥r̃∥_r, where r̃ is Gaussian.The supplied passage truncates the Gaussian specification after its mean notation.Source: Example: Optimal Radius Selection for… · p. 52

3.1 Dual Method

• Lagrangian duality addresses the inner infinite-dimensional maximization, and finite-growth losses satisfy weak duality, vP ≤ vD.The section notes that infinite growth can make the DRO problem intractable by causing strong duality to fail.Source: §3.1 · pp. 53–55, §3.1.1 · p. 56

3.1.2 Strong Duality

• When the loss has finite growth rate, the dual problem has a minimizer and strong duality holds with finite value: vP = vD < ∞.Source: §3.1.2 · p. 59

3.2 The Extreme Distribution

• A worst-case distribution exists when a dual minimizer exists and the stated pointwise minimizer set is nonempty P0-almost everywhere; one such distribution is a convex combination of two perturbations of the nominal distribution.Source: §3.2 · p. 60

3.3 A Discrete Empirical Nominal Distribution

• With an empirical nominal distribution and finite growth rate, the primal problem has a strong dual, and an existing worst-case distribution can be supported on at most N + 1 points.When Z is convex and h is concave, the stated empirical-distribution formulation can also be reduced.Source: §3.3 · p. 61

3.3.1 A Special Case

• For convex loss, the DRO problem can be relaxed to empirical loss plus a regularizer whose strength is the ambiguity-set size and whose form uses the dual norm.The relaxation becomes an equality when Z = Rd.Source: §3.3.1 · p. 62, §3.3 · p. 63

3.4 Finite Sample Performance

• Under Assumption B, choosing radius ε = εN(α) yields an order-1 Wasserstein DRO estimator whose out-of-sample loss can be certified not to exceed its training optimum with high probability.The significance parameter α is in (0, 1).Source: §3.4 · p. 65

3.5 Asymptotic Consistency

• As sample size N grows, choosing significance levels αN that converge to zero at a carefully chosen rate makes the DRO optimal value and solution converge to those of the original stochastic optimization problem.The ambiguity-set radius is εN(αN).Source: §3.5 · p. 66
• The DRO optimal value converges almost surely to the original stochastic optimum.The theorem assumes a Lipschitz loss and an ambiguity set with radius ϵN(αN).Source: §3.5 · pp. 66–67
• If the DRO solutions converge to β0, then β0 is almost surely an optimal solution to the original stochastic problem.The result assumes lower semicontinuity of the loss in β; the proof also concludes almost-sure convergence to an optimal solution.Source: §3.5 · pp. 67–68

4.1 The Problem and Related Work

• Least Absolute Deviation and M-estimation are not resistant to large predictor deviations, while high-breakdown methods address predictor-space contamination.Examples include Least Median of Squares, Least Trimmed Squares, S-estimation, and MM-estimation.Source: §4.1 · p. 70
• The method uses absolute residual loss and distributional hedging to reduce outlier influence, relaxing the Wasserstein DRO problem to a convex optimization problem.The ambiguity set contains distributions close to the discrete empirical distribution; the relaxation connects its radius to a regularization penalty.Source: Distributionally Robust Linear… · p. 69, §4.1 · p. 71
• Wasserstein DRO regression outperforms a host of regression models in prediction and estimation accuracy, and achieves higher outlier-detection AUC than M-estimation.The approach also establishes performance guarantees for out-of-sample loss and coefficient-estimation error.Source: §4.1 · p. 71

4.2 The Wasserstein DRO Formulation for Linear Regression

• The formulation chooses the order-1 Wasserstein metric because higher orders yield zero growth rate in the limit, removing the Wasserstein-ball structure.The loss must have finite growth rate for the stated generalization guarantee.Source: §4.2 · pp. 72–73
• Duality and convexity of the loss allow the inner supremum to be relaxed, and the loss-specific growth-rate term is computed for absolute loss.For this loss, κ(β) equals the dual norm of (−β, 1).Source: §4.2 · p. 73
• The Wasserstein DRO formulation coincides with regularized LAD, with a penalty equal to the loss growth rate times the Wasserstein-ball radius.A zero loss growth rate removes the effect of distributional uncertainty, leaving empirical loss minimization.Source: §4.2 · p. 75
• The chosen norm determines the tractable model form: the ℓ2-induced metric gives a convex quadratic problem, while the ℓ1-induced metric gives a linear program.A positive definite weight matrix can scale coordinates when predictor and response magnitudes differ substantially.Source: §4.2 · pp. 75–76
• The regularizer penalizes (−β, 1) in the dual Wasserstein space, accounting for transportation cost in both predictors and the response.Penalizing only β corresponds to infinite transportation cost along the response direction.Source: §4.2 · p. 77

4.3.1 Out-of-Sample Performance

• Rademacher complexity yields a generalization bound whose terms depend on training loss and the coefficient’s dual norm, linking the loss function to the regularizer.The empirical Rademacher complexity bound decreases inversely with the square root of sample size.Source: §4.3.1 · pp. 77–78
• The estimator’s expected loss on new data is bounded by sample-average loss plus terms depending on the dual norm of its regression coefficient.The result holds with high probability over the observed samples and extends to bounded loss functions.Source: §4.3.1 · pp. 79, 81
• The generalization bounds do not depend on the dimension of the data, supporting applicability in high-dimensional settings.The section also gives sample-size conditions for limiting the relative difference between expected new-data loss and sample-average loss; greater confidence or stricter tolerance requires more samples.Source: §4.3.1 · pp. 81–82

4.3.2 Discrepancy between Estimated and True Regression Planes

• The estimation analysis replaces those random parameters with probabilistic bounds using sub-Gaussian data, bounded covariance eigenvalues, and Gaussian width tied to the true coefficients’ geometry.The sample-size condition depends on the Gaussian width of a transformed geometric set; a further lemma relates that width to the covariance eigenvalues.Source: §4.3.2 · pp. 84, 86, 88
• A preliminary estimation-error bound relates the difference between estimated and true coefficients to the geometry of the true coefficients, but depends on random parameters from the observation matrix.The bound is described as crude, motivating the subsequent probabilistic analysis of those parameters.Source: §4.3.2 · pp. 85–86
• The main performance guarantee bounds the estimation bias of the solution to (4.15) under the stated assumptions and theorem conditions.The result combines Theorem 4.3.7, Corollary 4.3.10, and Lemma 4.3.11.Source: §4.3.2 · p. 91
• The estimation error bound depends on the variance of (x, y) and the geometrical structure of the true regression coefficient.Source: §4.3.2 · p. 91
• The bound does not decay to zero as N grows because the nonzero mean of the absolute residual |y−β′x| propagates into estimation bias.Source: §4.3.2 · p. 91

4.4 Experiments on the Performance of Wasserstein DRO

• The Wasserstein interpretation of ℓ1 regularization projects coefficient sparsity onto (x, y) by using a distance metric that extracts only a subset of coordinates.Penalizing β alone implies infinite transportation cost along y, whereas penalizing (−β, 1) corresponds to finite cost along y, equally weighted with x.Source: §4.4 · pp. 92–94

4.4.1 Dense β∗, Outliers in both x and y

• For dense β∗ with outliers in both x and y, Wasserstein ℓ2 performed best across all four metrics, beating ℓ2-regularized LAD, Wasserstein ℓ∞, ℓ1-regularized LAD, and SR-loss approaches.AD-based methods often drove all β coordinates to zero because the loss magnitude was small relative to the coefficient norm.Source: §4.4.1 · p. 98

4.4.2 Dense β∗, Outliers Only in x

• For dense β∗ with outliers only in x, Wasserstein ℓ2 matched the best methods, EN and ridge, while ℓ1- and ℓ2-regularized LAD and Wasserstein ℓ∞ performed poorly.The ℓ2-regularized LAD assumes infinite transportation cost along y, making it overly sensitive to moderate response noise; penalizing (−β, 1) helps Wasserstein ℓ2 remain competitive despite AD loss.Source: §4.4.2 · p. 101

4.4.3 Sparse β∗, Outliers in both x and y

• For sparse β∗ with outliers in both x and y, Wasserstein ℓ∞ and ℓ1-regularized LAD performed well, while AD-based formulations had better mean MSE than SR-based methods.The ℓ∞ metric emphasizes influential coordinates; choosing the wrong norm, such as Wasserstein ℓ2 here, could cause enormous estimation error.Source: §4.4.3 · p. 104

4.4.4 Sparse β∗, Outliers Only in x

• For sparse β∗ with outliers only in x, Wasserstein ℓ∞ and ℓ1-regularized LAD performed best; these ℓ1-regularized methods were comparable with EN and LASSO.Among improperly regularized ℓ2 formulations, SR appeared superior to AD when variation in y was reduced.Source: §4.4.4 · p. 107
• With a properly selected norm, Wasserstein DRO outperformed other methods in generalization and estimation; penalizing (−β, 1) improved performance, and AD was superior to SR under large response variation.These are the section's summary findings across the experiments.Source: §4.4.4 · p. 107
• Wasserstein DRO has more stable estimation performance than other methods when predictor correlation varies.Source: §4.4.4 · p. 107

4.5 An Application of Wasserstein DRO to Outlier Detection

• The application treats unlabeled two-class detection as regression-based outlier detection because samples are not known in advance to be normal or abnormal.The section compares the Wasserstein formulation with M-estimation for outlier-detection capability.Source: §4.5 · p. 107

4.5.1 Experiments on Synthetic Data

• Wasserstein DRO’s out-of-sample AUC curve was flatter across radius choices than regularized LAD’s, and its maximum achievable AUC was significantly higher.The passage attributes low AUC at small radii to low confidence that the Wasserstein ball contains the true distribution, and excessive conservatism to overly large radii.Source: §4.5.1 · p. 111, Figure 4.10 · p. 111
• Wasserstein DRO consistently outperformed the other approaches in synthetic ROC comparisons, with its curve lying well above theirs.Methods using the AD loss, including Wasserstein DRO and regularized LAD, tended to outperform methods using the SR loss, including M-estimation.Source: §4.5.1 · pp. 111–112, Figure 4.11 · p. 112

4.5.2 CT Radiation Overdose Detection

• For CT overdose detection, the method fits Wasserstein DRO regression after LASSO variable selection and flags visits whose predicted and received doses differ statistically.A 200-case manual validation reported specificity of 0.85 and sensitivity of 0.91.Source: §4.5.2 · p. 113
• 72.5% improvement in F1 score was reported for Wasserstein DRO over the cutoff and OLS alternatives on CT radiation data.Among the top 40 outliers, OLS had 7 false positives (17.5%), while all 40 identified by Wasserstein DRO were real outliers.Source: §4.5.2 · p. 114, Table 4.1 · p. 114

4.6 Summary

• The proposed linear-regression procedure uses an ℓ1 loss under Wasserstein DRO and establishes a connection between the data metric and the regularization term.The formulation’s specific model form depends on the norm space used for the Wasserstein metric.Source: §4.6 · p. 114
• The general formulation provides out-of-sample generalization guarantees and bounds estimation bias.Source: §4.6 · p. 114
• Numerical examples demonstrate the formulation’s advantages and examine the loss, regularizer, and Wasserstein norm-space choices.The regularizer can be adjusted to the characteristics of the data.Source: §4.6 · p. 114

Selection

• Wasserstein DRO with the (2, ∞)-norm recovers the GLASSO penalty for grouped-variable selection under absolute-residual regression and log-loss classification.This connects group sparsity to distributional robustness and gives a robustness interpretation of the penalty.Source: Selection · p. 115

5.1 The Problem and Related Work

• For LAD and logistic regression, the DRO reformulation yields the GLASSO penalty with magnitude equal to the ambiguity-set radius, linking regularization to robustness.The framework also generalizes beyond the square-root-loss setting used to recover GSRL.Source: §5.1 · pp. 116–117

5.2 The Groupwise Wasserstein Grouped LASSO

• The Groupwise Wasserstein Grouped LASSO (GWGL) covers LAD regression and logistic regression, with a separate formulation for overlapping groups.The setup considers continuous and binary responses.Source: §5.2 · p. 117

5.2.1 GWGL for Continuous Response Variables

• For continuous responses, GWGL uses a weighted (2, ∞)-norm on grouped predictors and the response to recover the GLASSO penalty through Wasserstein DRO.Groups are normalized by their sizes to prevent large groups from dominating the distance metric.Source: §5.2.1 · pp. 118, 119, 121

5.2.2 GWGL for Binary Response Variables

• For binary responses, GWGL applies the weighted (2, ∞)-norm to grouped predictors and extends to finite-growth losses such as the hinge loss.The formulation uses an effectively infinite response weight so observations with different labels are infinitely far apart.Source: §5.2.2 · pp. 121–123

5.2.3 GLASSO with Overlapping Groups

• For overlapping groups, Wasserstein DRO recovers latent GLASSO, whose selected support is a union of predefined groups.Latent variables avoid shrinking shared covariates to zero merely because they belong to a group that is shrunk.Source: §5.2.3 · pp. 123–125

5.3 Performance Guarantees to the DRO Groupwise Estimator

• The performance analysis considers prediction quality and the grouping effect, and applies generalization and estimation guarantees to GWGL-LR while deriving prediction results for GWGL-LG.The grouping effect concerns similarity of estimated coefficients for predictors in the same group as their sample correlation increases.Source: §5.3 · p. 125

5.3.1 Performance Guarantees for GWGL-LR

• For GWGL-LR, the general DRO theorems bound Relative Risk, Relative Test Error, and Proportion of Variance Explained for coefficient estimates on new samples from the training distribution.The bounds use the weighted (2, ∞)-norm with weights (1/√p1, . . . , 1/√pL, M).Source: §5.3.1 · p. 126
• The GWGL-LR performance guarantees include bounds for RR, RTE, and PVE, summarized in a corollary under the specifications of Theorem 4.3.12.The preceding result uses λmax(Σ), the maximum eigenvalue of Σ.Source: §5.3.1 · pp. 126–127
• For standardized predictors, higher within-group correlation makes GWGL-LR coefficient estimates more similar, with perfectly correlated predictors receiving equal estimates, supporting group-level sparsity.The discussion also suggests using predictor correlation as a grouping criterion.Source: §5.3.1 · pp. 127–128

5.3.2 Performance Guarantees for GWGL-LG

• Under Assumptions O and P, the GWGL-LG estimator has a prediction-error bound that holds with probability at least 1 −δ for any 0 < δ < 1.The theorem considers an optimal solution obtained from N training samples and a new i.i.d. sample.Source: §5.3.2 · p. 129
• GWGL-LG has the same coefficient-difference bound as GWGL-LR, with within-group predictor coefficients converging to the same value at rate O(√1 −ρ).This supports group-level sparsity when predictor correlation is used to define groups.Source: §5.3.2 · pp. 130–131

5.4.1 GWGL-LR on Synthetic Datasets

• GWGL-LR has better prediction and estimation performance than all other approaches considered on synthetic datasets.The experiments also report that its advantage over GLASSO with ℓ2-loss is more significant under high noise, and that its performance is more stable as within-group correlation varies.Source: §5.4.1 · pp. 135–136

5.4.2 GWGL-LG on Synthetic Datasets

• The GWGL-LG synthetic evaluation measures classification and logloss performance and uses within-group difference (WGD) to assess grouped variable selection.A smaller WGD indicates that coefficients within a group are closer and can be jointly selected or dropped.Source: §5.4.2 · p. 137
• On synthetic datasets, GWGL-LG achieved the smallest WGD among the models and successfully dropped one predictor group, while the alternatives dropped none of the four groups.Its WGD was significantly lower than LG-EN's; the result supports group sparsity, consistent with the stated GLASSO grouping effect when within-group correlation is high.Source: §5.4.2 · pp. 139–140, Table 5.4 · p. 139, Table 5.5 · p. 140
• GWGL-LG balanced parsimony and effectiveness on synthetic data, attaining very high TAR with significantly higher TDR than dense models that achieved almost perfect TAR.LG-LASSO and LG-EN had the highest TDRs but significantly worse TARs; the authors conclude GWGL-LG outperformed the alternatives when both measures are considered.Source: §5.4.2 · p. 140
• With q = 30% outliers and incorrectly grouped predictors, GWGL-LG still achieved satisfactory prediction, almost perfect TAR, and nonzero TDR.The incorrect spectral grouping used group sizes 2, 3, 5, and 10 rather than the true sizes 3, 4, 6, and 7.Source: §5.4.2 · pp. 140–141

5.4.3 An Application to Hospital Readmission

• GWGL-LR achieved the lowest mean out-of-sample MAD for hospital length of stay, improving it by 7.30% over the best alternative, GLASSO with ℓ2-loss.The authors note this could mean 1 or 2 days better prediction for longer stays.Source: §5.4.3 · p. 142, Table 5.6 · p. 142

5.5 Summary

• The chapter presents Wasserstein DRO formulations that recover the GLASSO penalty for LAD and logistic regression, connecting group-sparse regularization with robustness and deriving finite-sample prediction-error bounds.It also discusses the grouping effect and motivates spectral clustering with a Gaussian similarity function for grouping predictors.Source: §5.5 · pp. 143–144
• Across synthetic and surgical-record experiments, GWGL formulations produced more accurate and stable estimates, stronger group-level sparsity, and models that identified most effective predictors while remaining reasonably parsimonious.The summary highlights these properties especially for noisy data or data potentially contaminated with outliers.Source: §5.5 · p. 144

Distributionally Robust Multi-Output Learning

• The work develops Wasserstein DRO formulations for multi-output linear regression and multiclass classification, addressing a gap in robust learning with multiple responses.The formulations target settings where covariates and responses or labels may be contaminated by outliers; the prior exception noted is Hu et al., 2016, for multiclass classification under the φ-divergence metric.Source: Distributionally Robust Multi-Output… · p. 149, §6.1 · p. 151

6.1 The Problem and Related Work

• The optimization-based approach yields compact, computationally solvable models without explicitly modeling response correlations, while remaining agnostic to covariate and response correlation structure.The coefficient matrix’s novel matrix-norm regularizer associates coefficients for potentially correlated responses through the dual norm of the distance metric in data space.Source: §6.1 · pp. 150–152

the Wasserstein metric.

• The Wasserstein metric’s norm space determines the regularizer, while the loss is assumed Lipschitz continuous on that same norm space.If the loss is restricted to norms, this assumption suggests choosing l(z) = ∥z∥r, matching the data-space distance metric.Source: §6.2.1 · pp. 152–153, the Wasserstein metric · p. 157

6.2.1 Distributionally Robust Multi-Output Linear Regression

• For multi-output linear regression, the Wasserstein DRO relaxation produces MLR-SR and MLR-1S regularizers that aggregate coefficient norms across responses or predictors.MLR-SR cannot decompose into K independent terms when r ≠ 1; MLR-1S cannot decompose into K subproblems when s ≠ 1, because its regularizer entangles coefficients.Source: §6.2.1 · pp. 156–157

6.2.2 A New Perspective on the Formulation

• The Lr,s matrix norm summarizes columns using their ℓr norms and then takes the ℓs norm, giving a compact expression for both MLR relaxations.It is the Frobenius norm when r = s = 2; the relaxations are convex when the loss is convex, and regularization strength reflects the Wasserstein ambiguity-set size and loss Lipschitz constant.Source: §6.2.2 · pp. 157, 158, 160

6.2.3 Distributionally Robust Multiclass Logistic Regression

• The multiclass logistic regression setup represents each of K classes with a K-dimensional binary label vector and models its conditional distribution given predictors.The class label vector has exactly one coordinate equal to 1, and class-specific coefficient vectors are estimated to predict labels.Source: §6.2.3 · p. 160
• MLG-SR and MLG-1S are convex; when K=2, both reduce to regularized logistic regression, and their regularizers encode predictor and label uncertainty.The Wasserstein radius and class count also determine regularization magnitude.Source: §6.2.3 · pp. 163–164

6.3.1 Performance Guarantees for MLR Relaxations

• Under their stated assumptions, MLR-SR and MLR-1S have high-probability out-of-sample prediction-error bounds; smaller coefficient-norm bounds reduce the regularization contribution, but training loss can determine which model performs better.The text suggests MLR-SR for r ≥ 2 and MLR-1S for r = 1, while recommending validation to choose between them.Source: §6.3.1 · pp. 166–167

6.3.2 Performance Guarantees for MLG Relaxations

• Under their stated assumptions, MLG-SR and MLG-1S have high-probability out-of-sample log-loss bounds controlled by regularizer magnitude and training loss.The text suggests MLG-SR for r ≥ 2 and MLG-1S for r = 1 based on their respective norm bounds, but recommends validation to select a model.Source: §6.3.2 · pp. 168–169

6.4.1 MLR Relaxations

• For dense B*, r=2 relaxations had lower out-of-sample prediction bias; when B* implied an ℓ1 data-space metric, r=1 performed better.The experiments varied the Wasserstein-ball radius; cross-validated comparisons likewise found lower prediction error and variance with a properly chosen r.Source: §6.4.1 · p. 171, Figure 6.1 · p. 172, Figure 6.2 · p. 173
• In Table 6.1, the two r=2 relaxations improved WMSE by 4.6% versus MLR-SR (r=1) when B* was normally distributed.This comparison used cross-validated Wasserstein radius ϵ and averages over 100 repetitions.Source: §6.4.1 · p. 171, Table 6.1 · p. 172
• MLR-1S has the smallest out-of-sample prediction error with small variance, improving WMSE by 1%–5% at 20% outliers and 3%–7% at 30%.MLR-SR and MLR-1S also improve AUC by 2%–11% at 20% outliers and 6%–17% at 30%, except relative to FES; FES has high AUC but worse predictive performance.Source: §6.4.1 · pp. 174–175, Table 6.3 · p. 175, Figure 6.3 · p. 176

6.4.2 MLG Relaxations

• For dense B*, MLG-SR and MLG-1S with r=2 have higher accuracy and lower log-loss, while r=1 performs better when B* has equal row absolute sums.With normally distributed B*, r=2 formulations improve CCR and log-loss by 5% compared with r=1 formulations.Source: §6.4.2 · p. 177, Table 6.4 · p. 177, Table 6.5 · p. 178
• Across Tables 6.6 and 6.7, DRO-MLG improves CCR by 1%–5% and 5%–6%, MPD by 67%–400% and 50%–650%, and log-loss versus ridge MLG by 6% and 5%.The tables correspond respectively to type-1 B* with 30% outliers and type-2 B* with 40% outliers; DRO-MLG loses to vanilla MLG on log-loss while achieving higher classification accuracy and MPD.Source: §6.4.2 · p. 179, Table 6.6 · p. 179, Table 6.7 · p. 180

6.5 Summary

• The DRO-MLR and DRO-MLG formulations connect robustness with regularization, and their regularizers control out-of-sample prediction error.The summary reports out-of-sample performance comparable to or slightly better than other models, with significantly higher robustness to outliers.Source: §6.5 · p. 180

Informed K-NN

• The proposed informed K-NN model predicts outcomes under each action with a robust nonlinear predictor and uses predicted outcomes to determine a randomized prescription.The medical applications use diabetes and hypertension datasets from the EHRs of a major safety-net hospital in New England.Source: Informed K-NN · p. 183, §7.1 · p. 184

7.1 The Problem and Related Work

• Because observational data lack counterfactuals and can be noisy and nonlinear, the method estimates outcomes from similar individuals within each action group.Wasserstein DRO regression identifies outcome-relevant features, and neighbors are selected using a regression-coefficient-weighted distance.Source: §7.1 · pp. 183–184

7.1. The Problem and Related Work

• Unlike Bertsimas et al. (2017), the method robustifies regression, randomizes prescriptions, provides theoretical guarantees, and triggers treatment changes based on improvement over the previous regimen.It also derives a closed-form threshold, rather than selecting the threshold by cross-validation, and applies when the standard of care is unknown or ambiguous.Source: §7.1 · p. 186

7.2 Robust Nonlinear Predictive Model

• Wasserstein DRO estimates action-specific regression coefficients that weight K-NN distances, and the neighbors’ outcomes are averaged to predict each action’s outcome.Under the stated assumptions, the prediction is close to the true future outcome with high probability; the bias bound depends on sample size, predictor and response variation, and nonlinear smoothness.Source: §7.2 · pp. 187, 193

7.3 Prescriptive Policy Development

• The randomized policy selects actions with probability proportional to the exponential of the negative predicted outcome, allowing exploration among comparable options and reducing vulnerability to prediction errors.For an appropriate ξ, its expected true outcome is nearly optimal; as ξ tends to infinity, the policy converges to choosing the action with the lowest predicted outcome.Source: §7.3 · pp. 194, 196
• The prescriptive policy selects a threshold so the probability that expected improvement falls below it is small, estimating predicted-outcome means and standard deviations by simulation.The simulation repeatedly subsamples each prescription group and fits robust regression coefficients to estimate the conditional prediction distribution.Source: §7.3 · p. 199

A Special Case

• As ξ →∞, the randomized policy assigns probability 1 to the action with the lowest predicted outcome, yielding a deterministic policy.The threshold level T(x) is slightly modified for this special case.Source: A Special Case · p. 199

7.4 Developing Optimal Prescriptions for Patients

• The application uses Boston Medical Center records from 1999–2014 to recommend diabetes regimens minimizing future HbA1c and hypertension treatments minimizing future systolic blood pressure.The datasets contain 11,230 diabetes patients and 49,401 hypertension patients; the analysis defines therapy lines, visits, and future outcomes using different time windows for each condition.Source: §7.4 · p. 200, §7.4.1 · pp. 201–202

7.4.1 Description of the Datasets

• The datasets use 12,016 valid diabetes visits and 26,128 valid hypertension visits, grouped by standard of care, with 63 standardized features in each dataset.Diabetes outcomes average HbA1c 75–200 days after visits; hypertension outcomes average systolic blood pressure 70–180 days afterward.Source: §7.4.1 · p. 202

7.4.2 Model Development and Results

• DRLR-informed K-NN produces the largest outcome reductions with relatively stable performance, while randomized policies perform similarly to deterministic ones and slightly better for hypertension.All evaluated models outperform the current prescription and standard of care. DRLR+K-NN also outperforms the alternatives on predictive metrics, though using it for counterfactual imputation may favor the same model.Source: §7.4.2 · pp. 205–207, Table 7.1 · p. 205, Table 7.3 · p. 207

7.4.3 Refinement on the DRLR+K-NN Model

• A patient-specific neighbor count discards distant visits and generally yields larger outcome reductions than a uniform neighbor count.The cross-validated distance thresholds were 2.5 for diabetes and 1 for hypertension; truncation was tested in both training and imputation or only in imputation/evaluation.Source: §7.4.3 · p. 208

7.5 Summary

• The work combines distributionally robust optimization with K-nearest neighbors for interpretable prediction, establishes out-of-sample guarantees, and develops a randomized prescriptive policy with theoretical guarantees.Source: §7.5 · p. 208
• Applications to diabetes and hypertension datasets from a major safety-net hospital provide numerical evidence of predicted outcome improvement from the algorithm.The datasets were obtained from the hospital.Source: §7.5 · p. 209
• The methodology is optimal in expected true outcome and derives a closed-form clinically meaningful threshold for activating the randomized prescriptive policy.Source: §7.5 · p. 209

8.2 Distributionally Robust Reinforcement Learning

• In model-based MDPs, the section injects robustness into transition models and derives a regularized lower bound for the distributionally robust value function.The reinforcement-learning setting is multi-period optimization for learning a policy.Source: Learning · p. 210, §8.2 · p. 220

8.1 Distributionally Robust Learning with Unlabeled Data

• Unlabeled data can improve supervised-DRO generalization through ambiguity-set constraints, while robustifying SSL instead addresses perturbations in both labeled and unlabeled data.The two approaches have different objectives: one uses information from unlabeled data; the other adapts DRO to an existing SSL model.Source: §8.1 · p. 211, §8.1.1 · p. 213, §8.1.2 · p. 216

8.1.1 Incorporating Unlabeled Data into Distributionally Robust Learning

• Frogner et al., 2019 constrain feature marginals using unlabeled data, while Blanchet and Kang, 2017b restrict joint support, potentially hurting generalization to unseen data.The marginal-based constraint preserves robustness to distributions supported across the data space; the joint-support restriction excludes points outside the empirical observations.Source: §8.1.1 · pp. 212–213

8.1.2 Distributionally Robust Semi-Supervised Learning

• Najafi et al., 2019 adapt DRO to partially labeled SSL by selecting a soft-label-consistent distribution as the Wasserstein-ball center, hedging against perturbations around uncertain labels.The consistent set contains distributions that differ in their soft labels for unlabeled data, including the empirical measure for the true complete dataset.Source: §8.1.2 · pp. 216–217
• For optimistic learning, λ < 0 promotes higher-entropy soft labels, and Theorem 8.1.1 shows an SSDRO minimizer also minimizes the empirical SSAR objective.For fixed β, the inner infimum selects soft labels tending to reduce loss; the theorem’s equivalence holds for λ < 0 and the same ϵ and λ.Source: §8.1.2 · pp. 217–218

8.2.1 Deterministically Robust Policies

• Under rectangular uncertainty sets, the robust Bellman operator is a contraction, so its equation has a unique fixed point computable by successive operator applications.Transition probabilities may vary each time a state-action pair is encountered.Source: §8.2.1 · p. 222

8.2.2 Distributionally Robust Policies

• Distributionally robust MDPs generalize deterministically robust MDPs by placing a distribution over transition models; under rectangularity, a Wasserstein ambiguity set yields a unique value-function fixed point obtainable by value iteration.Derman and Mannor, 2020 introduce the Wasserstein formulation; Derman and Mannor, 2020 and Chen et al., 2019b establish existence of an optimal policy and unique value function.Source: §8.2.2 · pp. 222–224

Selecting a Nominal Distribution

• The nominal transition-model distribution can be estimated by assigning equal mass to episode-specific empirical transitions; for large state-action spaces, approximation architectures can instead learn model parameters by logistic regression.The logistic-regression parameters are learned from the sequence of state-actions in each episode.Source: Selecting a Nominal Distribution · p. 224

A Regularization Result for the Distributionally Robust MDP

• Theorem 8.2.1 gives a lower bound on the Wasserstein distributionally robust value function, usable for Bellman value iteration and optimal-policy computation in relatively small state-action spaces.For large state-action spaces, the result extends to linearly approximated value functions.Source: A Regularization Result for the… · pp. 225, 228, 229

• The framework covers six problems: robust linear, grouped, and multi-output learning; K-NN decision-making; semi-supervised learning; and reinforcement learning.Source: Discussion and Conclusions · p. 230
• For each problem, the authors derive equivalent regularized empirical-loss formulations and present tailored theory validated in real-world medical applications.Source: Discussion and Conclusions · p. 230
• Wasserstein ambiguity sets hedge against distributions near the empirical measure, including distributions with out-of-sample support, and link regularizer form to loss growth.The text contrasts this support with φ-divergence approaches, which consider distributions supported on observed samples.Source: Discussion and Conclusions · p. 231
• When loss growth is bounded, Wasserstein DRO can be tractable beyond linear and logistic regression, in both static and sequential-data settings.Source: Discussion and Conclusions · p. 231

228 Discussion and Conclusions

• The framework covers a comprehensive range of machine-learning problems, including both predictive and prescriptive models posed using Wasserstein DRO.Source: 228 Discussion and Conclusions · p. 232
• The monograph presents novel performance guarantees tailored to each learning problem, alongside synthetic and real-world case studies reporting superior performance to alternatives.The guarantees reflect each problem’s particularities and justify using Wasserstein DRO.Source: 228 Discussion and Conclusions · p. 232
• The novel Wasserstein prescriptive model inserts K-NN into decision-making and demonstrates a robustness benefit through an out-of-sample MSE result.Source: 228 Discussion and Conclusions · p. 232
• The multi-output DRO extension has implications for robust neural-network training, including multiclass logistic-regression robustness to optimized perturbations designed to fool the classifier.Source: 228 Discussion and Conclusions · p. 232