CVaR anchor regression protects against rare shifts

1 Introduction

• CVaR or GroupDRO applied to total environment-wise risks can emphasize environments with high noise, whereas CVaR anchor regression reweights only squared mean residuals.This distinction matters when noise levels vary across environments or covariates are heteroscedastic.Source: §1 · pp. 1–2
• CVaR anchor regression replaces the average squared mean residual with its CVaR, targeting rare shifts while leaving average residual variance unchanged.By contrast, reweighting total environment-wise risks can emphasize noisy environments and inflate variance without improving robustness.Source: §1 · p. 2
• For sufficiently small α, its robustness set is the scaled convex hull of training shifts and their negatives, covering every training shift when γ ≥ 1.At α = 1, the method recovers ordinary anchor regression; decreasing α enlarges its robustness set beyond the anchor-regression ellipsoid.Source: §1 · p. 2
• In the example with 90% of training data from P1 and 10% from P2, α = 0.1 covers P2's rare shifts with γ = 1.The shift second moment is 0.9 I_d2, so ordinary anchor regression requires γ ≥ 10 to cover those shifts.Source: §1 · p. 2, Figure 1 · p. 3

2 Population formulation and shift robustness

• The population formulation studies risk under interventions on A and specifies that the noise law under an intervention is marginalized over A.Expectations and CVaR are taken under the training distribution unless another law is specified.Source: §2 · p. 3

2.1 Anchor regression for heteroscedastic noise

• Anchor regression’s robustness guarantee does not require the homoscedastic-noise assumption used by Rothenhäusler et al. (2021).Its robustness set Cγ is defined through the second moment of the training anchors.Source: §2.1 · p. 3

2.2 CVaR anchor regression and shift robustness

• CVaR anchor regression replaces the expected squared mean residual penalty with CVaR and targets large shift contributions rather than environments selected solely for high noise.The training noise is marginalized over A when defining intervention risk, so covering a training shift need not bound the risk under that environment’s potentially different noise law.Source: §2.2 · p. 4

2.3 Geometry of the robustness set

• The robustness sets are nested as α and γ vary: with one parameter fixed, increasing the other expands the set.More generally, Cα,γ ⊆ Cα′,γ′ when γ ≤ γ′ and γ/α ≤ γ′/α′.Source: §2.3 · p. 5

2.4 Equivalences for elliptical shifts and homoscedastic noise

• For centered elliptical shifts, CVaR anchor regression is equivalent to anchor regression with a rescaled penalty.The result is stated for centered elliptical distributions of the shifts.Source: §2.4 · p. 5
• In a modified model with training-marginal noise independent of A and shifts scaled by √γ, CVaR of conditional risk equals the residual-variance term plus the CVaR anchor penalty.The identity is stated for every predictor b.Source: §2.4 · p. 6
• With homoscedastic noise and the stated conditional-mean and covariance assumptions, population group CVaR and CVaR anchor regression at γ = 1 have the same minimizers.For discrete A, the paper also identifies this setting with superquantile risk minimization.Source: §2.4 · pp. 5–6

3 Computation and practical considerations

• The plug-in estimator uses empirical projections onto the anchor span and its orthogonal complement, with empirical CVaR weighting each component by 1/n.The projection matrices are estimated from A.Source: §3 · p. 6

3.1 Asymptotic normality

• Consistency and asymptotic normality hold under the stated model and regularity conditions, including finite fourth moments and smoothness or a stable cutoff.The assumptions exclude certain cases with an atom at the CVaR cutoff.Source: §3.1 · pp. 6–7

3.2 Optimization

• The Rockafellar–Uryasev representation makes joint optimization over the coefficient and threshold convex, while softplus smoothing can improve gradient-solver convergence.The authors suggest annealing the smoothing parameter toward zero.Source: §3.2 · p. 7

3.3 Finite sample bias

• The plug-in squared mean residual is biased upward by the environment noise variance divided by sample size, and debiasing would make the objective non-convex.Small or noisy environments can consequently be selected by the CVaR tail despite small true mean residuals.Source: §3.3 · p. 8

3.4 Inclusion of an intercept

• For CVaR anchor regression, centering before fitting and including a constant column are no longer equivalent intercept choices.The implementation uses a constant column, corresponding to a penalty on group-wise mean residuals.Source: §3.4 · p. 8

Example 1

• In Example 1, CVaR anchor regression improves on standard anchor regression and GroupDRO in a setting with heteroscedastic noise and non-elliptical shifts.The training environments have sample proportions 0.45, 0.45, 0.05, and 0.05.Source: §4 · p. 8, Figure 4 · p. 11

Example 2 (Anticausal model)

• In Example 2, simultaneously decreasing α and γ can improve risk on both P1 and P3, while Group CVaR performs worse by emphasizing high-noise P2.Figure 2's left panel reports the population risks on P1 and P3.Source: §4 · p. 8, Figure 2 · p. 9

Example 3 (Misspecified model)

• In Example 3, at fixed γ, decreasing α lowers risk at both shift-range endpoints up to a point; at fixed α, changing γ trades off their risks.Figure 2's right panel reports population risks at δ = 0 and δ = 4.Source: §4 · pp. 8–9, Figure 2 · p. 9

Sampling variance

• For elliptical shifts, lowering α at fixed effective penalty leaves population risk unchanged but increases variance; for non-elliptical shifts or misspecification, lower α can reduce population risk.With γ fixed, variance rises sharply as α falls from 1 to 0.1; when γ scales inversely with cα, it remains approximately constant for α ≥ 0.1.Source: §4 · pp. 9–10, Figure 3 · p. 10

Examples 1 to 3

• Once n ≥ 1000, both median and 90th-percentile risks decrease as α decreases in the examples; for Example 2, smaller α can raise risk when n < 1000.Figure 4 summarizes analytically evaluated population test risks over 500 simulations.Source: §4 · p. 10, Figure 4 · p. 11

5 Nonlinear models

• For a nonlinear predictor, the anchor loss replaces the linear residual with the model residual; the shift-robustness guarantee requires separate analysis.Source: §5 · p. 11

5.1 Tree-based models

• For gradient boosting, worst-case reweighting updates environment weights at each iteration, and the approach is applied to the NYC taxi dataset.Source: §5.1 · p. 12

5.2 Deep learning

• For neural networks, joint Rockafellar–Uryasev optimization or alternating environment-weight updates are possible, but minibatches trade off accurate environment means against tail coverage.Larger batches help; applying CVaR reweighting to noisy estimates introduces further bias that variance correction does not eliminate.Source: §5.2 · p. 12

5.3 Distributional CVaR anchor regression (classification)

• For ordered or censored responses, applying CVaR to score-residual penalties would extend anchor regression to a distributional analogue.Source: §5.3 · p. 12

6 NYC taxi dataset application

• The linear models use 6′033′360 July trips for training and are evaluated on 202′790 April 2020 trips from 235 zones.Pickup zone defines 261 training environments.Source: §6 · p. 12
• For CVaR anchor boosting, simultaneously decreasing α and γ lowers CVaR0.1 at similar mean risk.The models use up to 1,000 trees, maximum depth 3, and learning rate 0.1.Source: §6 · p. 12
• The neural-network models use three 64-unit hidden layers and select the checkpoint with the lowest validation CVaR0.1.Training uses five warmup epochs followed by 25 epochs with the CVaR anchor regression loss.Source: §6 · pp. 12–13
• At similar mean risk, decreasing α improves CVaR0.1 and α-quantiles for α ⩽0.2 across the reported model fits.Figure 5 reports weighted mean and CVaR0.1 as γ and α vary, and ranked, trip-weighted zone risks for fits with similar mean risk.Source: §6 · p. 13, Figure 5 · p. 14

7.1 Other divergence penalties

• An f-divergence-bounded worst-case expectation can replace CVaR of squared projected residuals while preserving linear-estimator convexity and transferring theorem 3.For χ2 divergence and sufficiently small ρ, the penalty reduces to the mean plus 2ρ times the variance of the squared projected residuals; CVaR is used for its tail-average interpretability.Source: §7.1 · p. 15

7.2 Another parametrization

• For γ ⩾1, replacing the expectation in anchor regression’s penalized mean squared error with CVaR yields an alternative parametrization linked to bounded density reweightings.The stated density bounds are 1/γ ⩽ Q/Ptrain ⩽ (1 + γ + α)/(γα).Source: §7.2 · p. 15

7.3 Continuous anchors

• Although theorem 3 allows continuous anchors, the examples and NYC taxi application use discrete anchors.The authors identify evaluating continuous-anchor settings and when the method improves over standard anchor regression as future work.Source: §7.3 · p. 15

7.4 Projections with respect to worst-case reweightings

• The objective projects using Ptrain even when evaluating the penalty under Q; for continuous anchors, projection can change under reweighting.Restricting reweightings to be anchor-measurable is difficult to formulate with finite samples; unrestricted reweighting can change within-environment noise distributions and yield a positive penalty without between-environment differences.Source: §7.4 · p. 15

Declaration of AI use

• Claude Code and OpenAI Codex assisted with writing, proof development, examples, calculations, figures, and implementation; the author takes responsibility for the final work and reported results.The stated assistance included checking and rewriting AI-generated proofs.Source: Declaration of AI use · p. 16

A Details on including an intercept

• For anchor regression with α = 1, centering gives an intercept equal to the mean residual, and the losses share the same minimizer b⋆.The displayed decomposition supports the equivalence of the centered and constant-column formulations in this case.Source: A Details on including an intercept · p. 17
• For α < 1, the anchor penalty uses a supremum over worst-case distributions, so the intercept can no longer be extracted from the penalty.The reweighted projected residual mean need not be zero; in the example, the minimizing intercept is 1/2 rather than the training residual mean 1/4.Source: A Details on including an intercept · p. 17
• The two intercept approaches differ in their robustness sets: centering under the training distribution yields a set using reweighted second moments around the training mean.The passage notes that these reweighted second moments need not equal covariance under Q.Source: A Details on including an intercept · p. 18
• With a constant column in both X and A, the CVaR anchor loss is expressed as a residual-variance term plus a CVaR penalty on squared projected residuals.The displayed loss uses MA,1 and PA,1 projections.Source: A Details on including an intercept · p. 18
• The appendix gives the minimizing coefficients an interpretation as worst-case robust regression coefficients under a set C2.Its stated assumptions include the structural model, PA1 = 1, α ∈(0, 1], and γ ⩾0.Source: A Details on including an intercept · p. 18
• With α = 1 and γ > 0, losses (2), (3), and (4) have the same minimizers; for any α ∈(0, 1] and γ ⩾1, losses (3) and (4) have the same minimizers.The excerpt also states that these latter losses have the same [truncated].Source: A Details on including an intercept · p. 18

• Theorem 3 implies Theorem 1 as the special case α = 1.Source: §B.1 · p. 18
• The proof considers random shifts v with E[vvT] bounded by γ times the second moment of shifts under some Q in Qα.The bound is E[vvT] ⪯ γ M E_Q[AAT] Mᵀ.Source: §B.1 · pp. 18–19
• Taking the supremum over Q in Qα and shifts satisfying the second-moment bound yields the worst-case expression.Source: §B.1 · p. 19
• A deterministic shift v′ = Kr_b/(r_bᵀKr_b)^(1/2) attains equality in the bound.Source: §B.1 · p. 19

Proposition 4

• The set inclusion proof constructs a probability measure Q′ by adjusting the density of Q while preserving its probability mass.It uses a convex combination of the rescaled density and the constant density 1/α′.Source: §B.2 · p. 19

Proposition 5

• For α small enough that every probability vector is admissible, the shift set is the convex hull of the scaled training shifts and their negatives.The set contains ±√γδ_e, and the converse follows from the directional bound |uᵀv| ≤ √γ max_e |uᵀδ_e|.Source: §B.2 · p. 19

B.3 Proofs of section 2.4

• The proof bounds cα between 1 and 1/α using the mean and CVaR of Z², with E[Z²] = 1.The bounds are 1 ≤ cα ≤ 1/α.Source: §B.3 · p. 20

B.4 Consistency and asymptotic normality

• The estimator is consistent, and the proof establishes asymptotic normality under the stated cases and regularity conditions.Consistency follows from uniform convergence on compact sets and uniqueness of the population minimizer; the asymptotic-normality arguments use a positive-definite Hessian limit.Source: §B.4 · pp. 20–21

B.5 Sampling variance with a scalar Gaussian anchor

• In the scalar Gaussian-anchor example, the limiting variance is given by Vγ,α = 2(γcα)^4 + 4(γcα)^3 + 13(γcα)^2 + 12γcα + 12 + γ²vα.Source: §B.5 · p. 22

B.6 Proofs of appendix A

• The proofs show that the minimizers and minimum values coincide for the specified losses.For the first case, the text states this for all three losses; for the second, it states equivalence for both losses.Source: §B.6 · pp. 23–24