Lecture notes on ridge regression

• The document is a collection of many well-known results on ridge regression.Source: Disclaimer · p. 2
• Its work-in-progress status means it is incomplete and may contain inconsistencies or errors.Source: Disclaimer · p. 2
• Some original sources may be missing, with references intended for addition in later versions when known.Source: Disclaimer · p. 2

1.1 Linear regression

• In the linear model, response means are Xβ and errors are assumed independent normal with variance σ²; ordinary least squares is unbiased when X⊤X is invertible.The notes introduce this model and its estimator before discussing collinear designs.Source: §1.1 · pp. 7–8

1.2 Super-collinearity of high-dimensional designs

• 0.91 correlation between the second and third FLOT-1 probes contrasts with all other cross-correlations at or below 0.20, illustrating strong design-matrix collinearity.The regression output associates the first probe with ERBB2, while collinearity between the second and third probes leaves their individual contributions harder to distinguish.Source: §1.2 · p. 10
• When p > n, X⊤X is rank deficient, so the maximum-likelihood estimate cannot be evaluated and the normal equations do not uniquely determine β.The null space of X contains nonzero directions that leave the fitted values unchanged.Source: §1.2 · p. 10

1.3 The ridge regression estimator

• Adding λI to X⊤X makes the matrix positive definite and invertible for λ > 0, defining ridge regression; in the example, eigenvalues 9, 3, 0 become 10, 4, 1 at λ = 1.The example’s first design column is the row-wise sum of the other two, making X⊤X singular.Source: §1.3 · p. 11

Definition 1.1

• The ridge estimator remains well-defined for high-dimensional designs, and varying λ produces a solution path of potentially different estimates.The regularization parameter must be strictly positive for the estimator’s definition.Source: Definition 1.1 · p. 11

1.4 Eigenvalue shrinkage

• As λ increases toward infinity, all ridge coefficients shrink toward zero, but individual coefficient magnitudes need not decrease monotonically.The λ↓0 limit coincides with the minimum least-squares estimator.Source: §1.4 · p. 13

1.4.1 Principal component regression

• Ridge regularization shrinks singular values continuously, whereas principal component regression thresholds them; at full rank(X), principal component regression gives the minimum-length least-squares estimator.Source: §1.4.1 · p. 14

1.5.1 Expectation

• With an orthonormal design, the ridge estimator is (1 + λ)^-1 times the maximum-likelihood estimator, so its expectation is biased for λ > 0 and tends to zero as λ grows.The limiting behavior of its expectation does not depend on the particular data set.Source: §1.5.1 · pp. 14–15
• For p > n and rank(X) = n, the ridge estimator’s bias splits into bias relative to the projected true parameter and bias from high-dimensionality.The high-dimensional component vanishes when X has full row rank or β lies in the row space of X.Source: §1.5.1 · p. 16

1.5.2 Variance

• The ridge estimator’s variance tends to zero as λ grows and is smaller than the maximum-likelihood estimator’s variance.The notes describe this lower variance through smaller distribution level sets for the ridge estimates.Source: §1.5.2 · p. 16

Proposition 1.2

• The ridge estimator’s variance is no greater than the ML estimator’s, and is strictly smaller when λ > 0.The corresponding distribution level sets are smaller for ridge, indicating less variation.Source: Proposition 1.2 · p. 16, Figure 1.3 · p. 17
• For an orthonormal design, ridge variance is σ2(1 + λ)^−2Ipp, which is smaller than the ML variance σ2Ipp for nonnegative λ.The ridge variance tends to zero as λ → ∞.Source: Proposition 1.2 · p. 17
• Given λ and X, the ridge estimator is normally distributed; when p > n, its distribution is degenerate and confined to R(X).With n < p, its variance describes spread across repetitions with the same design matrix, within that subspace.Source: Proposition 1.2 · pp. 17–18

Lemma 1.1

• The ridge MSE comprises a variance term and a squared-bias term; as λ grows, variance vanishes while squared bias dominates.In the limit λ → ∞, MSE approaches β⊤β, the squared bias of the zero estimator.Source: Lemma 1.1 · pp. 18–19
• For some λ > 0, ridge has lower MSE than ML; in the illustrated example, MSE[βhat(λ)] < MSE[βhat(0)] for roughly λ < 7.The theorem establishes that such a λ exists; the plotted example shows the comparison for approximately λ < 7.Source: Lemma 1.1 · pp. 19–20, Figure 1.4 · p. 20

Remark 1.1

• Under Gauss–Markov assumptions, ridge’s lower MSE than the BLUE implies it is biased; this bias complicates inference and confidence-interval construction.The notes describe debiasing proposals as a response to this difficulty.Source: Remark 1.1 · p. 21, §1.5.4 · p. 21

1.5.4 Debiasing

• The proposed debiased estimator has bias −λ2(X⊤X + λIpp)^−2β; when λd−2_x,n tends to zero, its bias is much smaller and its variance difference becomes negligible.The notes say it is not immediate how this condition translates to finite samples for a class of designs and penalty values.Source: §1.5.4 · p. 21
• The ridge penalty λ∥β∥2^2 shrinks regression coefficients toward zero, with larger λ increasing the penalty’s influence on the loss.At λ = 0, minimizing the ridge loss yields the ML estimator when it exists.Source: §1.5.4 · p. 22
• Strict convexity of the ridge loss establishes that the ridge estimator is its global minimum.The loss combines a convex sum-of-squares term with a strictly convex penalty term.Source: §1.5.4 · p. 22
• Ridge estimation is equivalent to minimizing the sum of squares subject to a squared-norm constraint, with the estimate on the constraint boundary.The matching constraint radius is c = ∥βhat(λ)∥2^2.Source: §1.5.4 · pp. 24–25

Proposition 1.4

• As λ increases, the coefficient constraint and estimator size decrease, addressing overfitting; in the artificial example, the fit follows the true relationship but also captures errors.Large coefficient estimates are described as an indication of overfitting, and constraining them decreases its probability.Source: Proposition 1.4 · p. 25

1.7 Degrees of freedom

• Ridge degrees of freedom equal tr[H(λ)], decrease monotonically with λ, and approach zero as λ tends to infinity.Degrees of freedom represent the effective number of parameters used by the estimator.Source: §1.7 · p. 25, §1.8 · p. 26

1.8 Computationally efficient evaluation

• The Woodbury identity replaces p×p inversion with n×n inversion and avoids SVD-related numerical errors; the thin-SVD formulation also uses an n×n inversion and reduces operations from O(p^3) to O(pn^2).The notes report that the thin-SVD computation is feasible on standard computers when sample sizes are in the hundreds.Source: §1.8 · pp. 26–27

1.9 Penalty parameter selection

• Restrict λ to values yielding well-conditioned matrices; a degrees-of-freedom cap ν also bounds λ, though choosing ν depends on context or a conservative rule of thumb.A conditional-number plot can identify a sensible domain; the suggested fallback upper bound is max{0, n −30} for ν.Source: §1.9 · pp. 27–28

1.9.1 Analytic expressions

• Analytic penalty proposals are often ad hoc: 366 proposals were gathered by Mermi et al. (2024), and none in their overview guarantees lower MSE than the maximum-likelihood estimator.Hoerl and Kennard (1970) derive a choice under an orthonormal design; in high dimensions, their proposal’s dependence on the maximum-likelihood estimator limits its use.Source: §1.9.1 · p. 28

1.9.2 Information criterion

• AIC balances loglikelihood fit against ridge-model complexity measured by the trace of the ridge hat matrix, but its use is questioned because λ does not change the covariate set.The penalty minimizing AIC is presented as the optimal balance of model complexity and overfitting; prediction-focused use may not benefit from an explanatory-model criterion.Source: §1.9.2 · pp. 28–29

1.9.3 Cross-validation

• Cross-validation selects λ by average held-out predictive performance; PRESS can calculate leave-one-out performance directly from the ridge hat matrix and response, avoiding refitting n models.K-fold calculations can also be reduced to matrices of at most n × n by precomputing XX⊤.Source: §1.9.3 · pp. 29–30

1.9.4 Generalized cross-validation

• Generalized cross-validation approximates PRESS and can be easier to compute using tr[H(λ)]; it also avoids a case where PRESS has no unique minimizer.The non-unique-minimizer example uses a unit diagonal design matrix.Source: §1.9.4 · p. 30

1.9.5 Other methods; 1.10.1 Early stopping; 1.10.2 Data augmentation and adding noise

• Other penalty-selection approaches include empirical Bayes and stability selection; early stopping constrains iterative solutions, while data augmentation yields the ridge estimator as a maximum-likelihood estimate.Early stopping is one-to-one related to ridge under equal nonzero eigenvalues of X⊤X and a zero initial value.Source: §1.9.5 · p. 30, §1.10.1 · p. 33, §1.10.2 · pp. 33–34

1.9.6 Randomness

• Bootstrap estimates with re-tuned penalties depart from the conditional normal distribution, and penalty-selection randomness changes both the estimator’s distribution shape and moments.The illustration uses 1,000 bootstrap samples with 10-fold cross-validation; the figure compares re-tuned and fixed-penalty estimates.Source: Figure 1.7 · p. 31, §1.9.6 · p. 31

1.9.7 Behavior around n ≈p

• Around n ≈ p, both cross-validated penalty uncertainty and ridge-estimator mean squared error peak, as fold-specific near-zero eigenvalues make the required regularization vary.The simulation attributes fold-to-fold variation in the smallest eigenvalues to variation in the regularization needed for stable predictions; similar behavior is noted for empirical Bayes.Source: §1.9.7 · pp. 31–32, Figure 1.8 · p. 32

1.10.2 Data augmentation and adding noise

• Adding p random covariate rows drawn from N(0p, λIpp), with zero responses, yields an augmented design satisfying ˜X⊤˜X ≈ X⊤X + pλIpp.The added rows provide an approximate alternative for evaluating the ridge regression estimator.Source: §1.10.2 · p. 34
• Repeatedly duplicating observations with small covariate perturbations prevents overfitting, and the perturbed-data maximum likelihood estimator approximates ridge regression.The perturbed data satisfy ˜X⊤˜X ≈ qX⊤X + qλIpp and ˜X⊤˜Y ≈ qX⊤Y.Source: §1.10.2 · p. 34

1.11.1 Role of the variance of the covariates

• Larger-variance covariates dominate the ridge coefficient paths and receive larger estimates when effect sizes are comparable.For the simulated diagonal covariance, [β(λ)]_j = j(j + 50λ)^−1(β)_j.Source: §1.11.1 · p. 34
• Scaling is equivalent to covariate-wise penalization, so scaled ridge estimates generally differ from unscaled estimates and are not obtained by simply rescaling them.The notes say the context determines whether standardization is appropriate, particularly where preprocessing is part of measurement.Source: §1.11.1 · pp. 35–36

1.11.2 Ridge regression and collinearity

• Strongly positively correlated covariates’ coefficient paths group and are shrunk less; for similar effect sizes, ridge estimates also favor positively correlated covariates.In the two-covariate explanation, when Var(U) ≫ Var(V), the estimate for the V component vanishes for large λ.Source: §1.11.2 · pp. 36–37, Figure 1.10 · p. 37
• With strongly positively collinear covariates, the expected ridge-constraint radius is larger than with orthogonal covariates, yielding a larger estimate on the β1 = β2 line.The explanation attributes the difference to the constraint radius depending on λ, X, and Y.Source: §1.11.2 · p. 37

1.11.3 Variance inflation factor

• Figure 1.11 presents contour plots of Marquardt and Garcia VIFs, showing penalization versus collinearity at fixed n = 100 and p = 50, and penalization versus sample size at fixed p = 50.Source: Figure 1.11 · p. 39

1.12 Illustration

• The section also introduces simulations examining how collinearity and penalization affect the ridge estimator’s VIF.The simulations compare two ridge VIF definitions across varied collinearity, penalty, or sample-size settings.Source: §1.12 · p. 40

1.12.1 MCM7 expression regulation by microRNAs

• The application tests whether reported microRNA regulation of MCM7 can also be observed in an independent prostate cancer dataset.The illustration uses the MCM7 finding reported by Ambs et al. (2008) as its example.Source: §1.12.1 · pp. 40–41
• The GSE20161 prostate cancer dataset includes mRNA and microRNA profiles; preprocessing leaves 90 profiles with 735 microRNA expression measurements each.Further manipulations include averaging duplicate profiles, centering expression data, and averaging probes interrogating MCM7.Source: §1.12.1 · p. 41

MCM7 expression regulation by microRNAs

• R^2 = 0.4492 for unconstrained ridge, and the response-predictor Spearman correlation is 0.6295, suggesting an association between microRNA and MCM7 expression.Penalization shrinks coefficient estimates, compressing the linear predictor’s range relative to observed expression.Source: §1.12 · p. 42
• The unconstrained ridge fit has 394 of 735 probe coefficients negative, and their distribution is nearly symmetrical around zero, so the analysis neither confirms nor refutes the negative-regulation dogma.The negative coefficients are a small majority, but their sizes do not substantially differ from positive estimates.Source: §1.12 · p. 42, Figure 1.12 · p. 42
• With 401 rather than 735 parameters, the dogma-obeying model has Spearman correlation 0.679 and R^2 0.524, a slight fit improvement over unconstrained ridge.The notes describe this as a somewhat better description with close to half the number of parameters.Source: §1.12 · p. 43
• The 401 selected microRNAs show no enrichment for predicted MCM7 targets: χ2 = 0.0478 and p = 0.827.The reported chi-squared test uses Yates’ continuity correction; selection was based on predictive power, not target status.Source: §1.12 · p. 44, Table 1.1 · p. 44

1.13 Conclusion

• Adding a ridge penalty to the sum-of-squares loss addresses empirical non-identifiability in high-dimensional linear regression.The conclusion presents ridge regression as a modification of the linear model.Source: §1.13 · pp. 44–45
• Ridge penalization is equivalent to constraining the parameter domain, which identifies the coefficients but introduces estimation bias.Source: §1.13 · p. 45
• The chapter reviews ridge properties, including moments and mean squared error, and illustrates its behavior and use in simulation and omics data.Source: §1.13 · p. 45

1.14 Exercises

• Exercises apply ridge estimation to numeric data, asking readers to compare unpenalized and penalized estimates, check shrinkage as λ increases, and assess the effect of centering data.For centered data, the stated predictor is 40 + 1.75X, with only the slope shrunk.Source: §1.14 · p. 45

Question 1.3

• Exercises examine ridge geometry and residuals, including why the ridge hat matrix is not a projection, how bias components arise, and how ill-conditioning causes numerical deviations from the regularization path.They also ask readers to identify the constrained-estimation solution and compare ridge and maximum-likelihood estimator MSEs.Source: Question 1.3 · pp. 45–46
• A proof exercise asks readers to establish that ridge can have lower estimator MSE than maximum likelihood for positive λ, and that the corresponding linear-predictor MSE is no greater.The supplied exercise gives the interval 0 < λ < σ²||β||^-2 for the estimator-MSE comparison; the notes’ proof is described as covering a larger interval.Source: Question 1.3 · pp. 46–47

Question 1.10

• Exercises explore negative penalty values and collinear or identical covariates, asking when estimates are defined, how paths relate, and how coefficient allocation affects MSE.They also ask whether explaining response variation with more identical covariates is justified.Source: Question 1.10 · pp. 47–48

Question 1.14

• Further exercises address ridge-estimator variance and interval interpretation, the effect of duplicating a design column, and large-λ behavior of coefficient signs and variance estimation.They also ask readers to relate a constraint on the ridge parameter to its penalty value and degrees of freedom.Source: Question 1.14 · p. 49

Questions 1.17–1.18

• The computational exercises compare direct and efficient ridge-estimator formulas and ask readers to benchmark their runtimes as the design dimension increases.They also ask readers to rewrite LOOCV prediction errors using the hat matrix to avoid n leave-one-out estimator evaluations.Source: Question 1.14 · pp. 49–50

Questions 1.19 and 1.22

• Penalty-selection problems ask readers to minimize ridge-estimator MSE, Allen’s PRESS, or AIC, then compare AIC and LOOCV choices for a PSA–VEGF pathway model.The PSA exercise uses prostate cancer data and asks readers to discuss how the different penalties affect the regression estimates.Source: Question 1.14 · pp. 50–51

Question 1.20

• For the supplied LOOCV exercise, the penalty minimizing Allen’s PRESS statistic is λ = ∞.The exercise asks readers to show this result.Source: Question 1.14 · p. 50

Question 1.21

• The GCV-selected penalty lies in (0, 6.53), an interval containing the unique global minimum of GCV(λ).This is the requested result for the supplied design matrix and response.Source: Question 1.14 · p. 51

Question 1.23

• A covariate-rescaling exercise investigates whether scaling changes ridge coefficient ordering, linear predictors, and the largest coefficients, both with fixed λ and LOOCV selection.It uses the microRNA–mRNA illustration and motivates rescaling by noting that covariate variance affects ridge shrinkage.Source: Question 1.14 · pp. 51–52

Questions 1.24–1.25

• Other problems ask readers to evaluate ridge estimates, bias, degrees of freedom, predictor variance, and prediction limits, including whether adding an observation changes bias or variance.They also examine optimal MSE penalties under a presumed error variance and under a potentially incorrect variance belief.Source: Question 1.14 · p. 52

2.1 A minimum of prior knowledge on Bayesian statistics

• A Bayesian specifies parameter beliefs as a prior distribution, then updates them with data to obtain a posterior, unlike the frequentist treatment of parameters as fixed quantities.The posterior is proportional to likelihood × prior.Source: §2.1 · p. 53

2.2 Connection to ridge regression

• With a Gaussian prior, the ridge estimator is both the posterior mean and the MAP estimate because the likelihood and conjugate prior yield coincident estimates.The conditional posterior mean of β equals the ridge estimator; the section also states that the posterior mean and MAP coincide.Source: §2.2 · p. 55, §2.3 · p. 56
• Ridge’s quadratic penalty corresponds to Gaussian-prior level-set constraints, whose enclosed prior mass is controlled by λ; more generally, penalized estimates are Bayesian MAP estimates under matching priors.Replacing the ridge penalty with another penalty corresponds to choosing a prior whose density makes the penalized estimator coincide with the MAP estimate.Source: §2.2 · p. 55

2.3 High-dimensionality

• In high dimensions, the likelihood provides no information orthogonal to the design-matrix rows, so posterior uncertainty in that direction reflects the prior rather than the experiment.As λ approaches zero, the posterior becomes stretched in the uninformed direction and collapses to the degenerate frequentist ridge distribution.Source: §2.3 · pp. 56–57

2.3.1 Credible set

• ≈(2p)^-1 bounds the credible interval’s coverage for θ when p is large and θ finite, so coverage tends to zero rather than the nominal 1−α.This example uses a flat prior in the normal means model.Source: §2.3.1 · p. 57, §2.3 · p. 58

2.3.2 Efficient sampling

• The Bhattacharya et al. (2016) sampler produces draws from the posterior for any (n, p), with the greatest computational speed gains when p ≫ n.Its sampled β has the ridge estimator as its mean.Source: §2.3.2 · pp. 58–59
• In high-dimensional settings, representative posterior samples can be difficult to obtain for credible-region estimation, though smaller samples may suffice for a summary such as the posterior mean.The challenge is covering relevant features across the parameter space.Source: §2.3.2 · p. 59

2.4 Markov chain Monte Carlo

• After burn-in, MCMC uses samples from a Markov chain’s stationary distribution to characterize a target posterior, including through statistics such as its moments.Monte Carlo estimates of posterior quantities converge as the sample size grows; sampling can also estimate the marginal posterior of β when σ^2 is known.Source: §2.4 · p. 59
• Metropolis-Hastings handles targets known up to a normalizing constant by proposing candidate states and accepting or rejecting them, but acceptance-rejection sampling can be impractical when no suitable dominating density exists or acceptance is too low.The chain may stay at its current state when a proposal is rejected.Source: §2.4 · pp. 60–61
• A unimodal-kernel Metropolis-Hastings sampler produced samples from the illustrated bimodal mixture distribution.The passage reports this result from a histogram, which is not shown.Source: §2.4 · p. 63
• Gibbs sampling updates one partition of the parameter vector at a time using known full conditional distributions, with each proposal accepted and regression parameters sampled alternately from their full conditionals.The method requires analytically known full conditionals and is presented as enhancing convergence to the stationary distribution.Source: §2.4 · p. 63

2.5 Empirical Bayes

• The Gibbs-sampler pseudocode specifies removing burn-in draws and selecting every f_thinning-th sample.Its inputs include the sample size, burn-in length, and thinning factor.Source: §2.5 · p. 64
• Empirical Bayes estimates prior hyperparameters from the data and then uses the specified prior to obtain posterior distributions and summaries.Reusing the data is considered acceptable when there are many model parameters relative to hyperparameters, so little information is spent estimating the hyperparameters.Source: §2.5 · p. 64
• Marginal likelihood maximization estimates hyperparameters by integrating out random model parameters and maximizing the resulting marginal posterior over the observed data and hyperparameters.This avoids fixing the model parameter at a chosen value, which would make the hyperparameter estimates depend on that choice.Source: §2.5 · p. 64
• With a conjugate Gaussian prior, the empirical Bayes estimate of the regression parameter is the ridge estimator using the estimated λ.For the coefficient prior β | σ² ∼ N(0_p, σ²λ^-1I), the notes recommend making the inverse-gamma prior on σ² very flat because the joint prior is too flexible; most rather flat choices yield similar coefficient estimates.Source: §2.5 · pp. 64–65

2.6 Conclusion

• Bayesian regression with a conjugate Gaussian prior on the coefficients coincides with ridge regression, giving the ridge penalty an interpretation as a prior; the chapter also presents resampling and empirical Bayes for choosing the hyperparameter.Resampling is presented as useful for evaluating estimators with other penalties and non-conjugate priors.Source: §2.6 · p. 65

3.1 Generalized ridge regression

• A positive-definite generalized penalty yields a strictly convex loss and unique estimator, while broader positive-semidefinite penalties allow different and correlated coefficient shrinkage toward a specified target.The usual spherical ridge penalty is recovered when the penalty matrix is a scalar multiple of the identity.Source: §3.1 · p. 68

Example 3.1

• Generalized penalty paths can move coefficients toward nonzero targets and temporarily pull one coefficient toward another; the framework also allows high-dimensional predictors to be penalized while selected covariates remain unpenalized.The example motivates this arrangement for clinical-trial covariates such as age and sex alongside molecular predictors.Source: Example 3.1 · p. 69

3.2 Constrained estimation

• Penalized and constrained generalized ridge estimates coincide when the constraint matches the penalty at the penalized estimate; with a positive-definite penalty, strong penalization drives the estimate to its target.For a positive-semidefinite penalty, the limiting constraint may retain a subspace, and the limiting estimate is determined by the sum-of-squares within it.Source: §3.2 · p. 70

3.3 Moments

• Conditionally, the generalized ridge estimator is normally distributed, and its bias separates regularization effects from high-dimensionality; bias worsens as the target moves farther from the true parameter.As all penalty eigenvalues tend to infinity, the expectation tends to the target and the variance tends to zero.Source: §3.3 · pp. 71–73

Example 3.2

• For an orthonormal design with zero target and the specified exchangeable penalty matrix, regular ridge has greater efficiency than its generalized counterpart, with generalized variance minimized at ρ = 0.At ρ = 0, the two estimators coincide.Source: Example 3.2 · p. 73

Lemma 3.1

• With unpenalized and penalized covariates, their coefficient estimators have linked moments: an error in the penalized estimate can be compensated by the unpenalized estimate.Without unpenalized covariates, the moments reduce to those for generalized ridge estimation without this covariate split.Source: Lemma 3.1 · p. 73

Corollary 3.2

• The biases of unpenalized and penalized coefficient estimates are related, and their linear-predictor biases offset; if their design columns are orthogonal, these bias and covariance relationships vanish.In that orthogonal case, the unpenalized coefficient estimator is unbiased.Source: Corollary 3.2 · p. 74

Proposition 3.2

• A sufficiently small positive penalty can lower generalized ridge regression’s MSE below maximum likelihood’s, while a perfect target with ∆ = λI gives lower MSE for every λ.The initial MSE improvement holds for any target, but a target farther from the true coefficient narrows the penalty range where it occurs; with a perfect target, the estimator is unbiased for any ∆.Source: Proposition 3.2 · pp. 74–75, §3.3.1 · p. 74

3.4 Degrees of freedom

• Degrees of freedom decrease as the penalty grows, approaching zero for a positive-definite penalty direction but not necessarily for a nonnegative-definite one.The positive-definite case has a nonrandom shrinkage limit; with a nonnegative-definite penalty, the limit may still depend on Y.Source: §3.4 · p. 75

3.5 The Bayesian connection

• Generalized ridge regression is a Bayesian posterior mean under a normal prior centered at β0 with variance σ²∆⁻¹, retaining the inverse-gamma prior on σ².The notes say a better initial guess improves the posterior mean and mode, while smaller prior variance yields a more concentrated posterior.Source: §3.5 · pp. 75–76

3.6.1 Eigenvalue-wise regularization

• Eigenvalue-wise regularization assigns separate penalties to singular-vector directions, with MSE-minimizing λ_j = σ²/α_j² when α and σ² are known.When these quantities are unknown, an iterative procedure estimates the penalties, but its limit depends on observed Y and need not attain minimum MSE.Source: §3.6.1 · pp. 76–77

3.6.2 Fused ridge estimation

• Fused ridge shrinks contiguous coefficients toward one another, producing a smooth estimated coefficient vector in the simulation.Its penalty matrix is only semi-positive definite, so the loss need not be strictly convex; a regular ridge penalty is often added.Source: §3.6.2 · pp. 77–78, Figure 3.3 · p. 78

3.6.3 A common but unknown target

• A common-target penalty shrinks selected coefficients toward their learned mean, and as regularization increases, all coefficients converge to that shared value.Because its penalty matrix is nonnegative definite, an additional penalty is required for a well-defined estimator under (super-)collinearity; this approach can suit near-duplicate covariates.Source: §3.6.3 · p. 79

3.6.4 Group-wise penalization

• Group-wise ridge assigns each covariate group a penalty parameter, so groups with lower penalties are shrunk less; groups can be formed from co-data, regions, pathways, or collinearity.The cited approaches include Van De Wiel et al. (2016), Nunez-Elizalde et al. (2019), Aldahmani and Zoubeidi (2020), and Yang and Emura (2017).Source: Figure 3.3 · p. 78, §3.6.4 · pp. 79–80

3.6.5 Updating via targeted ridge regression

• In sequential learning, each study’s generalized ridge estimate uses the preceding study’s estimate as its target to update the shared regression parameters.The data sets may have different covariate settings and sample sizes, while sharing the same β and σ².Source: §3.6.5 · pp. 80–81
• Updated ridge estimates become less biased and their 5%–95% quantile spread vanishes over time; under conditions, the estimator sequence is consistent in t.In the simulation, traditional ridge estimate quantiles remain virtually constant; the theoretical result also covers the associated linear predictor.Source: §3.6.5 · p. 81
• Targeted ridge updating is a frequentist analogue of Bayesian updating: the normal posterior becomes the next normal prior, whose posterior mean coincides with the frequentist estimator.The stated prior for the next update is centered at the previous estimate.Source: §3.6.5 · p. 81

3.6.6 Hierarchical ridge

• The hierarchical ridge estimator learns a low-dimensional target β0 = Zγ from data and has an analytic form that is most efficiently evaluated when p ≫ n > q.The target is estimated alongside β with separate penalties; its low-rank parametrization plays the role of the target in targeted ridge regression.Source: §3.6.6 · p. 82

3.7 Other connections

• The generalized ridge estimator also appears in contexts where it was not proposed as a penalized estimator.The notes introduce dropout as one such context.Source: §3.7 · p. 83, §3.7.1 · p. 83

3.7.1 Dropout

• Randomly fitting reduced models makes dropout a form of regularization that counters co-adaptation, where covariate coefficient learning depends strongly on other covariates.In linear regression, random competitors prevent estimates from relying on particular other covariates, helping avoid very large estimates and overfitting.Source: §3.7.1 · p. 83
• Dropout updates converge in expectation exponentially fast to a generalized ridge estimator, which equals the maximum likelihood estimator when X has orthogonal columns.The convergence result assumes ακ∥X⊤X∥2 < 1, independent initialization randomness, and no zero columns in X; dropout updates have larger variance due to their added randomness.Source: §3.7.1 · p. 84

3.8 Application

• The KRAS application fits ridge models to relate colorectal-cancer expression to chromosome 12 copy-number aberrations, using a banded penalty to encourage similar effects at contiguous locations.The described profiles comprise 195 samples and 927 aberrations, but the analysis is limited to ten samples; the penalty parameter is selected by leave-one-out cross-validation.Source: §3.8 · p. 84
• Generalized ridge emphasizes the KRAS peak and nearby elevated estimates, supporting a dominant cis-effect; the isolated biologically nonsensical spike is absent under ordinary ridge and is ignored.With all samples included, both ordinary and fused ridge estimates show a discernible cis-effect and virtually similar profiles.Source: §3.8 · p. 86
• The generalized ridge penalty can encode prior knowledge through β0 and parameter relationships through ∆, but exact parameter knowledge is better incorporated explicitly in the likelihood.Choosing an informative ∆ is generally less straightforward than specifying β0, with spatial structure noted as an exception.Source: §3.8 · p. 87

Question 3.14

• The notes motivate a sparse Bayesian regression prior by the possibility that most coefficients are near zero while a few clearly differ from zero.The proposed prior represents effectively zero, negative, and positive coefficients as separate mixture components.Source: Question 3.14 · pp. 90–91
• Implement the computationally efficient Gibbs sampler in R, using Algorithm 6 to draw from a high-dimensional multivariate normal distribution.The prompt also references the posterior of the error variance.Source: Question 3.14 · p. 91
• Fit Bayesian regression to model MCM7 expression from microRNA levels, then examine the posteriors of Z_j and β_j to identify coefficients clearly different from zero.The prompt also asks readers to investigate the robustness of the posterior mean of the regression parameters to hyperparameter choices.Source: Question 3.14 · p. 91
• Fit the Bayesian regression model with a normal prior and study differences between estimates; the supplied prompt cuts off before specifying which estimates.Source: Question 3.14 · p. 91

4 Mixed model

• Mixed models distinguish fixed from random covariate effects; random effects contribute to response covariance, not its mean, and accommodate generalization beyond sampled batches or cells.The notes describe estimating mixed-model parameters by maximum likelihood or REML; variance-component estimates generally require numerical optimization and iterative updating. In the cell-growth example, mixed-model fits show petri-dish-specific relationships, whereas linear regression gives the grand-mean relationship (Figure 4.1).Source: §4 · pp. 92–95
• In the cell-growth example, mixed-model fits show petri-dish-specific relationships, whereas linear regression gives the grand-mean relationship.The random cell effect can model both intercept and slope variation.Source: §4 · p. 93, Figure 4.1 · p. 93

Definition 4.1

• With covariance parameters known, Henderson’s equations give γ̂ = E(γ | Y) and a GLS estimate of β that is BLUE.These estimates are used to construct the linear predictor.Source: Definition 4.1 · p. 96

Definition 4.2

• The predictor Xβ̂ + Zγ̂ is the BLUP of Xβ + Zγ: it is linear and unbiased, with minimum prediction-error variance among unbiased linear predictors.The theorem identifies the predictor as optimal in the BLUP sense.Source: Definition 4.2 · pp. 96–97

4.1 Link to ridge regression

• Random-effects estimation is equivalent to ridge estimation up to penalty reparametrization, yields shrinkage, and avoids identifiability problems when q > n if Rθ is low-dimensionally parametrized.Fixed effects correspond to unpenalized covariates in the ridge analogy.Source: §4.1 · p. 97
• When coefficients are random, expected-GCV selects the ridge penalty as error variance divided by coefficient variance, unchanged by adding unpenalized covariates.The variance estimates can be obtained with mixed-model methods and plugged into the ratio; without adjustment for the unknown fraction of nonzero effects, the penalty may be overestimated, causing excessive shrinkage.Source: §4.1 · p. 97, §4.2 · pp. 98–99, §4.3 · p. 100

4.2 REML consistency, high-dimensionally

• Under Theorem 4.3’s high-dimensional conditions, adjusted REML estimates of the variance ratio and standard REML estimates of error variance are consistent even when the mixed model is misspecified (Jiang et al. (2016)).The result assumes positive error and random-effect variances and conditions on the design and growth of the sample, random-effect dimension, and number of nonzero effects.Source: §4.2 · p. 99

4.3 Illustration: P-splines

• P-splines translate a difference penalty on spline coefficients into a mixed model that penalizes random effects while leaving fixed effects unshrunk.The illustrated fits match the transcript data, but minor derivative discontinuities indicate slight overfitting associated with an overly large random-effect variance estimate, apparently from numerical instability when the basis dimension is large relative to sample size.Source: §4.3 · pp. 101, 103

5.1 Logistic regression

• Logistic regression links binary-response probabilities to covariates through log-odds, and each response is modeled as a Bernoulli draw with the resulting success probability.A unit increase in covariate j multiplies the odds by exp(βj).Source: §5.1 · p. 104

5.2 Separable and high-dimensional data

• Separable data make logistic maximum likelihood undefined, and high dimensionality also prevents a well-defined estimator because infinitely many coefficients can yield the same likelihood.Separation is more likely when one response class has low prevalence; with p > n, the design matrix has a nontrivial null space.Source: §5.2 · pp. 106–107

Example 5.1

• In the mutually exclusive-indicator example, the Firth-penalized likelihood lacks a well-defined high-dimensional maximizer, while early stopping offers a finite-iteration workaround.Firth penalization may resolve separation in low dimensions but does not ensure estimator existence in high dimensions.Source: Example 5.1 · p. 107, §5.2 · p. 107

5.3 Ridge estimation

• Ridge penalization makes the logistic log-likelihood strictly concave, ensuring a unique estimator; Newton–Raphson and IRLS are adapted by adding the penalty to the Hessian and update.As λ tends to infinity, coefficients shrink toward zero; with an unpenalized intercept, the intercept remains to model the success probability.Source: §5.3 · pp. 108–109

Example 5.2

• Stronger ridge penalties flatten the predicted-probability curve and enlarge the uncertain classification region between failure and success, yielding less decisive class assignments.The simulation used p = 2, n = 200, standard-normal covariates, and Bernoulli responses generated from a logistic model.Source: Example 5.2 · p. 110

Remark 5.1

• The historical approximate ridge logistic estimator addresses collinearity by adding λI to X⊤WX, but requires an available maximum-likelihood estimate and therefore does not apply to separable or high-dimensional data.Its alternative motivation uses a second-order Taylor approximation around the maximum-likelihood estimate, assumed to exist for sufficiently large n.Source: Remark 5.1 · p. 110

5.4 Moments

• The ridge logistic estimator’s moments are approximated from the final Newton–Raphson update; its coefficients and variances both vanish as λ tends to infinity.The approximation treats the penultimate estimate as non-random, and its form agrees asymptotically with the approximate estimator in Remark 5.1.Source: §5.4 · pp. 110–111

5.5 Constrained estimation

• Ridge logistic estimation is equivalent to constrained estimation, with a coefficient-norm constraint that limits large estimates and helps prevent overfitting.Its squared radius equals the squared norm of the ridge estimate, but no explicit expression for that radius is available.Source: §5.5 · p. 111

Proposition 5.1

• The squared Euclidean length of the ridge logistic estimator strictly decreases with λ and tends to zero as λ increases.The bound establishing the zero limit holds whether or not the model includes an unpenalized intercept.Source: Proposition 5.1 · p. 111

5.6 Degrees of freedom

• For known W, the ridge logistic estimator’s degrees of freedom are approximately tr[(X^T W X + λI_pp)^-1 X^T W X].The approximation uses the weights from the final IRLS iteration and treats W as nonrandom.Source: Proposition 5.2 · p. 112, §5.6 · p. 112

Proposition 5.2

• With W known, the approximate degrees of freedom decrease monotonically to zero as λ increases.This trend follows from the approximation in Proposition 5.2.Source: Proposition 5.2 · p. 112

5.7 The Bayesian connection

• Under a multivariate normal prior, the posterior mode equals the ridge logistic estimator, and Laplace approximation gives a Gaussian covariance from the negative penalized log-likelihood’s Hessian.The approximation is reported to be accurate under regularity, a smooth prior, and sufficiently large sample size; Polson et al. (2013) give a Gibbs sampler for the full posterior.Source: §5.7 · pp. 112–113

5.8 Computationally efficient evaluation

• For high-dimensional IRLS, precomputing λ^-1XX^T avoids repeated p × p matrix inversion; the LOOCV approximation can use one inverse, reducible to n × n.The general K-fold shortcut reuses submatrices of XX^T rather than evaluating the estimator for each fold.Source: §5.8 · p. 114, §5.9 · p. 115

5.10 Generalizing ridge logistic regression

• Generalized ridge logistic regression permits a nonzero shrinkage target, a nonnegative definite penalty matrix, and covariates left unpenalized; IRLS computes its estimator numerically.The penalty matrix should be positive definite in high-dimensional settings.Source: §5.10 · pp. 115–116

5.11 Application

• Baseline gene-expression profiles showed clear separation of surviving and subsequently diseased ovarian cancer patients, despite some overlap between groups.The example used 295 samples and 19,990 transcript reads; expression counts were asinh-transformed, and the intercept was unpenalized.Source: §5.11 · pp. 116–117
• The ten-fold left-out predictions nearly overlap the observed responses, indicating that the apparent in-sample predictive power largely disappears on new samples.The notes contrast this with the fit on the full data, which may reasonably describe the data at hand but extrapolates poorly.Source: Figure 5.5 · p. 118, §5.11 · p. 118

5.12 Conclusion

• Ridge logistic regression addresses high-dimensional empirical non-identifiability, and the notes state that ridge regression properties carry over to the logistic equivalent.The ridge penalty can be combined with link functions other than the identity.Source: §5.12 · p. 118

Question 5.1

• The exercises use logistic regression examples to examine an unbounded maximum-likelihood estimate and how a ridge penalty yields finite coefficient estimates.One exercise asks students to establish the unbounded behavior in a two-covariate example and show that the ridge logistic estimates are finite.Source: Question 5.1 · p. 118

Question 5.2

• An exercise asks students to show that stopping logistic-regression gradient ascent early produces a constrained parameter estimator, using a Taylor expansion of the update.The prompt gives an approximate update involving the weight matrix at the preceding iterate and asks students to iterate the argument to obtain a bound.Source: Question 5.2 · pp. 118–119

Question 5.3; Question 5.4

• The leukemia exercise probes whether an almost-perfect ridge-logistic fit reflects overfitting, using 38 patients and 3,051 genes, shuffled subtype labels, and comparing λ = 1 with λ = 1000.The breast-cancer data contain 337 patients and 24,481 genes; the exercise uses the first 1,000 genes and ER status as the response.Source: Question 5.3 · p. 119, Question 5.4 · pp. 119–120

Question 5.5; Question 5.6; Question 5.7; Question 5.8

• Further exercises cover computationally efficient IRLS, generalized ridge logistic penalties, approximation-based estimator properties, and logistic-regression behavior under specified data and prior assumptions.The IRLS prompt proposes using one SVD before iterations and inverting an n × n matrix per iteration; the remaining prompts ask for generalized-penalty algorithms, Taylor-based estimator analysis, and Bayesian or dimension-scaling properties.Source: Question 5.5 · p. 120, Question 5.6 · p. 120, Question 5.7 · p. 120

6 Lasso regression

• The ℓ1 penalty can yield a well-defined lasso estimate in high-dimensional settings, although its loss is not always uniquely minimized.The penalty is the sum of absolute regression coefficients; its non-differentiability at zero makes the analysis more intricate.Source: §6 · pp. 122–123, Figure 6.1 · p. 123

6.1 Uniqueness

• Coefficient estimates can be nonunique, but fitted predictors are unique; with continuously distributed design entries, the lasso solution is unique with probability one.When multiple coefficient estimates minimize the loss, their convex combinations are also minimizers.Source: §6.1 · pp. 123–124

Example 6.3

• The algorithms’ estimates differ in support, but each has realized loss approximately equal to 2.99, with the difference possibly due to convergence criteria; their corresponding predictors correlate at 0.999.Source: Example 6.3 · p. 125

6.2 Analytic solutions

• The lasso regularization path is piecewise linear in λ1 and tends to zero as λ1→∞; an explicit soft-threshold solution is available for orthonormal designs.No general explicit expression for the lasso estimator is known, though particular cases have analytic solutions.Source: §6.2 · pp. 126–127

6.3 Sparsity

• Lasso can set coefficients exactly to zero, enabling variable selection; if λ1 > 2∥X⊤Y∥∞, the estimate is the zero vector.The section characterizes this variable-selection property as arising from the geometry of the constrained estimation problem.Source: §6.3 · p. 128
• The lasso’s ℓ1 constraint forms a diamond whose axis-aligned corners can set coefficients to zero, producing sparse estimates without proving excluded effects are truly zero.As λ1 increases, the ℓ1 norm decreases, but the number of nonzero coefficients can temporarily increase before eventually falling to zero; non-uniqueness occurs with probability zero. Sparse models may be attractive, but some systems are not believed to be sparse.Source: §6.3 · pp. 129–131

6.3.1 Maximum number of selected covariates

• The lasso selects at most min{n, p} covariates, including at most n when p > n, so high-dimensional fits can achieve dimension reduction but their size depends on the study’s sample count.The bound is established by Osborne et al. (2000); the section also notes that letting sample size determine the upper bound can be unsatisfactory.Source: §6.3.1 · p. 131

6.4 Estimation

• Because the lasso loss has no analytic optimum, estimation relies on numerical procedures.The section introduces quadratic programming, iterative ridge, gradient ascent, and coordinate descent as approaches to finding the estimator.Source: §6.4 · p. 131

6.4.1 Quadratic programming

• Tibshirani (1996) reformulates lasso estimation as a quadratic program, which is viable for relatively small p but impractical for large p because its linear constraints grow as 2p.For p = 2, the worked quadratic-programming estimates coincide with those obtained earlier.Source: §6.4.1 · p. 131, §6.4 · pp. 132–133

6.4.2 Iterative ridge

• Fan and Li (2001) approximate the lasso’s absolute-value penalty quadratically and iteratively optimize the resulting weighted ridge loss.Coefficients approaching zero can lead to covariate removal that is irreversible, which may be problematic for (nearly) collinear covariates.Source: §6.4.2 · p. 133, §6.4 · p. 134

6.4.3 Gradient ascent

• Goeman (2010) uses a generalized derivative to handle the lasso penalty’s nondifferentiability at zero in gradient ascent, with Newton–Raphson updates used to speed convergence near the optimum.The generalized derivative supplies a steepest-ascent direction for the penalized likelihood.Source: §6.4.3 · pp. 134–135

6.4.4 Coordinate descent

• Coordinate descent minimizes the lasso loss one coefficient at a time, applying soft-thresholding to each coordinate’s corresponding maximum likelihood estimate until convergence.The section identifies glmnet as an implementation of this approach.Source: §6.4.4 · pp. 135–136
• Coordinate descent converges to a lasso-loss minimum, but can be slow when strongly collinear covariates contribute nearly equally and the weaker one is initialized first.In such cases, simultaneous updating may be preferable.Source: §6.4.4 · p. 136

6.5 Moments

• Lasso moments are generally unknown; ridge-based approximations require a nonsparse estimate, while even a better variance approximation can be poor when many coefficients are zero.The ridge approximation assumes the estimated penalty matrix is nonrandom.Source: §6.5 · p. 136
• As λ1 grows, the lasso estimate’s mean and variance tend to zero; sufficiently large penalties improve MSE for an empty model, but performance for non-sparse models depends strongly on β.For non-sparse models, the bias contribution makes whether lasso can outperform maximum likelihood unclear.Source: §6.5 · pp. 136–137

6.6 Degrees of freedom

• The lasso estimator’s number of nonzero coefficients estimates its degrees of freedom and is consistent when rank(X) = p.This result holds for λ1 > 0 under the theorem’s stated Gaussian-error model.Source: §6.6 · p. 137

6.7 The Bayesian connection

• A Laplace prior makes the lasso estimate the posterior mode; its prior scale is inversely related to λ1, while its scale-mixture Gibbs sampler does not ensure reliable uncertainty quantification in high-dimensional sparse settings.As λ1 increases, the posterior mode can reach zero; Park and Casella (2008) construct the sampler using a normal-exponential mixture.Source: §6.7 · pp. 138–139

6.8.1 Stability selection

• Stability selection chooses λ1 using covariate selection frequencies across subsamples, retaining covariates exceeding a threshold and permitting control of false stable selections under exchangeability assumptions.The notes caution that exchangeability may be difficult to assess in practice.Source: §6.8.1 · pp. 139–140, Definition 6.2 · p. 140

6.9.1 Linearity; 6.9.2 Shrinkage; 6.9.3 Simulation I: covariate selection

• Unlike ridge’s linear estimator and proportional shrinkage, lasso is nonlinear and subtracts an equal absolute amount; in simulation it shrank low-variance covariates most, favoring high-variance covariates for selection.The simulation’s regularization paths show this pattern for sufficiently large λ1.Source: §6.9.1 · p. 141, §6.9.2 · p. 141, Figure 6.10 · p. 142

6.9.4 Simulation II: correlated covariates

• The next simulation examines lasso behavior with correlated covariates, repeating a ridge-regression simulation that had shown joint shrinkage of strongly collinear covariates.The supplied passages describe the simulation setup but do not report its results.Source: §6.9.4 · pp. 142–143
• With correlated covariates, the lasso tends to select one and force the others to zero rather than shrink covariates in the same correlation block together.This behavior occurs for sufficiently large penalty values.Source: §6.9.4 · p. 143

6.9.5 Simulation III: sparsity

• Ridge performed better for dense parameters, while lasso performed better for sparse parameters; with fixed 10% relative sparsity, ridge overtook lasso beyond p = 500.For fixed absolute sparsity, the lasso’s selection frequency favored larger coefficients and weakened as dimension increased; predictive performance also became more variable as dimensions grew. The simulations suggest estimator choice depends on assumptions about coefficient structure, which may be difficult to verify.Source: §6.9.5 · pp. 144, 146

6.10 Application

• The cross-validated lasso selected 20 genes, and diagnostic plots showed some value in the predictor, but the cross-validated loglikelihood had multiple local maxima.The authors caution that lasso selection aims to explain the response, not provide an interpretable gene set; alternative gene sets may perform similarly, so a search algorithm may settle on a local penalty maximum.Source: §6.10 · p. 148

Question 6.2

• For the listed data, the straight-line model Y = β0 + β1X + ε has lasso fit Ŷ = 40 + 1.75X at λ1 = 14, with the penalty applied only to the slope.A separate excerpt specifies λ1 = 0.2 for models without and with an intercept, leaving the intercept unpenalized.Source: Question 6.2 · p. 149

Question 6.4

• Question 6.4 asks readers to plot the lasso regularization path over λ1 ∈ (0, 160] using Example 1.2.Source: Question 6.4 · p. 149

Question 6.5

• Question 6.5 uses penalty geometry to explain why lasso tends to select covariates with larger spread and how unequal penalty parameters affect estimates.It also asks readers to show equivalence between the two cases by rewriting their loss functions.Source: Question 6.5 · p. 149

Question 6.6

• Question 6.6 asks how covariate variance affects lasso selection in a toy model where one covariate is a scaled copy of another.It also presents a lasso-estimator exercise for two standardized, collinear covariates.Source: Question 6.6 · pp. 149–150
• The exercises ask students to evaluate lasso estimates at specified penalties and determine when solutions have one nonzero coefficient, including whether proposed coefficient vectors are feasible.Source: Question 6.6 · p. 150

Question 6.10

• With identical covariates, an exercise asks whether all lasso coefficient estimates are equal, given that the ridge estimates are equal for every penalty value.Source: Question 6.10 · p. 150

Question 6.11

• Exercises derive lasso estimates, examine coefficient sparsity and penalty monotonicity under orthonormal designs, and connect the normal-means MAP estimate to soft-thresholding.Source: Question 6.11 · pp. 150–151
• A two-covariate exercise asks how the lasso estimate behaves for specified data and penalties, including whether separate single-covariate estimates persist when both covariates are included.Source: Question 6.11 · p. 151

Question 6.15

• In a nested LOOCV exercise, each inner-loop optimum yields an intercept-only model, and the resulting predictions have Spearman correlation -1 with observations; students are asked why.Source: Question 6.15 · p. 151

Question 6.16

• The breast-cancer exercise uses 337 patients, 1,000 centered gene-expression features, and estrogen-receptor status to fit lasso logistic regression, select λ1 by cross-validation, check the likelihood maximum, assess fold sensitivity, and compare the fit with ridge.The feature reduction is described as a computational-speed measure.Source: Question 6.16 · pp. 151–152

7.1 Elastic net

• Elastic net combines lasso and ridge penalties to obtain a unique solution with selection potential for up to p features; joint shrinkage may require a large ridge penalty.Its constraint combines the ridge boundary’s strict convexity with lasso-like corners that enable selection.Source: §7.1 · p. 153
• Many elastic-net penalty pairs can yield similar cross-validated performance, so the chosen fit does not reliably establish whether the relationship is sparse.Fixing the penalty ratio simplifies tuning, but the mixing parameter still requires tuning.Source: §7.1 · pp. 154–155

7.2 Fused lasso

• The fused lasso penalizes adjacent coefficient differences as well as coefficient magnitudes, encouraging sparse, piecewise-constant estimates by selecting both coefficients and neighboring differences.With reasonably comparable penalty parameters, shrinking a neighboring difference to zero can also shrink its individual coefficients to zero.Source: §7.2 · p. 155

7.3 Group lasso

• Group lasso selects covariates as whole groups but does not produce within-group sparsity; sparse group lasso combines group-level selection with lasso-driven sparsity within groups.When all groups are singletons, the group and sparse group lasso penalties are equivalent to the regular lasso penalty.Source: §7.3 · pp. 156–157

7.4 Adaptive lasso

• Adaptive lasso uses inverse-initial-estimate, covariate-specific weights in a two-step procedure, varying shrinkage and potentially undoing some lasso bias.An initial coefficient of zero yields an adaptive-lasso coefficient of zero; the estimator was proposed to achieve desirable asymptotic properties.Source: §7.4 · p. 158

7.5 The ℓ0 penalty

• The ℓ0 penalty charges for each nonzero coefficient, directly targeting model size, but its 2^p subset search makes it infeasible for large-dimensional problems.Adaptive lasso approximates this penalty using estimated coefficient-specific weights; the approximation depends on the quality of those weights. Hypothesized structure is ideally incorporated explicitly into the model.Source: §7.5 · p. 158, §7.6 · p. 159

Question 7.1

• The exercises compare collinear covariate regularization paths and fit a fused lasso with λ1 = 10 and λf = ∞.They ask why the paths differ despite perfect collinearity and use a fused penalty to fit both covariates.Source: Question 7.1 · p. 159
• The exercises ask for all lasso estimates at λ1 = 2 and the fused-lasso threshold above which the estimate is zero.The fused-lasso threshold is defined as λf^(0), with the estimate (0, 0)^T for all larger λf.Source: Question 7.1 · p. 159
• The sparse fused-lasso exercises evaluate estimates and test uniqueness after dropping an observation and setting λ1 = 0.Another exercise asks how many covariates the fused lasso can select.Source: Question 7.1 · p. 160
• The elastic-net exercises use data augmentation to reformulate the loss as a traditional lasso objective and derive a coordinate-descent update.Source: Question 7.1 · p. 160
• For an elastic net with (λ1, λ2) = (3, 2), the exercises compare adding an orthogonal covariate with duplicating the first covariate.They ask how the first coefficient changes and state that duplicating the covariate halves it when λ2 is doubled.Source: Question 7.1 · p. 160
• The exercises use an elastic-net-type penalty combining |β1| and β2^2, ask for its constraint shape, and evaluate the estimate at λ1 = 4.The evaluation uses data augmentation and optimization over β2.Source: Question 7.1 · p. 160
• An orthonormal-design exercise asks to show the adaptive-lasso coefficient update when the estimator is initialized by OLS/ML.Source: Question 7.1 · p. 161, Question 7.10 · p. 161

Question 7.10

• The exercises apply the equivalence between regularized and constrained estimation to match three penalty functions to each of two sets of plotted constraints.Figure 7.4 shows the constraints for penalties involving absolute differences, squared differences, and combined ridge, lasso, and fusion terms.Source: Question 7.10 · p. 161, Figure 7.4 · p. 161