Group Variable Selection via a Hierarchical Lasso and Its Oracle Property

1 Introduction

• Grouped predictors occur in ANOVA factors, polynomial or basis expansions, and biological pathways, where identifying relevant groups and their components can matter.In gene-set selection, a pathway may relate to a process without every gene in that pathway being relevant.Source: §1 · pp. 3, 5
• Lasso can select individual coefficients in an unimportant group rather than removing the group, potentially selecting more groups than necessary.The authors note that shrinkage and variable selection address prediction accuracy and interpretability, respectively.Source: §1 · pp. 3–4
• Earlier approaches include blockwise thresholding of empirical wavelet coefficients and extensions of lasso using group-level norms.The paper cites Antoniadis and Fan (2001), Yuan and Lin (2006), and Zhao, Rocha and Yu (2009) in discussing prior approaches.Source: §1 · pp. 4–5
• L2- and L∞-norm group penalties remove groups but select all variables within any selected group, limiting within-group flexibility.For the L∞ penalty, coefficients within a group may also tend to have equal magnitudes, potentially causing bias that jeopardizes prediction accuracy.Source: §1 · pp. 5–6
• Hierarchical lasso (HLasso) removes unimportant groups while retaining within-group variable selection, and its improved form is shown to have the oracle property under appropriate tuning.The oracle property means performing as well as if the correct underlying model were known in advance.Source: §1 · p. 6

2 Hierarchical Lasso

• The decomposition β_kj = d_kα_kj gives each group a shared factor d_k and variable-specific factors α_kj, encoding the hierarchy.The group factor controls coefficients across the group; the α_kj factors represent differences among its variables.Source: §2 · p. 6
• The penalty on d_k can set every coefficient in group k to zero, while the variable-level penalty can set some coefficients to zero when d_k remains nonzero.Thus the criterion supports both group removal and selection within retained groups.Source: §2 · p. 7
• The two tuning parameters can be replaced by λ = λ1 · λ2, and the resulting criterion is equivalent to the two-parameter form.The paper states that the final fitted models from the two formulations are the same, though their d_k and α_kj representations may differ.Source: §2 · p. 7

3 Algorithm

• Alternating updates estimate α_kj with d_k fixed and d_k with α_kj fixed; because each step decreases the objective, the procedure converges.The α_kj step is a lasso problem and the d_k step a non-negative garrote problem, each solvable with path algorithms or quadratic programming.Source: §3 · p. 8

3.1 Orthogonal Case

• When predictors are orthogonal, the algorithm's Step 3 and Step 4 updates have closed-form solutions.The orthogonal-design discussion includes a wavelet-setting example, with groups representing frequency scales and within-group variables representing time locations.Source: §3.1 · p. 9
• The orthogonal-case updates use soft thresholding, with variable-level shrinkage inversely proportional to the group factor.A larger group factor indicates a more important group and less shrinkage; a smaller factor indicates a less important group and more shrinkage.Source: §3.1 · p. 9

4 Asymptotic Theory

• The criterion can be written equivalently using β_kj, and the local maximizers of the two formulations map to one another.The reparameterization is β_kj = d_kα_kj; the original-coefficient penalty is singular at β_kj = 0 even when other coefficients in the group are nonzero.Source: §4 · p. 10
• Theorem 2 establishes that hierarchical lasso can remove unimportant groups under its stated asymptotic conditions.The analysis allows the total variable count, group count, and group sizes to change with sample size.Source: §4 · p. 11
• For the stated root-(n/P_n)-consistent estimate, all zero coefficients within nonzero groups are not necessarily removed with probability tending to one.The paper says the method can remove some, but not necessarily all, unimportant variables within important groups.Source: §4 · pp. 11–12
• The next section improves hierarchical lasso to address its limitation in removing all unimportant variables within important groups.The paper identifies this limitation when the set of zero coefficients within nonzero groups is nonempty.Source: §4 · p. 12

5 Adaptive Hierarchical Lasso

• Adaptive hierarchical lasso penalizes coefficients differently, assigning smaller weights to variables with stronger effects and larger weights to weaker ones.Ordinary least-squares or ridge estimates may be used to compute the weights; γ is specified as a positive constant.Source: §5 · p. 12

5.1 Oracle Property

• Theorem 4 states the existence of a consistent local maximizer, with its asymptotic results framed for independent, identically distributed observations satisfying regularity conditions.The section says the required conditions can be satisfied by appropriately choosing the tuning parameter and adaptive weights.Source: Problem Setup · p. 14

5.2 Likelihood Ratio Test

• The section develops a likelihood-ratio test for linear hypotheses, including testing the significance of several covariates simultaneously, with an asymptotic null distribution under Theorem 4(b)'s conditions.The proofs are described as non-trivial extensions of Fan and Li (2001) and Fan and Peng (2004) because of the grouping structure and adaptive weights.Source: Problem Setup · p. 13, §5.2 · pp. 14–15

Simulation results

• In the not-all-in-all-out case, hierarchical lasso performs slightly better than L2-norm group lasso in prediction accuracy; in the all-in-all-out case, L2-norm group lasso performs slightly better.Across variable selection, the four shrinkage methods identify important variables similarly, while L2-norm group lasso and hierarchical lasso remove unimportant variables more effectively than lasso and L∞-norm group lasso.Source: §6 · pp. 16–17, Table 1 · p. 17

Adaptive hierarchical lasso

• Adaptive hierarchical lasso removes unimportant variables more effectively than SCAD and adaptive lasso in the all-in-all-out case, and outperforms them in prediction accuracy.In the not-all-in-all-out case, its advantage is smaller but it still performs slightly better, especially in removing unimportant variables.Source: §6 · pp. 17–18, Table 2 · p. 18

Sample-size comparison

• As sample size increases from 200 to 3200, all three oracle methods improve, with MSEs approaching Oracle OLS and correct-model selection probabilities approaching one.Adaptive hierarchical lasso performs best overall, with its largest gap in removing unimportant variables at moderate sample sizes.Source: §6 · pp. 18–19, Figure 1 · p. 19

Data and evaluation

• The analysis considered 391 overlapping gene sets, each containing more than 15 genes, and evaluated methods over 100 random train-test splits.Each split used 33 training samples and 17 test samples; tuning used five-fold cross-validation.Source: §7 · pp. 20–21

Modeling

• Because genes may belong to multiple gene sets, the analysis extends hierarchical lasso to overlapping groups and uses logistic regression for the binary response.When each variable belongs to only one group, the overlapping-group criterion reduces to the non-overlapping criterion.Source: §7 · pp. 20–21

Prediction accuracy

• 14% average misclassification rate (standard error 1.8%) for logistic hierarchical lasso, versus 23% (1.7%) for logistic lasso and 32% (1.2%) for logistic L2-norm group lasso.Across most samples, hierarchical lasso classified correctly in most random splits, with predictions seeming slightly more stable than the alternatives.Source: §7 · p. 21

Gene selection

• Logistic lasso and logistic hierarchical lasso selected CDKN1A most frequently, while logistic L2-norm group lasso rarely selected it.The paper notes CDKN1A's expression is tightly controlled by p53 (Loh, Moritz, Contente and Dobbelstein 2003).Source: §7 · p. 21

Gene-set selection

• Logistic hierarchical lasso selected the atm pathway and radiation sensitivity gene sets most frequently, and frequently selected CDKN1A, MDM2, and RELA or BCL2 within them.Radiation sensitivity was also selected by GSEA and GSA; atm pathway shared genes with several gene sets selected by those methods.Source: §7 · p. 22

Discussion

• Numerical results indicate hierarchical lasso works well, especially when variables within a group are associated with the response in a not-all-in-all-out pattern.The method removes unimportant groups while retaining the ability to select variables within groups, and the improved version has the oracle property.Source: §8 · p. 22

Multi-response regression

• In multi-response regression, coefficients for the same predictor across responses form a group; shrinking its group effect to zero removes that predictor from every response.If the group effect is nonzero, individual coefficients can still be zero, allowing a predictor to be relevant to some responses but not others.Source: §8 · pp. 23–24

Oracle property

• The paper reports an oracle property for both group selection and within-group selection, whereas Huang, Ma, Xie and Zhang (2009) considered the property only for group selection.The theory applies to generalized maximum likelihood estimation, while Huang, Ma, Xie and Zhang (2009) considered penalized least squares.Source: §8 · p. 24

Overlapping groups

• For overlapping groups, a variable belonging to multiple groups is not penalized more heavily than variables belonging to only one group, unlike the approach of Huang, Ma, Xie and Zhang (2009).The paper notes that it is uncertain whether the heavier penalty in that approach is desirable.Source: §8 · p. 24

Proof of Lemma 1

• The proof sets up the criteria Q* and Q⋆ and considers a local maximizer of Q* as the target of the argument.The supplied passage fragments the subsequent local-maximizer statement.Source: Proof of Lemma 1 · p. 27
• The text states that a neighborhood around a local maximizer of Q* contains a corresponding local maximizer, though the statement is incomplete in the passage.The passage begins a similar claim for another criterion but does not provide its full conclusion.Source: Proof of Lemma 1 · p. 28
• For a local maximizer of criterion (8), the related hierarchical-lasso estimate is defined by β̂_kj = d̂_k α̂_kj; if d̂_k = 0, then α̂_k = 0.The supplied lemma fragment does not include its remaining conclusion.Source: Proof of Lemma 1 · p. 28

• If the estimated group scale is nonzero, the within-group parameter and the group’s coefficient 1-norm are also nonzero.Source: Proof of Lemma 2 · p. 28
• A small scaling perturbation can increase the criterion, contradicting local maximality when the scale parameter is inconsistent with its paired within-group parameter.Source: Proof of Lemma 2 · p. 29

Proof of Theorem 1

• A sufficiently small coefficient perturbation does not increase Q, so the mapped estimate is a local maximizer of Q(λ, β).Source: Proof of Theorem 1 · pp. 29, 31

Regularity Conditions

• The regularity conditions include iid observations, a common support, identifiability, and bounded higher-order likelihood derivatives for diverging parameter counts.The text says these conditions guarantee asymptotic normality of ordinary maximum-likelihood estimates for a diverging number of parameters.Source: Regularity Conditions · pp. 32–33

Proof of Theorem 3

• On the sphere ∥u∥ = C, D_n(u) < 0 uniformly, completing the stated proof.Source: Proof of Theorem 3 · p. 35

Proof of Theorem 4

• With probability tending to 1, the estimated coefficients for inactive coordinates are zero, and the inactive-group coefficient vector is zero.Source: Proof of Theorem 4 · p. 37

Proof of Theorem 2

• A root-n consistent local maximizer is stated to satisfy Pr(β̂_C = 0) → 1.Source: Proof of Theorem 2 · p. 39

Proof of Theorem 5

• The proof of Theorem 5 introduces two lemmas and states that their combination yields equation (33).Source: Proof of Theorem 5 · pp. 40, 43, Proof of Lemma 4 · p. 42

Proof of Lemma 4

• Under condition (b) of Theorem 4 and the null hypothesis H0, Lemma 4 is stated; combining (36) and (37) yields (33).Source: Proof of of Lemma 3 · p. 41, Proof of Lemma 4 · p. 42