PaperlayerView exact revision on arXiv

Source-linked AI summary

Transfer Learning under High-dimensional Generalized Linear Models

Ye Tian, Yang Feng

arXiv:2105.14328v4stat.MLcs.LGstat.ME

TL;DR

High-dimensional GLM transfer learning must use source information carefully because some sources may be far from the target and harmful, while the relevant sparsity structure remains unclear. This paper proposes multi-source GLM transfer methods and source detection, showing improved target-coefficient error bounds when target and source are sufficiently close and positioning the work as the first study of high-dimensional GLMs under a transfer-learning framework.

  • Problem

    High-dimensional algorithms require sufficient training data, while the relevant sparsity structure remains unclear and some source data may be harmful when far from the target.

  • Method

    The paper proposes multi-source transfer learning algorithms for generalized linear models, including algorithm-free transferable source detection.

  • Results

    When target and source are sufficiently close, the target-coefficient estimation error bound improves over the classical penalized estimator using only target data.

  • Takeaways & Limitations

    The work establishes a high-dimensional GLM transfer-learning framework and studies transferable-source detection within it.

  • Takeaways & Limitations

    The paper obtains sub-optimal rates, and it remains unclear how to achieve the minimax rate when source-predictor distributions deviate.

Abstract

from arXiv · show

In this work, we study the transfer learning problem under high-dimensional generalized linear models (GLMs), which aim to improve the fit on target data by borrowing information from useful source data. Given which sources to transfer, we propose a transfer learning algorithm on GLM, and derive its $\ell_1/\ell_2$-estimation error bounds as well as a bound for a prediction error measure. The theoretical analysis shows that when the target and source are sufficiently close to each other, these bounds could be improved over those of the classical penalized estimator using only target data under mild conditions. When we don't know which sources to transfer, an algorithm-free transferable source detection approach is introduced to detect informative sources. The detection consistency is proved under the high-dimensional GLM transfer learning setting. We also propose an algorithm to construct confidence intervals of each coefficient component, and the corresponding theories are provided. Extensive simulations and a real-data experiment verify the effectiveness of our algorithms. We implement the proposed GLM transfer learning algorithms in a new R package glmtrans, which is available on CRAN.

1 Introduction

The paper develops theoretically grounded transfer-learning tools for high-dimensional GLMs, addressing limited evidence and unclear benefits of borrowing source information. It studies multi-source estimation, source detection, and confidence intervals, with simulations and real-data results supporting the proposed algorithms.

  • Motivation: Transfer learning in high-dimensional GLMs has received limited theoretical attention, despite its potential when sample size is smaller than dimension.The paper identifies the unclear impact of transfer learning under sparsity as a central gap.
  • Contributions: The authors propose multi-source GLM transfer-learning algorithms assuming both target and source data are high-dimensional and coefficient contrasts are ℓ1-sparse.This extends prior work using a multi-source setting and differs from earlier ℓ0-sparsity assumptions.
  • Theoretical results: When target and source coefficients are sufficiently close, the target estimation error bound can improve over the classical penalized estimator using only target data.The analysis also establishes minimax-optimal error rates under certain conditions.
  • Source detection: Because distant sources can cause negative transfer, the paper introduces an algorithm-free transferable source detection method that distinguishes useful from useless sources under certain conditions.The detection procedure is designed for settings where informative sources are not known in advance.
  • Statistical inference: The paper develops confidence intervals for individual coefficient components using the two-step transfer procedure and nodewise regression, with corresponding asymptotic theory.This addresses inference beyond point estimation.
  • Empirical evaluation: Extensive simulations and a real-data study demonstrate the effectiveness of the proposed GLM transfer-learning algorithms.The implementations are provided in the R package glmtrans, available on CRAN.

2 Methodology

The paper formulates high-dimensional GLM transfer learning across target and source datasets, using coefficient similarity to identify useful sources. Its two-step estimator pools selected data, then corrects transfer bias with target data.

  • Problem setup: The target coefficient β is assumed sparse, with only s of p variables contributing to the response.
  • Problem setup: A source is considered useful when its coefficient w^(k) is close to the target coefficient β.
  • Problem setup: The transferring level is ||δ^(k)||_1 for δ^(k)=β−w^(k), and A_h contains sources with transferring level at most h.
  • Two-step transfer: The general transfer algorithm first fits an ℓ1-penalized GLM on pooled target and selected-source data.
  • Two-step transfer: It then estimates the source-target contrast using target data and ℓ1 regularization to correct transfer bias.
  • Source selection: The algorithm requires knowing which sources to transfer, motivating a separate detection method for settings where source usefulness is unknown.

2 Debiasing step: Compute

The paper addresses unknown source usefulness through Trans-GLM and constructs coefficient confidence intervals using transfer-based debiasing. Its inference procedure combines an initial transfer estimator with target-data correction and nodewise regression.

  • Transferable source detection: Trans-GLM is a simple, algorithm-free, data-driven method for determining an informative transferring set.
  • Transferable source detection: The method splits target data into three folds, evaluates source-specific transfer estimators by held-out negative log-likelihood, and compares them with target-only Lasso.
  • Transferable source detection: Algorithm 2 does not require the transfer level h as input and is theoretically linked to a level-h transferring set under suitable conditions.
  • Confidence intervals: The confidence-interval procedure uses an Ah-Trans-GLM estimator as the initial point estimator.
  • Confidence intervals: It estimates the target inverse Fisher information by nodewise regression, debiases the estimator with target data, and constructs intervals for each coefficient.
  • Confidence intervals: The resulting debiased estimator is asymptotically unbiased, supporting level-(1−α) intervals for individual coefficient components.

3 Theory

The theory establishes guarantees for transfer estimation, transferable-source detection, and coefficient inference under explicit GLM assumptions. When sources are sufficiently compatible and sample-size conditions hold, transfer can improve target-only bounds while detection preserves those guarantees.

  • Theoretical scope: The theory analyzes three proposed algorithms: Ah-Trans-GLM, Trans-GLM, and a confidence-interval procedure.
  • Assumptions: Under stated GLM assumptions, predictors are subGaussian with well-behaved correlation structure, and ψ is strictly convex and infinitely differentiable.
  • Estimation guarantees: When n_Ah is much larger than n_0, the estimation-error upper bounds can improve over using only target data.
  • Estimation guarantees: Without Assumption 4, a weaker bound can still be better than the classical Lasso bound when n_Ah is much larger than n_0.
  • Detection consistency: Under identifiability and related conditions, Trans-GLM recovers a level-h transferring set with high probability.
  • Detection consistency: The detected-set estimator inherits the ℓ1/ℓ2 estimation-error bounds established for the corresponding known-set procedures.
  • Confidence intervals: Under its assumptions, Algorithm 3 provides approximately level-(1−α) inference for each coefficient component.

4 Numerical Experiments

The experiments evaluate GLM transfer learning across simulations and a county-level election study, comparing target-only, pooled, selected-source, and benchmark methods. Results generally favor transferring informative sources, while inference gains involve a trade-off between coverage and interval length.

  • Transfer learning on Ah: Ah-Trans-GLM outperforms na¨ıve-Lasso for most combinations of h and K, with performance improving as more source data become available.The improvement is consistent with sharper ℓ2-estimation error bounds as nAh grows.
  • Transfer learning on Ah: Increasing h worsens Ah-Trans-GLM performance, indicating that transfer benefits depend on source-target closeness.
  • Inference: Ah-Trans-GLM confidence intervals approach approximately 95% coverage on both signal and noise variables when K is large, but become wider as K increases.Desparsified-Lasso intervals fail to reach 95% coverage on the signal set in all three models, whereas Ah-Trans-GLM intervals achieve approximately 95% coverage on both sets for large K.
  • Transfer learning when Ah is unknown: When informative sources are unknown, Trans-GLM closely mimics Ah-Trans-GLM and can recover useful sources, whereas pooling few informative sources can cause negative transfer.As KAh increases, Pooled-Trans-GLM improves and eventually matches the selected-source methods.
  • Real-data study: In the election study, Trans-GLM performs best in four of eight scenarios and ranks among the top three in all but the MS target-state case.Pooled-Trans-GLM always improves on na¨ıve-Lasso, and Trans-GLM's gains are larger for states sharing more similarities with other states.

5 Discussions

The paper develops high-dimensional GLM transfer learning methods with theoretical guarantees, including transferable-source detection and coefficient confidence intervals. Its results improve estimation and prediction when sources are sufficiently close, while exposing limitations under predictor heterogeneity and beyond GLMs.

  • This work is presented as the first study of high-dimensional GLMs under a transfer learning framework.
  • The framework provides a prediction error measure with fast and slow rates under different conditions.
  • An algorithm-free transferable source detection method is developed to avoid negative transfer, with detailed theoretical properties.
  • The two-step transfer method constructs confidence intervals for each coefficient component with theoretical guarantees.
  • Simulations and a real-data study demonstrate the effectiveness of the proposed algorithms.
  • Extending the framework to multinomial regression, the Cox model, and other machine learning models remains an open direction.
  • Algorithm 1 achieves the minimax ℓ1/ℓ2 estimation error rate under the homogeneity assumption, but only sub-optimal rates without predictor homogeneity.
  • It remains unclear how to achieve the minimax rate when source-predictor distributions deviate substantially from the target distribution.

S.1.1 A schematic to illustrate A-Trans-GLM

Figure 5 schematically illustrates A-Trans-GLM for sources A = {1, 2, 3}. It shows how transferring pooled information from selected sources updates an initial target estimate before debiasing produces the final estimator.

  • The schematic represents the target coefficient β = w(0) and its estimation-error neighborhood.
  • The classical Lasso estimator using only target data is shown separately as a comparison point.
  • Pooling target and selected-source data yields the rough population estimator wAh and its estimation-error neighborhood.
  • When h is small, the pooled estimator wAh is close to the target coefficient β.
  • Starting from an initial target estimate, the transferring step updates it to ˆwAh using source data in A, followed by debiasing to obtain ˆβA-Trans-GLM.
  • Figure 5 depicts A-Trans-GLM using the selected source set A = {1, 2, 3}.

S.1.2.1 Explicit forms of convergence rates in Assumption 5

The appendix gives explicit GLM-specific convergence-rate conditions, prediction-error comparisons, implementation details, and supplementary simulation and real-data findings for the proposed transfer-learning methods.

  • S.1.2.1 Explicit forms of convergence rates in Assumption 5: The stated rates depend on sparsity, dimension, effective target-plus-source sample size, transfer discrepancy h, and source-specific quantities.
  • S.1.2.1 Explicit forms of convergence rates in Assumption 5: The convergence-rate analysis treats logistic, linear, and bounded-predictor Poisson regression models separately.
  • S.1.2.2 Bound of a prediction error measure: The GLM prediction measure is connected to the symmetrized Bregman divergence, and reduces to the linear-model form when ψ(u)=u^2/2.
  • S.1.2.2 Bound of a prediction error measure: The prediction-error bounds improve on the classical target-only Lasso bound when n0 is small and nAh is much larger than n0.
  • S.1.2.2 Bound of a prediction error measure: The paper provides prediction-error bounds for Ah-Trans-GLM both with and without Assumption 4 under conditions corresponding to earlier theorems.
  • S.1.3.2 Transfer learning on Ah: In simulations, KAh-Trans-GLM's estimation error declines as K increases and becomes lower than target-only naїve-Lasso's error.
  • S.1.3.3 Transferable source detection: The detection-based Trans-GLM generally matches Ah-Trans-GLM and achieves the best performance, but larger h can cause missed or wrongly included sources.
  • S.1.3.3 Transferable source detection: More cross-validation folds may improve detection accuracy but increase computation, motivating moderate fold numbers such as 3 or 5.
Loading 2105.14328v4…