Source-linked AI summary
Hypothesis Transfer Learning via Transformation Functions
Simon Shaolei Du, Jayanth Koushik, Aarti Singh, Barnabas Poczos
TL;DR
The paper addresses limited HTL theory on excess risk and the restriction to particular transfer forms. It introduces a transformation-function framework and reports faster excess-risk convergence for kernel methods on related domains.
Problem
Existing HTL analyses often focus on specific algorithms or generalization bounds rather than excess risk, leaving the risk behavior of other transfer functions open.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
Small target-domain validation samples can avoid HTL when it is not useful and select a transformation function from a pool.
Source: §1 · p. 2The paper presents upper bounds only; lower bounds for HTL and other transfer-learning settings remain future work.
Source: §7 · p. 9
Abstract
from arXiv · showhide
We consider the Hypothesis Transfer Learning (HTL) problem where one incorporates a hypothesis trained on the source domain into the learning procedure of the target domain. Existing theoretical analysis either only studies specific algorithms or only presents upper bounds on the generalization error but not on the excess risk. In this paper, we propose a unified algorithm-dependent framework for HTL through a novel notion of transformation function, which characterizes the relation between the source and the target domains. We conduct a general risk analysis of this framework and in particular, we show for the first time, if two domains are related, HTL enjoys faster convergence rates of excess risks for Kernel Smoothing and Kernel Ridge Regression than those of the classical non-transfer learning settings. Experiments on real world data demonstrate the effectiveness of our framework.
1 Introduction
The paper studies regression HTL when source and target functions differ but are related, and proposes a transformation-function framework to represent that relation. It analyzes excess risk and reports faster convergence rates for kernel methods when domains are related.
1 Introduction
- Source data are typically abundant while target data are relatively scarce, and the domains may be related without being identical.The regression setting considered here has different but related source and target functions.Source: §1 · p. 1
- Existing HTL theory often studies particular transfer forms or bounds generalization error rather than excess risk, which may not track true risk.The cited offset-transfer analysis estimates the residual between target and source functions.Source: §1 · pp. 1–2
- The framework treats a transformation function as an algorithm input that captures source–target relevance, unifies prior work, and induces new learning procedures.The paper names Wang and Schneider [2014], Kuzborskij and Orabona [2013], and Wang et al. [2016] as unified prior works.Source: §1 · p. 2
- For stable learning subroutines, source-domain estimation error has limited effect on target estimation, and small target validation samples can reject unhelpful transfer or select a transformation.The analysis gives explicit risk bounds for Kernel Smoothing and Kernel Ridge Regression.Source: §1 · p. 2
- The paper reports faster excess-risk convergence for Kernel Smoothing and Kernel Ridge Regression than non-transfer learning when the domains are related.It also reports conducting experiments on real-world data.Source: §1 · p. 2
2 Preliminaries
The preliminaries set up source and target regression samples, introduce Kernel Smoothing and Kernel Ridge Regression, and review related transfer-learning analyses.
2.1 Problem Setup
- The setup uses i.i.d. source and target samples, with additional target-domain samples for cross-validation; inputs, outputs, and noise are bounded, and the noise has zero mean.The regression functions model source and target responses separately.Source: §2.1 · p. 2
2.2 Kernel Smoothing
- Kernel Smoothing assumes a Hölder-smooth function and uses a positive kernel supported on [0, 1] that is highest at zero and decreases with distance.The cited smoothness condition bounds function differences by a distance raised to the Hölder exponent.Source: §2.2 · p. 3
2.3 Kernel Ridge Regression
- Kernel Ridge Regression uses a bounded positive-semidefinite kernel to define an RKHS and predicts from kernel evaluations with a regularized inverse.The paper gives the estimator in terms of training inputs, labels, and the kernel matrix.Source: §2.3 · p. 3
2.4 Related work
- Related work includes discrepancy-based bounds with unlabeled target data and methods using labeled target data; Wang et al. [2016] is a special case with G(a, b) = a + b.The cited related work also covers conditional matching and excess-risk bounds.Source: §2.4 · pp. 3–4
3 Transformation Functions
The paper represents source–target regression relations with transformation functions and estimates the target by learning an associated auxiliary function.
3 Transformation Functions
- Offset transfer makes a difficult target function easier to estimate when the source estimate is available and the residual f_ta − f_so is simpler.The example estimates the residual from target samples and adds it to the source estimate.Source: §3 · p. 4
- A transformation function G maps a source regression value and an auxiliary value to the target regression value, with G invertible in its second input.The auxiliary function is the inverse-transform value that combines with the source function to produce the target function.Source: §3 · p. 4
- Scale transfer models f_ta as a multiple of f_so, while non-transfer sets the target equal to the auxiliary function and ignores source data.For scale transfer, the auxiliary function is the constant multiplier when f_ta = c f_so.Source: §3 · p. 4
3.1 A Meta Algorithm
- The meta-algorithm trains a source estimator, uses target samples to learn the auxiliary function, then applies G to produce the target estimate.The auxiliary-function learner uses newly constructed data based on the source estimate and target samples.Source: §3.1 · pp. 4–5
- When G is nonlinear in its second input and target noise is present, estimating the auxiliary function can become a measurement-error problem without a universal estimator.The paper points to regression calibration as one common technique for this case.Source: §3.1 · p. 5
4 Excess Risk Analyses
The excess-risk analysis links estimation error in the target to stability of the learner for the auxiliary function. Under stated conditions, Kernel Smoothing and Kernel Ridge Regression can achieve improved rates when the auxiliary function and sample sizes meet method-specific conditions.
4 Excess Risk Analyses
- Theorem 1 bounds how source-function estimation error propagates to the target; with the true source function known, the bound depends on estimating wG, while the additional source-error contribution can grow with n_ta.That contribution also depends on source-estimation error, which is relatively small when source-domain data are abundant.Source: §4 · p. 6
4.1 Kernel Smoothing
- For Kernel Smoothing, a better statistical rate follows when α_wG > α_f_ta, under the theorem’s support, density, and Hölder conditions.The risk has contributions from estimating f_so and wG; the source-estimation term is relatively small in the typical setting n_so >> n_ta.Source: §4.1 · p. 6
4.2 Kernel Ridge Regression
- The KRR estimation error has contributions from estimating f_so and wG; the latter depends on the induced RKHS and how far wG is from that space.The error in estimating f_so is amplified by a factor described in the text as worse than for nonparametric Kernel Smoothing.Source: §4.2 · p. 7
- For Kernel Ridge Regression, the analysis gives an improved convergence rate when β_wG ≥ β_ta and n_so is sufficiently large.The theorem assumes equal source and target input distributions and eigenvalue decay conditions.Source: §4.2 · pp. 6–7
- Theorem 2 and Theorem 3 are not directly comparable because they assume different function spaces.The stated distinction is that Hölder space is a Banach space but not generally a Hilbert space.Source: §4.2 · p. 7
5 Finding the Best Transformation Function
The section discretizes a class of transformation functions and uses validation to select among candidates. The selection bound can preserve no-transfer performance when the class includes the no-transfer function.
5 Finding the Best Transformation Function
- A finite ϵ-cover approximates a potentially uncountable transformation class, illustrated by quantizing G(a,b)=αa+b into a grid.The example bounds α and a; the grid uses increments of ϵ.Source: §5 · p. 7
- Theorem 4 selects the transformation whose estimator has the lowest empirical risk on the validation set, under its stated assumptions.It assumes the transformation class has an ϵ-cover and that w_G varies Lipschitz-continuously with G.Source: §5 · p. 7
- If the candidate class includes the no-transfer function, the selected transformation has excess risk of the same order as no-transfer learning, avoiding negative transfer.Source: §5 · p. 7
- The validation-set size depends only logarithmically on the number of candidate transformations, so cross-validation can use little target-domain data.Source: §5 · p. 7
6 Experiments
Experiments on robotics and fMRI data compare transfer methods with source-only, target-only, and combined-data baselines. Transfer methods perform especially well when target data are scarce in the robotics experiment, while fMRI results favor offset methods for consistency.
6 Experiments
- On kin-8fm-to-kin-8nh, the proposed methods outperform other methods at nearly all target sample sizes, especially when the target set is small.When target and source sample sizes are equal, training only on target performs best; offset methods are somewhat better than scale methods in this experiment.Source: Table 1 · p. 8, §6 · p. 8
- The comparison includes target-only, source-only, and combined-data baselines, CDM, and the paper’s scale and offset transfer methods.Source: §6 · p. 8
- The fMRI experiment uses all but one subject as source, trains with 300 source and 80 target points, and evaluates on the remaining 35 target points.The dataset contains 28 subjects, and the experiment is repeated five times per algorithm.Source: §6 · p. 8
- On the fMRI task, both offset methods have low-variance R-squared results, while Only Target KRR has a positive mean but high variance.Scale methods perform less well than offset methods in this case.Source: Table 2 · p. 9, §6 · p. 9
7 Conclusion and Future Works
The paper proposes a general HTL regression framework for settings with some target-domain data and argues that transfer can achieve better statistical rates than standard supervised learning. It identifies unlabeled target data and lower bounds as future directions.
7 Conclusion and Future Works
- The proposed framework addresses HTL regression when some target-domain data are available.Source: §7 · p. 9
- Theoretical analysis shows that transfer learning can achieve better statistical rates than standard supervised learning.Source: §7 · p. 9
- Future work includes combining the framework with methods that use unlabeled target data and deriving lower bounds for transfer learning.Source: §7 · p. 9
A.1 Proof of Theorem 1
The appendix sketches proofs for the main results by applying transformation and stability assumptions, using kernel-smoothing bounds, and analyzing kernel-ridge-regression stability.
A.1 Proof of Theorem 1
- Theorem 1’s proof uses assumptions on the transformation function together with stability of the training algorithm.Source: §A.1 · p. 12
A.2 Proof of Theorem 2
- Theorem 2’s kernel-smoothing proof uses supporting lemmas to bound terms in Theorem 1 before integrating over the target input distribution.Source: §A.2 · pp. 12–13
A.3 Proof of Theorem 3
- Theorem 3’s proof bounds its terms separately and analyzes the stability of kernel ridge regression.Source: §A.3 · pp. 13–14
A.4 Proof of Theorem 4
The appendix proves a cross-validation result using Bernstein’s inequality, then completes Theorem 4’s risk argument with a finite-cover approximation and a decomposition of the error.
A.4 Proof of Theorem 4
- The cross-validation result is included as a standard result, with its proof supplied for completeness.The proof applies a Bernstein-type inequality to obtain the stated probability bound.Source: §A.4 · pp. 14–15
- Theorem 5 considers the hypothesis minimizing cross-validation error and gives a high-probability bound, proved using Bernstein’s inequality.The proof bounds the variance of the relevant quantity; it also invokes boundedness of the response domain to establish a moment condition.Source: §A.4 · p. 15, var · p. 15
var
- Theorem 4’s proof approximates G⋆ by G′ from an ϵ-cover, then bounds three terms in the resulting risk decomposition.The cover error is controlled by ϵ, and the argument uses a Lipschitz assumption on the transformation-function class.Source: var · p. 16
B Regression Calibration for Measurement Error Problem
This appendix adapts regression calibration to the HTL setting: estimate the target regression function, then calibrate that estimate using a function tied to the transformation and noise distribution.
B Regression Calibration for Measurement Error Problem
- Regression calibration estimates the target regression function and then applies a calibration function F to that estimate.The procedure is adapted to HTL; directly using the observed target response is one option for the initial estimate.Source: B Regression Calibration for… · p. 17
- The calibration function F depends on the chosen transformation G and the noise distribution.The appendix illustrates the procedure with a loglinear mean model.Source: B Regression Calibration for… · p. 17
- Regression calibration is one method for measurement error; alternatives such as simulation extrapolation and likelihood may apply, depending on G and the noise assumptions.Distribution-specific parameters may be replaced by estimates when they are unknown.Source: B Regression Calibration for… · p. 17
C.1 Synthetic data
The synthetic examples report better target-function estimates with suitable transformation functions, especially in nonsmooth regions; on the kin-8nh-to-kin-8fm task, transfer adds little except at the smallest target sample size.
C.1 Synthetic data
- The synthetic experiments use 10,000 source samples and 100 target samples, with Gaussian label noise of variance 0.01 in both domains.Kernel Smoothing with a Gaussian kernel estimates f so and wG.Source: §C.1 · p. 18
- With G(a, b) = a + b for the offset example and G(a, b) = ab for the scale example, transfer estimates the target function better, especially where it is not smooth.The offset target is f ta(x) = f so(x) + x; the scale target is f ta(x) = 5f so(x).Source: §C.1 · p. 18
C.2 Transferring from kin-8nh to kin-8fm
- Transfer from kin-8nh to kin-8fm yields little advantage except for a minimal gain at the smallest nta, and does not negatively affect performance.When smoothness conditions are unknown, the authors recommend using cross-validation to decide whether to use source data.Source: §C.2 · p. 18