Source-linked AI summary
Learn on Source, Refine on Target:A Model Transfer Learning Framework with Random Forests
Noam Segev, Maayan Harel, Shie Mannor, Koby Crammer, Ran El-Yaniv
TL;DR
The paper addresses adapting a source model to a target domain when target training data are limited and source data are unavailable for adaptation. It locally adjusts decision forests and combines two transfer methods in a union ensemble; the authors report that the resulting methods often outperform competing transfer approaches and can exceed some instance-transfer methods.
Problem
Model transfer adapts a source model using limited target training data when source examples are unavailable during adaptation, a restrictive setting compared with frameworks that can use source data.
Source: §1 · p. 1Method
The paper adapts source-trained decision forests through local tree adjustments and combines the resulting forests in a majority-vote union ensemble.
Source: §1 · p. 1, §3.1 · p. 3, §3.3 · p. 6Results
The combined method often outperforms its constituent methods and is reported to outperform leading model-transfer algorithms while competing with instance-transfer algorithms and outperforming some of them.
Source: §1 · p. 2, §7 · p. 11Takeaways & Limitations
The union algorithm is described as easy to implement, requiring modest hyper-parameter tuning, and able to exploit intensive computation on large-scale problems.
Source: §1 · p. 2The work does not address sample complexity or formal generalization ability, and a comprehensive learning theory may depend on formally defining source-target relations.
Source: §7 · p. 11
Abstract
from arXiv · showhide
We propose novel model transfer-learning methods that refine a decision forest model M learned within a "source" domain using a training set sampled from a "target" domain, assumed to be a variation of the source. We present two random forest transfer algorithms. The first algorithm searches greedily for locally optimal modifications of each tree structure by trying to locally expand or reduce the tree around individual nodes. The second algorithm does not modify structure, but only the parameter (thresholds) associated with decision nodes. We also propose to combine both methods by considering an ensemble that contains the union of the two forests. The proposed methods exhibit impressive experimental results over a range of problems.
1 INTRODUCTION
Model transfer adapts a source-trained model to a target context using relatively little target data when the source data cannot be shared. The paper proposes locally adapting decision forests and combining two methods whose strengths differ, reporting strong results across problems.
1 INTRODUCTION
- The target adaptation problem arises when consumers need source-trained models for context-specific domains but cannot access the source data or afford retraining on both domains.The paper also describes model transfer as a restrictive setting because source examples are unavailable during adaptation.Source: §1 · p. 1
- Existing model-transfer techniques often regularize target learning with the source model, but their global focus can limit accommodation of local distribution changes.The paper cites biased regularizers and aggregated source-target predictors as examples.Source: §1 · p. 1
- The proposed decision-tree methods use local, greedy model transformations to address feature-distribution shifts and changes that refine or coarsen the relevant features.The methods are designed around these two commonly occurring source-to-target transformation types.Source: §1 · pp. 1–2
- The union ensemble often performs better than, or almost as well as, either constituent and improves state-of-the-art performance across a range of problems.The authors describe it as easy to implement, requiring modest hyper-parameter tuning, and able to exploit intensive computation for large-scale problems.Source: §1 · p. 2
- Experiments across multiple datasets indicate that the method often outperforms several state-of-the-art transfer-learning methods.The paper also reports synthetic examples illustrating the algorithms’ strengths and weaknesses.Source: §1 · p. 2
2 PRELIMINARY DEFINITIONS
The framework formulates supervised model transfer from a source prediction function and limited target training data, then specifies a shared-feature, shared-label setting. It uses random-forest trees with node and threshold parameters, and outlines SER’s recursive expansion and reduction operations.
2 PRELIMINARY DEFINITIONS
- Supervised model transfer seeks a target predictor with low target risk using a source prediction function and typically limited target training samples.The setting assumes inductive transfer with shared feature and label spaces, allows feature marginals to differ, and covers binary and multiclass classification with zero-one loss.Source: §2 · p. 2
- SER takes a node and its labeled samples, builds a tree at leaves, recurses through children, and prunes when leaf empirical error is lower than subtree error.Leaf decisions in the modified tree use the target empirical distribution.Source: §2 · p. 2
2.1 Random Forests Models:
- A random-forest tree represents leaves with decision values and internal nodes with features and, for numeric features, thresholds.Samples are classified by the leaf reached along their path through the tree.Source: §2.1 · p. 2
3 ALGORITHMS
The paper presents SER, which locally expands or reduces tree structures, and STRUT, which refits thresholds while retaining tree structure; MIX combines both forests. Synthetic examples illustrate when each method fits different source-to-target transformations and how combining them performs.
3.1 Structure Expansion/Reduction
- SER expands source leaves on target data, then prunes a subtree when its empirical error exceeds the error of replacing it with a leaf.The modified leaves use decision values from the target empirical distribution.Source: §3.1 · p. 3
3.1.1 Visual Illustration of the SER Algorithm
- The box illustration shows expansion transferring a tree to a more refined domain and reduction transferring it in the reverse direction.The paper presents these as small structural changes that can preserve similar portions of the source and target models.Source: §3.1.1 · p. 3
3.1.2 Logical Regularization in SER
- SER expands before reducing so each transformed rule remains a specialization or generalization of a source rule; reversing the order can let the model drift farther from the source.The stated rule relation is that either the transformed rule satisfies the source rule or the source rule satisfies the transformed rule.Source: §3.1.2 · pp. 3–4
3.2 Structure Transfer
- STRUT refits numeric thresholds top-down on target data, prunes nodes unreachable there, and can swap child subtrees when reversing their original distributions improves DG.The tree structure is otherwise retained, and leaf decisions are computed from target training data.Source: §3.2 · pp. 4–5
- STRUT selects informative thresholds using IG local maxima while favoring thresholds whose induced label distributions remain similar to the original distributions under DG.The resulting nonconvex optimization is solved by line search over a limited set of thresholds.Source: §3.2 · p. 4
3.2.1 On the Use of IG and DG
- In the respective examples, restricted STRUT using only IG yields a decision stump with ∼30% error and using only DG yields ∼10% error; unrestricted STRUT using both measures achieves 100% accuracy in both.Source: §3.2.1 · p. 5
3.3 A MIX Approach
- MIX applies majority voting over the union of the SER and STRUT forests, whose different strengths can make it outperform or nearly match either constituent.The paper proposes the union because the methods tend to excel on different problems.Source: §3.3 · pp. 5–6
3.4 Numerical Examples and Intuition
- Across 1,000-trial synthetic examples, MIX is a close runner-up on moving source and beats both methods on mixed boxes; SER excels without feature correspondence, while STRUT can excel when it is maintained.The examples define moving source as a shift retaining concept geometry and mixed boxes as a target refinement of the source concept.Source: §3.4 · p. 6
4 EMPIRICAL EVALUATION
The evaluation tests SER, STRUT, and MIX on varied source–target datasets, against simple baselines and competing transfer methods. MIX is often best or close to best, including in comparisons with instance-transfer methods.
4.1 Datasets
- The datasets use meaningful domain splits or constructed transformations, including foliage-to-barren landmines, user-pair activity data, species-separated mushrooms, and font-separated letters.The mushroom split uses stalk shape; the letter split thresholds x2bar at its median for each letter.Source: §4.1 · p. 7
4.2 Experiments and Results
- Experiments generally used all source data and 5% of target samples for training, with 50-tree forests; imbalanced landmine and activity tasks used balanced error rate.The remaining target samples were used for testing, and class ratios in training were kept similar to the full target set.Source: §4.2 · pp. 7–8
- MIX was best or close to best among the evaluated methods, always beat SrcOnly, and matched or beat TgtOnly in many cases.It also consistently beat relabeling and bias, and beat pruning except on wine and landmines; comparisons with relabeling, bias, and pruning were significant at p-value < 0.005.Source: Table 4 · p. 8, §4.2 · p. 8
- On the letter task, MIX transfer took 3.1s serially and 0.31s on a 10-core machine; trees can be processed independently for parallelism.The reported serial times were 1.6s for consensus regularization and 11s for ASVM.Source: §4.2 · p. 8, Table 5 · p. 9
4.3 Comparing to Instance Transfer Algorithms
- MIX often matched FEDA and Mixed-Entropy and consistently outperformed TradaBoost and TrBagg, despite the comparison favoring instance transfer.The TradaBoost and TrBagg advantages were statistically significant (p-values < 0.01); no significant difference was detected between MIX and FEDA or Mixed-Entropy.Source: Table 6 · p. 9, §4.3 · p. 9
5 CAN WE EXPLAIN THE ADVANTAGE OF MIX?
The paper links MIX’s advantage to diversity between its SER- and STRUT-transformed trees: their differing structures and thresholds are expected to reduce correlation. In the mixed-box experiment, MIX has higher disagreement and a tighter Bayes-risk bound than either constituent.
5 CAN WE EXPLAIN THE ADVANTAGE OF MIX?
- SER and STRUT transformations are expected to create lower-correlated tree pairs because SER changes tree size while STRUT mainly changes thresholds.The authors attribute MIX’s ability to perform well when only one constituent does well, and sometimes outperform both, to diversity and correlation among ensemble members.Source: §5 · p. 9
- 0.07 is the MIX Bayes-risk bound, versus 0.14 for STRUT and 0.13 for SER, alongside empirical Gibbs risks of 0.17 for STRUT and 0.15 for SER.The authors report that MIX’s bound is much tighter and its actual test results are also better than those of its constituents.Source: §5 · p. 10
- MIX’s empirical joint error is similar to SER’s and STRUT’s, but its disagreement is much higher, yielding a lower CQ bound.The paper describes CQ as a bound related to expected joint error.Source: §5 · p. 10
- When SER and STRUT disagree, the mixed-box margin CDFs for MIX and SER are lower at the origin than STRUT’s.The paper connects better empirical margin profiles with better generalization through the cited theorem.Source: §5 · p. 10, Figure 5 · p. 10
- The authors argue that weak correlation lets MIX benefit from correctly performing constituents even when the ensemble is erroneous.They also state that, under the likely non-correlation condition for paired trees, the Bayes-risk bound for MIX is nearly halved as n doubles.Source: §5 · p. 10
6 RELATED WORK
The related-work section places model transfer among several transfer-learning paradigms and reviews prior approaches within each. It then distinguishes decision-tree transfer from stream adaptation and multi-task learning, noting limited prior work in the paper’s model-transfer setting.
6 RELATED WORK
- Transfer learning includes model, instance, feature, domain-adaptation, multi-task, and semi-supervised settings, distinguished by their assumptions about source data, features, domains, or tasks.Model transfer adapts a source predictor using target training data, including when source examples cannot be retained or distributed.Source: §6 · pp. 10–11
- Prior model-transfer methods use biased regularizers, multiple source-target predictors, parameter-transfer priors, or feature-weight estimation.The section also notes that model-transfer methods are effective when a similar inductive bias works well across related tasks.Source: §6 · pp. 10–11
6.1 Transfer Learning using Decision Trees
- Decision-tree transfer was underexplored, and the cited tree-update method expanded trees iteratively without refitting numeric thresholds.Stream-adaptation methods instead handle incrementally changing distributions, retain original samples, and continually modify trees; multi-task work includes boosting and task-driven splits.Source: §6.1 · p. 11
7 CONCLUDING REMARKS
The paper proposes adapting source-trained decision forests through local changes to tree parameters or architecture, and reports that MIX often outperforms its constituent algorithms and leading model-transfer methods. It identifies formalizing source-target transformations and establishing sample complexity and generalization ability as open issues.
7 CONCLUDING REMARKS
- The two base algorithms adjust tree parameters and/or architecture, while MIX combines them and often outperforms its constituents, leading model-transfer algorithms, and some instance-transfer algorithms.The paper reports that performance on synthetic data varies with the transformation type.Source: §7 · p. 11
- Source data can be discarded after training, allowing model transfer to a target to be computed later when needed.The authors suggest that a bank of models trained on varied source domains could be used to construct models for related target domains.Source: §7 · p. 11
- The paper leaves open a systematic account of how source-target transformations relate to tree adaptation, as well as sample complexity and formal generalization ability.The authors note that a comprehensive learning theory may depend on formally defining an effective model for possible source-target relations.Source: §7 · p. 11
APPENDIX A OTHER NUMERICAL EXAMPLES ON SYNTHETIC DATA
The appendix defines ten synthetic source-target transformation challenges and reports test-error results for STRUT, SER, and MIX. The challenges use binary concepts over numeric features and share a repeated evaluation protocol.
APPENDIX A OTHER NUMERICAL EXAMPLES ON SYNTHETIC DATA
- The appendix presents ten synthetic problems designed to capture source-target transformations, with each challenge defined by a binary concept and a transformation.The concepts and transformations are specified for each challenge.Source: Appendix A · p. 12
- Each challenge is repeated 1,000 times, with test error averaged over 10,000 random target-domain points; Table 7 presents results for STRUT, SER, and MIX.The experiments use numeric features and a uniform marginal distribution over the input space.Source: Appendix A · p. 12, Table 7 · p. 12
- The transformations include box mixtures, inversion, displacement, size changes, axis swaps, label noise, rotation, a fish-eye mapping, and a refined sine boundary.The appendix describes source/target mix, source inversion, moving, expanding and shrinking source, axis-swap, noisy target and source, rotated source, fish-eye, and refined sine-boundary challenges.Source: Appendix A · p. 12