PaperlayerView exact revision on arXiv

Source-linked AI summary

MotherTree: Meta-learning on synthetic data improves decision tree training

Ziyuan Wang, Fredrik D. Johansson

arXiv:2610.10832v1cs.LG

TL;DR

Decision trees are inspectable but usually trained task by task, while tabular foundation models can learn from synthetic priors without producing a standalone task model. MotherTree meta-learns to generate a hard, axis-aligned tree from new-task training data, and its tuned trees outperform from-scratch gradient-based learning while remaining competitive with other tree learners.

  • Problem

    Decision trees are usually trained one task at a time, while tabular foundation models do not produce reusable standalone task models and scalable, efficient meta-learning of trees remains open.

    Source: §1 · p. 1

  • Method

    MotherTree meta-learns a mapping from labeled training sets to standalone hard, axis-aligned decision trees, using a tree-structured decoder pre-trained on a synthetic prior without reference-tree supervision.

    Source: §1 · p. 2, §3 · p. 2, §3.1 · p. 3

  • Results

    MotherTree consistently outperforms the corresponding from-scratch GradTree learner, with AUC gains of 0.069 on TabArena and 0.077 on OpenML CC-18 at B = 100; tuning makes it competitive with four baseline classes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Synthetic-prior meta-learning can provide effective inductive biases for learning standalone small decision tree classifiers.

    Source: §1 · p. 2

    MotherTree generates complete binary trees with fixed topology and depth, so it cannot adapt tree size or topology to individual tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

Conventional decision tree algorithms produce effective, transparent models that can be audited, communicated, and deployed independently of the training data, but require learning every new task from scratch. In contrast, tabular foundation models demonstrate that meta-learning from a synthetic prior distribution enables strong in-context prediction for previously unseen tasks, especially in small-sample regimes. However, this approach does not produce a standalone model that can be inspected in isolation. We introduce MotherTree, a tabular transformer that meta-learns decision tree induction: given a training set for a new task, it outputs a hard, axis-aligned decision tree, equivalent in form to classically trained trees, in a single forward pass. MotherTree is pre-trained on a synthetic prior using stochastic gradient descent without requiring reference trees for supervision. On established benchmarks with controlled sample size, the approach is competitive with size-matched trees from common algorithms: recursive partitioning, gradient-based tree learning, globally optimal trees, and distillation from tabular foundation models. Notably, MotherTree consistently improves over from-scratch gradient-based learning and acts as a strong initializer: task-specific tuning of the generated tree outperforms the corresponding from-scratch learner on all benchmarks and sample sizes. These results show that meta-learning can provide effective inductive biases for learning stand-alone, small decision tree classifiers.

1 INTRODUCTION

The introduction asks how synthetic-prior meta-learning can benefit decision tree induction and presents MotherTree as a way to generate standalone trees from new-task data. It reports gains over from-scratch gradient-based learning and stronger performance after task-specific tuning.

1 INTRODUCTION

  • Tabular foundation models show that synthetic-prior meta-learning can support small-sample prediction, but in-context learning does not yield a reusable standalone task model.Classical learners, including tree-based models, remain competitive in small samples and preferable in large-sample regimes.Source: §1 · p. 1
  • Tree induction remains an open meta-learning problem: conventional methods train trees from scratch, while earlier meta-learned fitting relied on gold-standard reference trees.Greedy methods can overfit small samples, optimal-tree methods are computationally intensive, and gradient-based methods lack guarantees of optimality.Source: §1 · pp. 1–2
  • MotherTree uses a tree-structured decoder to generate standalone hard, axis-aligned decision trees in one forward pass, without reference-tree supervision.It is pre-trained on datasets from a diverse synthetic prior and generalizes to real-world tasks with different feature sets.Source: §1 · p. 2
  • At B = 100, MotherTree exceeds GradTree AUC by 0.069 on TabArena and 0.077 on OpenML CC-18.The evaluation covers 64 benchmark classification tasks in controlled small-sample regimes.Source: §1 · p. 2
  • Task-specific tuning adds gains that grow with training-set size and makes MotherTree competitive with four baseline classes of tree learners.Synthetic parity experiments also suggest its utility includes improved initialization for local optimization.Source: §1 · p. 2

2 RELATED WORK

Related work distinguishes hard, soft, and oblique trees and reviews conventional tree learners that fit each task from scratch. Prior amortized and meta-learning approaches differ in their inputs and supervision, with MotherTree learning from instance labels rather than reference trees.

2 RELATED WORK

  • Hard trees use discrete rules and deterministic routing, soft trees use differentiable fuzzy or probabilistic routing, and oblique trees split on feature combinations rather than one feature and threshold.Soft trees require post-hoc rounding to form hard, axis-aligned trees [Pace et al., 2022; Tanno et al., 2019].Source: §2 · p. 2
  • Greedy recursive partitioning and global discrete optimization are common hard-tree strategies, while gradient-descent tree methods are sensitive to initialization and fit trees from scratch.These strategies are described in the paper with markers, [Hu et al., 2019], and [Tanno et al., 2019; Yang et al., 2018; Li et al., 2022], respectively.Source: §2 · p. 2
  • Earlier tree-related approaches include metadata-only generation, distillation, and MetaTree, which uses reference-tree supervision; MotherTree instead learns directly from instance labels.MetaTree requires high-quality ground-truth trees for pre-training tasks [Zhuang et al., 2024].Source: §2 · p. 2

3 MOTHERTREE: META-LEARNED DECISION TREE TRAINING

MotherTree meta-learns a mapping from labeled training sets to hard, axis-aligned trees that can be used as standalone models. A dataset encoder and tree-structured decoder generate the tree, which can then be tuned for a new task.

3 MOTHERTREE: META-LEARNED DECISION TREE TRAINING

  • MotherTree uses a tabular data encoder and tree-structured decoder to generate a hard, axis-aligned decision tree from a new task’s labeled data in one forward pass.The generated tree supports stand-alone inference.Source: §3 · pp. 2–3, Figure 1 · p. 3

3.1 META-LEARNING DECISION TREE PARAMETERS

  • For a new task, the generator maps its labeled training set to tree parameters, producing deterministic class-probability predictions that require neither the generator nor training data at inference.The tree has a fixed complete-binary topology with a specified depth; internal nodes select features and thresholds, and leaves store class values.Source: §3.1 · p. 3

3.2 TABULAR DATA SET ENCODER

  • The encoder forms row representations, aggregates them by class into a concatenated dataset embedding, and supplies that embedding to the tree decoder.The encoder’s default row representations use the MotherNet transformer encoder.Source: §3.2 · pp. 3–4

3.3 ANCESTRAL DECODER FOR AXIS-ALIGNED TREES

  • A tree-aware transformer maps the dataset embedding to node outputs and masks attention to each node and its ancestors, conditioning nodes on ancestor splits while blocking flow from unrelated branches.Ancestral decoding performs substantially better than an MLP decoder in ablations.Source: §3.3 · p. 4

3.4 PRE-TRAINING, TUNING, AND INFERENCE

  • The encoder-decoder is trained end-to-end on synthetic structural-causal-model tasks using instance-label supervision and straight-through gradients for discrete feature selection and routing.Training does not require reference-tree supervision; for a new task, generated leaves can be recalibrated or all tree parameters fine-tuned.Source: §3.4 · pp. 4–5

4 EXPERIMENTS

The experiments compare MotherTree with tree learners across controlled sample budgets and benchmarks, then use synthetic parity tasks to probe tree optimization with distractors. MotherTree-Tune remains competitive with other tree-learning approaches and is especially robust when a tight depth budget forces rejection of misleading features.

4.1 EVALUATION ON TABULAR BENCHMARKS

  • MotherTree-Tune remains competitive across benchmarks, budgets, and label types, consistently outperforming GradTree and GradTree←TabPFN; at B = 100, MotherTree and MotherTree-Tune achieve the best results on both benchmarks.The mean-AUC figure varies the training-set budget, while the critical-difference diagram compares average ranks across both benchmarks.Source: Figure 2 · p. 6, §4.1 · pp. 6–7, Figure 3 · p. 7
  • MotherTree outperforms GradTree on both benchmarks through B = 800, with comparable performance on OpenML-CC18 at B = 1000; its advantage narrows as training data grows.The reported B = 200 binary-task mean AUC rises from 0.635 to 0.692 on TabArena and from 0.725 to 0.766 on OpenML-CC18-Small.Source: §4.1 · p. 7
  • Task-specific tuning generally adds to MotherTree’s initialization advantage, with larger gains on multiclass tasks and consistent wins over GradTree and GradTree←TabPFN.At B = 200, tuning raises multiclass AUC from 0.698 to 0.757 on TabArena and from 0.764 to 0.804 on OpenML-CC18.Source: §4.1 · p. 7
  • MotherTree transfers a structural prior conditioned on the observed training context, unlike distillation from TabPFN predictions, reference-tree meta-learning, or trees built from column semantics..Source: §4.1 · p. 7
  • The tree-structured Transformer decoder provides a stronger initialization than the MotherNet-style MLP decoder, especially after tuning; the latter’s advantage diminishes as training data increases.The ablation compares changes in mean ROC-AUC relative to GradTree across context sizes on OpenML-CC18-Small and TabArena.Source: §4.1 · p. 8, Figure 4 · p. 8

4.2 INTROSPECTION ON SYNTHETIC PARITY (XOR) TASKS

  • The parity tasks use labels defined by joint interactions among parity bits and append individually informative noisy copies of the label as distractors.The setup tests feature selection because greedy splitting is non-optimal by design.Source: §4.2 · p. 8
  • Even without distractors, gradient-based trees have a gap to the DL85 oracle and CART on larger parity orders, reflecting optimization difficulty rather than feature-selection error.The authors focus on the additional degradation caused by distractors rather than treating the absolute oracle gap as feature-selection error.Source: §4.2 · p. 8
  • When λ = m − (D − k) > 0 exhausts the depth budget, MotherTree-Tune degrades more gradually and retains more attainable improvement across distractor strengths.At least λ distractors must be refused to learn an optimal tree; the results suggest meta-learning provides a stronger initialization for resisting locally predictive but structurally sub-optimal features.Source: §4.2 · pp. 8–9, Figure 5 · p. 9

5 DISCUSSION

MotherTree’s fixed tree structure limits how well tree size can match individual tasks. The discussion also notes that single-tree classifiers have lower predictive performance than ensembles.

5 DISCUSSION

  • Figure 6 compares the number of uninformative paths, where lower is better, and leaf utilization, where higher is better.Source: Figure 6 · p. 9
  • MotherTree generates complete binary trees with fixed topology and depth, so task-specific trees cannot adapt their size or structure.This can leave unnecessary nodes in some trees or too little capacity for more complex tasks.Source: §5 · p. 9
  • Tree ensembles universally achieve stronger predictive performance than single-tree classifiers by combining predictions from complementary trees.Extending amortized generation to ensembles raises questions about coordinating tree diversity and dependency structures to reduce redundancy and improve coverage.Source: §5 · p. 10

A.1 ARCHITECTURE DETAILS

The architecture encodes context rows into a dataset representation, then uses a tree-structured transformer decoder to generate tree parameters. The encoder aggregates row representations by class before decoding.

A.1 ARCHITECTURE DETAILS

  • The encoder uses the MotherNet design, embedding each table row with a linear layer and one-hot encoding classes before transformer blocks.Unlike TabPFN and TabICL, it attends only to context data and does not require test points to participate in attention.Source: §A.1 · p. 13
  • The encoder projects each context row and label into a d′-dimensional embedding, combines them, and applies 12 inter-sample attention layers.It class-wise mean-pools row representations and concatenates the k class means into a fixed-dimensional dataset embedding for the decoder.Source: §A.1 · p. 13
  • After 8 Transformer layers, decoder representations are projected to split and leaf parameters; the decoder has 23.6M parameters.The decoder uses dn = 32 and was sized to remain comparable to the original MotherNet decoder’s 17.7M parameters.Source: §A.1 · p. 13

A.2 COMPARISON WITH TABICL ENCODER

Replacing the MotherNet encoder with TabICLv2 does not improve performance across evaluated context sizes. The comparison reports mean AUC for different encoders on TabArena.

A.2 COMPARISON WITH TABICL ENCODER

  • The comparison evaluates encoders by mean AUC on TabArena.Source: Table 2 · p. 13
  • Replacing the MotherNet encoder with TabICLv2 does not improve performance across evaluated context sizes; the MotherNet encoder performs slightly better in all settings.The results suggest performance is not driven by a stronger tabular prediction backbone, and greater encoder sophistication does not necessarily improve tree-generation representations.Source: §A.2 · p. 13, Table 2 · p. 13

A.3 TRAINING DETAILS

MotherTree is trained in two stages using TabICL and TabICLv2 synthetic priors. Adding TabICL-prior pre-training substantially improves validation AUC over TabICLv2-only training and reduces training time.

A.3 TRAINING DETAILS

  • TabICL’s prior provides diverse activation functions and tree-based SCMs for hierarchical variable interactions, while some activation functions can cause overflow errors.The TabICLv2 prior uses a more sophisticated but more expensive generation process.Source: §A.3 · pp. 13–14
  • Stage 1 uses on-the-fly data generation, while Stage 2 uses 120k generated files, each containing four graph configurations with 16 batches per configuration.Stage 1 generates 128 data batches for each step; Stage 2 uses warm-up followed by cosine annealing.Source: §A.3 · p. 14
  • Adding Stage 1 pre-training with the TabICL prior substantially improves validation AUC over training only on the TabICLv2 prior.Stage 1 took 68 hours and Stage 2 took 18 hours.Source: §A.3 · p. 14
  • The two-stage setup is more training-efficient than using the TabICLv2 prior alone because generating TabICLv2 samples is substantially more expensive.Source: §A.3 · p. 14

A.4.1 DUPLICATE TASKS

Four data sets overlap between TabArena and OpenML-CC18, with dataset-level differences in some copies.

A.4.1 DUPLICATE TASKS

  • The benchmarks share blood-transfusion-service-center, diabetes, credit-g, and qsar-biodeg.The blood-transfusion-service-center and diabetes copies are identical apart from row order.Source: §A.4.1 · p. 14
  • The credit-g copy reverses the original ordinal encoding, which does not affect results.Source: §A.4.1 · p. 14
  • The qsar-biodeg copies differ: TabArena has one fewer row and changes some numeric-ordinal features to categorical.Source: §A.4.1 · p. 14

A.4.2 DISTILLATION BASELINE

The appendix specifies the distillation and evaluation procedures, reports runtime comparisons, and describes synthetic parity experiments and their distractor-only AUC ceiling.

A.4.2 DISTILLATION BASELINE

  • Distillation augments the original training data with teacher-sampled hard labels, doubling its size before fitting CART or GradTree.TabPFN2.6 and TabPFN-3 were tested as teachers; the main paper reports TabPFN2.6 because performance did not differ significantly and it was more time-efficient.Source: §A.4.2 · p. 14

A.4.3 BASELINES & HYPERPARAMETER TUNING

  • The appendix lists publicly available baseline implementations and hyperparameter settings, and its split procedure ensures both stratified partitions contain every class.The split algorithm removes missing samples and classes with fewer than two instances, then samples and splits the data.Source: §A.4.3 · p. 14, §A.4.4 · p. 15, Table 4 · p. 15

A.4.5 PREPROCESSING PIPELINE AND MISSING VALUE HANDLING

  • GradTree preprocessing uses leave-one-out encoding for every categorical feature, median imputation, and quantile transformation of all transformed features.This replaces the original cardinality-dependent one-hot/leave-one-out encoding choice.Source: §A.4.5 · p. 16

A.5 RESULTS ACORSS ALL LENGHTS

  • The appendix provides overall and binary/multiclass mean-AUC tables by budget, plus critical-difference diagrams across context lengths and joint benchmarks.The supplied captions specify these displays' scope but do not provide the underlying performance values.Source: Table 5 · p. 16, Table 6 · p. 17, Table 7 · p. 17

A.6 RUNNING SPEED

  • 0.15–0.20 seconds on GPU and 0.17–0.35 seconds on CPU: direct MotherTree inference is consistently fastest across evaluated training-set sizes.MotherTree generates each task-specific tree in one forward pass; MotherTree-Tune reaches 4.75 seconds on GPU and 5.47 seconds on CPU at B = 1000, trading runtime for improved predictive performance.Source: §A.6 · pp. 17–18, Table 8 · p. 18

A.8.1 SYNTHETIC XOR DATA SETS WITH DISTRACTOR FEATURES

  • The parity data-generation procedure constructs labels as the XOR of k parity bits and adds binary distractor features that are noisy copies of the label.Source: §A.8.1 · p. 20
  • Parity experiments vary k, distractor count m, and agreement probability q across 59 configurations, with 2,000 samples per generated data set and depth budget D = 5.The figure marks λ ≤ 0 as slack in the depth budget and λ > 0 as requiring at least λ distractors to be refused.Source: Table 9 · p. 20, §A.8.1 · p. 20, Figure 8 · p. 22

A.8.2 DISTRACTOR-ONLY BAYES AUC CEILING

  • The distractor-only Bayes-optimal score increases with the number of distractors equal to one, and its AUC provides a ceiling for distractor-only prediction.When the model class can represent the depth-k parity tree, the full-feature optimum is AUC 1; performance significantly above the distractor-only ceiling indicates use of parity-feature information.Source: §A.8.2 · pp. 21–22
Loading 2610.10832v1…