PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data

Kazi F. Akhter, Ibna Kowsar, Manar D. Samad

arXiv:2608.28209v1cs.LG

TL;DR

Transfer learning for tabular data remains difficult across heterogeneous domains because existing methods commonly depend on shared features or representations. CATTLE learns generalized context from transformer projection weights and transfers it between disjoint source and target tables. Across the reported evaluation, the method outperforms state-of-the-art tabular classification and transfer-learning baselines, while its attention-layer selection remains theoretically underpinned.

  • Problem

    Tabular transfer learning is challenging across domains because heterogeneous feature spaces are often disjoint, whereas existing methods commonly assume shared features or representations.

  • Method

    CATTLE derives data-agnostic generalized context from pretrained transformer projection weights and uses source weights with target query weights for cross-domain attention.

  • Results

    CATTLE outperforms state-of-the-art tabular classification and transfer-learning methods, including models pretrained on large tabular datasets.

  • Takeaways & Limitations

    A single source dataset can provide generalized context for downstream learning on a different tabular domain without shared feature spaces.

  • Takeaways & Limitations

    Attention-layer selection relies on prior knowledge about how knowledge is distributed across deep layers and lacks a theoretical underpinning.

Abstract

from arXiv · show

Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. \mds{This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for $key$, $value$, and $query$ provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for $key$ from the source domain interact with the weight for $query$ in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods.} The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.

1 Introduction

Tabular transfer learning is difficult because feature types and feature spaces vary across datasets, while existing approaches often rely on shared features or representations. CATTLE addresses this gap by learning generalized context for cross-domain transfer between disjoint tables.

  • Challenge: Tabular data combine heterogeneous feature types, and no single learning method consistently performs best across diverse datasets.Traditional feature engineering and classifier selection therefore remain common practices.
  • Challenge: Disparate tabular domains often have disjoint feature spaces, such as banking transactions and electronic health records.These differences make broadly applicable knowledge difficult to acquire for transfer learning and foundation models.
  • Prior work: Existing tabular transfer methods typically exploit shared features, similar distributions, or common representations between source and target datasets.Examples include TransTab, CM2, and XTab, whose source-target settings retain feature or domain overlap.
  • Contribution: CATTLE introduces generalized context from transformer projection weights to enable cross-domain transfer without shared feature spaces.The method is presented as a transfer-learning framework for disjoint tabular datasets and can obtain generalized context from a single tabular source dataset.

2 Methodology

The methodology represents tabular columns as tokens, computes between-feature attention, and pretrains a gFTT on source data using masked reconstruction. CATTLE transfers selected source projection weights into a target model to provide data-agnostic cross-domain attention.

  • Attention preliminaries: A transformer treats each tabular feature as a token and projects its embedding into query, key, and value vectors using trainable weights.The resulting context vector represents a column in the context of all columns.
  • Attention preliminaries: Attention weights quantify how strongly each query column attends to each key column, and the context vector is their attention-weighted sum of value vectors.The attention score uses a scaled dot product followed by normalization with the softmax function.
  • Generalized context: CATTLE replaces data-dependent activation context with generalized context derived from transformer projection weights for cross-domain transfer.Source key-value projection weights provide the foundation, while the target query projection weight interacts with the source key projection to yield cross-domain attention.
  • Source pretraining: Source pretraining uses random masking and reconstructs masked entries from unmasked values while preserving at least one visible feature per sample.Categorical and numerical features use different reconstruction procedures and their loss components are normalized separately.
  • Target transfer: Attention-related weights from a pretrained source gFTT are transferred into a newly initialized target gFTT, which is then fine-tuned on target data.Selected transferred weights remain frozen to retain the generalized-context foundation while other upper-layer weights are updated.

3 Experiments

Experiments evaluate CATTLE on cross-domain tabular transfer using disjoint source-target data sets and compare it with machine learning, deep learning, and transfer-learning baselines. The method uses source pretraining and target adaptation through cross-attention weight transfer.

  • 3.1 Tabular data in cross-domain pairing: 10 source-target pairs are formed from 14 OpenML tabular data sets spanning health, finance, manufacturing, software testing, and industrial design.The data sets contain mixed numerical and categorical features, 540–70,000 samples, and 8–76 features.
  • 3.2 CATTLE implementation: CATTLE pretrains a source gFTT and uses its attention weights to construct a cross-attention model for target transfer.The algorithm takes source and target data sets, trains gFTTs, extracts source attention-layer weights, and performs target-side weight replacement for cross-attention.
  • 3.1 Tabular data in cross-domain pairing: Each transfer pair has disjoint feature spaces with no shared features, while differing in feature-space size and semantics.This setting is stricter than conventional tabular transfer-learning assumptions requiring feature overlap.
  • 3.3 Model implementation and evaluation: The baseline suite includes XGBoost, Logistic Regression, ResNet, MLP, TabNet, FT-Transformer, and tabular transfer-learning methods.Baselines cover direct target classification without transfer learning and state-of-the-art transfer-learning approaches for tabular data.
  • 3.3 Model implementation and evaluation: Figure 2 compares loss curves for self-supervised and supervised pretraining and their corresponding target-data fine-tuning procedures.The four panels cover self-supervised pretraining, supervised pretraining, self-supervised fine-tuning, and supervised fine-tuning.
  • 3.3 Model implementation and evaluation: Target classification performance is selected using validation-based hyperparameter optimization and reported on the held-out test fold.Optuna evaluates 100 randomly sampled hyperparameter configurations, and the best validation model is used for test performance.

4 Results

Across ten source–target pairs, the experiments compare CATTLE with direct target-data models and tabular transfer methods using AUROC, accuracy, ranks, and Wilcoxon tests. Ablations examine pretraining, attention-layer selection, source context, weight-transfer mechanisms, and feature encoding.

  • Evaluation: AUROC and accuracy provide complementary evaluations, so model rankings can vary with the chosen metric.AUROC evaluates sensitivity-specificity tradeoffs across thresholds, whereas accuracy uses a single fixed threshold.
  • Baseline comparison: XGBoost ranks better than the best deep-learning baseline on both AUROC and accuracy.XGBoost has ranks of AUROC 4.7 (1.6) and ACC 5.0 (2.9), versus the best deep-learning rank of AUROC 8.1 (1.4) and ACC 6.6 (3.8).
  • Statistical comparison: CATTLE with self-supervised pretraining wins against every baseline on statistically significant comparisons.Against XGBoost, the comparison is significant on four data sets, with CATTLE winning three.
  • State-of-the-art comparison: Self-supervised CATTLE is competitive with large-scale pretrained models while using only a single source data set.It outperforms TabPFNv2 in several pairs and TARTE in most cases.
  • Ablation studies: 0.018 is the AUROC range caused by varying source data sets, indicating negligible source-specific context effects.The ablation interprets improved performance despite this small range as evidence for generalized context effectiveness.
  • Ablation studies: Top-to-bottom attention-layer configurations achieve the best performance on six of ten source–target pairs.The reported differences among attention-layer choices are marginal.
  • Ablation studies: CATTLE attains the highest performance on seven of ten source–target combinations when comparing weight-transfer mechanisms.Direct fine-tuning yields the best result on two pairs.
  • Ablation studies: The BERT-based encoder outperforms conventional encoding variants by preserving feature-name and categorical-value semantics.The reported interpretation is that retaining semantic and context information improves representations and model performance.

5 Discussion

The discussion presents CATTLE as a weight-level cross-domain transfer method for tabular data without shared features, while noting unresolved questions about attention-layer selection and knowledge distribution.

  • Generalized context: A single source data set can provide data-agnostic generalized context for downstream learning across disparate domains.The approach does not require large volumes of disjoint data sets from multiple domains.
  • Pretraining: Self-supervised pretraining outperforms supervised pretraining for data-agnostic transfer learning on the reported tasks.
  • Method: CATTLE transfers attention-layer weights rather than key, value, and query representations to perform cross-domain attention.This design is intended to avoid dependence on source-domain representations.
  • Limited samples: On Cylinder Bands, CATTLE with self-supervised pretraining outperforms XGBoost despite the data set having only 540 samples and 39 features.The data set contains an even mix of numerical and categorical variables.
  • Limitations: Attention-layer selection relies on prior knowledge about knowledge distribution in deep layers and lacks a theoretical underpinning.The paper calls for a more systematic selection method and clearer explanations of knowledge distribution across multi-head attention layers.

6 Conclusions

The conclusion presents weight-level cross-domain attention as a way to transfer generalized context for heterogeneous tabular data without shared features. It reports that this approach outperforms state-of-the-art tabular classification and transfer-learning methods, including models pretrained on large tabular data sets.

  • Conclusions: Weight-level attention provides data-agnostic generalized context for cross-domain transfer learning on heterogeneous tabular data.
  • Conclusions: A single source data set can learn generalized context for a downstream task in a different domain.
  • Conclusions: The proposed generalized context outperforms state-of-the-art classification and transfer-learning methods on tabular data.The comparison also includes models pretrained on large tabular data sets.

A.1 Effects of the number of active attention layers

Table 8 reports how varying the number of active attention layers in supervised pretrained gFTT affects downstream target-data classification.

  • Experimental setup: Table 8 varies the number of active attention layers in the pretrained gFTT model.
  • Evaluation: The reported model is supervised pretrained gFTT evaluated on downstream target data.
  • Metric: Performance is measured using AUROC scores for target-data classification.

A.2 Performance against recent state-of-the-art models

Table 9 compares supervised and self-supervised CATTLE with TARTE and TabPFNv2 across multiple source–target tasks.

  • Compared methods: Table 9 compares CATTLE in supervised and self-supervised forms with TARTE and TabPFNv2.
  • Evaluation scope: The comparison spans multiple source–target tasks.
  • Evaluation scope: Table 9 evaluates CATTLE against TARTE and TabPFNv2 in cross-domain transfer settings.
Loading 2608.28209v1…