PaperlayerView exact revision on arXiv

Source-linked AI summary

LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains

Ibna Kowsar, Kazi F. Akhter, Manar D. Samad

arXiv:2511.06161v2cs.LGcs.AI

TL;DR

Cross-domain tabular transfer is difficult because domains can have disparate feature spaces, while prompt-based LLM approaches face token, scalability, and prompt-engineering limitations. LATTLE transplants attention weights from an LLM adapted to source data into a tabular transformer for target learning, and the paper reports superior performance across disjoint-domain benchmarks against traditional ML, deep tabular, and large pretrained alternatives. The results support lightweight, context-focused transfer without shared features or large-scale tabular pretraining.

  • Problem

    Cross-domain tabular transfer lacks effective methods for heterogeneous feature spaces, while in-context LLM approaches are limited by token inefficiency, scalability, and subjective prompt engineering.

    Source: §1 · p. 1, §6 · p. 11

  • Method

    LATTLE fine-tunes an LLM with source tabular data and transplants its attention-related weights into a gFTT model fine-tuned on target data.

    Source: §1 · p. 2, §6 · p. 11

  • Results

    LATTLE achieves better performance than state-of-the-art ML and DL tabular methods, with selective attention transfer outperforming end-to-end LLM fine-tuning and a single source data set sufficient for transfer.

    Source: §6 · pp. 10–11, §7 · p. 12

  • Takeaways & Limitations

    Cross-domain tabular transfer can use lightweight, context-focused attention learning without shared features, prompt engineering, or large-scale tabular pretraining.

    Source: §6 · pp. 11–12, §7 · p. 12

    LATTLE underperforms on several data sets, uses a distilled GPT because larger LLM training is computationally demanding, and cannot exclude memorization or hallucination after end-to-end training.

    Source: §6.1 · p. 12

Abstract

from arXiv · show

Transfer learning on tabular data is challenging due to disparate feature spaces across domains, in contrast to the homogeneous structures of image and text. Large language models (LLMs) offer a knowledge base to improve the limited effectiveness of cross-domain transfer learning for tabular data. However, LLM performance often stagnates due to subjective text prompts and the computational limitations of in-context learning. We present a novel language-to-tabular context-learning method that uses attention-specific transformer weights, enabling seamless transfer learning across disparate tabular data sets. The LLM attention transplant mechanism facilitates a domain-agnostic transfer learning, eliminating the need for shared features between tables, LLM prompt engineering, and large-scale pretrained models. Our experiments using ten pairs of disjoint source-target data sets and 12 baseline methods demonstrate the superiority of the proposed LLM-attention transplant for transfer learning (LATTLE) method over traditional ML models, state-of-the-art deep tabular architectures, and models trained on thousands to billions of tabular samples. The proposed cross-domain attention transfer demonstrates an effective solution for adapting LLMs to learning non-text tabular data in a low-resource environment. The source code of the LATTLE implementation is publicly available.

1 Introduction

Tabular transfer learning remains difficult because mixed feature types, limited samples, and disjoint domain schemas weaken deep representation transfer. LATTLE addresses this gap by transplanting attention weights from an LLM adapted to source data into a tabular transformer for target classification.

1 Introduction

  • LATTLE targets cross-domain transfer between tabular data sets whose feature spaces may be completely disjoint.Existing methods often require shared features or perform poorly when source and target schemas differ substantially.Source: §1 · p. 1, §1.1 · p. 2

1.1 Literature review and contributions

  • LLMs provide a potential knowledge reservoir for tabular transfer, but zero-shot and few-shot approaches often underperform traditional machine learning.In-context learning also requires serialization, manual prompt engineering, and limited numbers of examples.Source: §1 · p. 2, §1.1 · p. 2

1 Introduction

  • LATTLE establishes cross-attention by transplanting attention-related weights from an LLM fine-tuned on source data into a gFTT fine-tuned on target data.The framework is designed for downstream tabular classification rather than text generation.Source: §1 · p. 2, Figure 1 · p. 3

1.2 Contributions

  • The method uses a tabular learning objective for the LLM instead of the standard autoregressive next-token objective.This design adapts the language model to mixed-type tabular data before cross-attention transfer.Source: §1.2 · p. 3
  • Transplanted attention weights capture context between heterogeneous domains without requiring shared features, prompt engineering, or in-context learning.The approach performs downstream cross-attention learning with fine-tuned LLM weights.Source: §1.2 · p. 3
  • LATTLE demonstrates lightweight cross-domain transfer using a modest source data set without millions of samples or heavy computing resources.Its stated aim is to provide an alternative to large-scale tabular pretraining approaches.Source: §1.2 · p. 3, §1.1 · p. 3

2 Methodology

LATTLE adapts an open-source language model to tabular data, transfers selected attention weights across disjoint domains, and fine-tunes a target gFTT classifier. The method uses hierarchical key-value projections to provide source context while preserving target-specific adaptation.

2.1 Preliminaries

  • The attention mechanism converts feature embeddings into query, key, and value vectors and computes directional feature-to-feature attention contexts.Attention is directional because the attention from feature p to feature j need not equal the reverse direction.Source: §2.1 · p. 4

2.2 Fine-tuning LLM using source tabular data

  • LATTLE jointly fine-tunes DistilGPT2 and gFTT on serialized source tabular data, then uses their attention projections for language-to-tabular transfer.Feature-value pairs are serialized into text, tokenized, and processed with a DistilGPT2-gFTT framework optimized with cross-entropy loss.Source: §2.2 · p. 4, §2.4 · p. 6

2.3 Hierarchical attention weight selection

  • LATTLE selects key and value projection weights from the LLM’s topmost attention layer for downstream transplantation, while freezing the target transformer’s base layer.Higher layers are treated as richer contextual representations, whereas lower layers are preserved as more general knowledge.Source: §2.3 · p. 5

2.4 Proposed LATTLE algorithm

  • The proposed algorithm freezes transplanted source key-value weights in the target gFTT base layer and fine-tunes upper layers on target data.The target query interacts with frozen source-derived values, producing cross-domain attention at the transformer-weight level rather than through shared feature representations.Source: §2.4 · pp. 5–6

3 Language-encoded tabular embedding

The target gFTT represents categorical and numerical features with BERT-based embeddings before applying transferred attention during target training. Its classification output is formed from the aggregated [CLS] context vector.

3 Language-encoded tabular embedding

  • Categorical feature names and values are tokenized into BERT embeddings, while numerical values scale embeddings of their feature names.These representations and a special [CLS] embedding are concatenated into the gFTT input tensor.Source: §3 · p. 6
  • The target gFTT queries the source LLM through frozen transplanted weights while fine-tuning other layers to adapt to the target distribution.The lowest attention layer retains the transferred source context, whereas higher gFTT layers are updated.Source: §3 · p. 6
  • The [CLS] context vector aggregates numerical and categorical feature contexts and feeds a linear classifier that produces target-class probabilities.Training applies cross-entropy loss to logits derived from the aggregated [CLS] representation.Source: §3 · p. 6

4 Experiments

The experiments evaluate LATTLE on disjoint-feature tabular transfer tasks against traditional machine learning, deep tabular architectures, and transfer-learning baselines. Ten source-target pairs, consistent evaluation procedures, and AUC and ACC metrics support the comparison.

4.1 Tabular data sets

  • Ten source-target pairs use publicly available data from diverse domains with strictly disjoint feature spaces.The 14 data sets span healthcare, finance, manufacturing, and software testing, with 540–70,000 samples and 8–76 features.Source: §4.1 · p. 7

4.2 Baseline methods

  • The baselines include traditional ML, deep tabular models, specialized transfer methods, and LLM-based approaches.The comparison includes LR, MLP, XGBoost, ResNet, TabNet, FT-Transformer, TransTab, XTab, CM2, Tabula-8B, and DistilGPT2 variants.Source: §4.2 · p. 7
  • The evaluation tables report average AUC and ACC with and without source pre-training, including cross-domain DistilGPT2 and Tabula-8B comparisons.Tabula-8B AUC values are taken from its corresponding paper, while its ACC values are reported as unavailable.Source: Table 2 · p. 8, Table 3 · p. 8

4.3 Implementation and evaluation

  • Transfer learning uses source pre-training followed by target fine-tuning, while traditional ML and deep-learning baselines train directly on target data.The same training, validation, and test ratios are used for source pre-training and target fine-tuning.Source: §4.3 · p. 8
  • LATTLE uses a five-layer gFTT with transplanted LLM attention weights and reports target-test AUC and ACC averaged over ten random seeds.The gFTT has eight attention heads per layer, a 2048-dimensional feedforward network, ReLU activations, and dropout.Source: §4.3 · p. 8

5 Results

Across ten disjoint-feature source-target pairs, LATTLE is compared with 12 baselines under consistent training and evaluation procedures. It achieves the strongest average ranking, while the ablation indicates limited sensitivity to source data and attention-layer selection.

5.1 Cross-domain data set pairs

  • The ten cross-domain pairs combine source and target data sets from different application domains while enforcing no common features.The listed pairs include banking–metabolic disease, heart disease–metabolic disease, manufacturing–software testing, and other domain combinations.Source: §5.1 · pp. 8–9

5.2 Model training and selection

  • Training uses 70%/10%/20% train-validation-test splits, ten predefined random seeds, source DistilGPT2 fine-tuning, and target validation-based gFTT selection.The LLM is fine-tuned for 200 epochs, while gFTT hyperparameters use 100 Optuna trials and model selection over 150 epochs.Source: §5.2 · p. 9

5.3 Transfer learning performance

  • LATTLE achieves the best average AUC rank at 2.0 (0.9), ahead of TransTab at 4.8 (2.5), across the transfer-learning comparison.TransTab uses one source data set, whereas XTab and CM2 are pretrained on 52 and 2000 tabular data sets, respectively.Source: §5.3 · pp. 9–10
  • LLM-based transfer baselines underperform, with Tabula-8B ranked 13 and cross-domain DistilGPT2 ranked 9, whereas target-only DistilGPT2 performs better than ML and DL baselines.For VH, Tabula-8B performs below random guessing while other competitive baselines exceed 0.90 AUC.Source: §5.3 · p. 10
  • LATTLE also achieves the best average ACC rank at 2.3 (1.6), followed by TransTab at 4.7 (3.0).The ACC ordering differs in some cases: XGBoost surpasses XTab and CM2, and the single-domain LLM surpasses XTab and CM2.Source: §5.3 · p. 10

5.4 Ablation study

  • The best attention-transplant strategy uses the top LLM attention layer in the lowest gFTT attention layer, while alternative layer choices differ minimally in mean AUC.Across varying source data sets, downstream AUC scores remain within 1.5%.Source: Table 4 · p. 9, §5.4 · p. 10

6 Discussion of the results

LATTLE uses selective attention-weight transfer and a tabular transformer to learn cross-domain relationships without shared features, while outperforming leading tabular transfer methods. The discussion also reports that performance remains data-dependent and that model scaling is unresolved.

6 Discussion of the results

  • LATTLE outperforms state-of-the-art ML and deep learning methods for tabular transfer learning through selective attention-weight transplants.The method achieves better performance than traditional and deep tabular approaches, with domain attention learning more effective than end-to-end LLM fine-tuning.Source: §6 · pp. 10–11
  • The method combines LLM–gFTT representation alignment, cross-attention between source and target data, and tabular-transformer fine-tuning.This design uses language-to-tabular alignment while retaining the inductive bias of a tabular model for downstream learning.Source: §6 · p. 11
  • Prompt-based in-context learning is limited by token inefficiency, scalability, and prompt subjectivity, while structured-to-text conversion can lose tabular information.Direct weight fine-tuning is presented as an alternative, although knowledge transfer and representation alignment between text and tables remain open problems.Source: §6 · p. 11
  • Cross-attention learns contextual relationships rather than depending on source-domain knowledge or shared features between heterogeneous tables.The method uses key and value representations to bridge language and tabular representations, and its robustness to varying source data sets suggests limited dependence on source-domain knowledge.Source: §6 · pp. 11–12
  • A single source data set and the uppermost LLM attention layer are sufficient for transfer, reducing reliance on large-scale tabular pretraining.The upper layer is treated as more task-specific, while lower layers remain frozen to retain general language knowledge.Source: §6 · pp. 11–12

6.1 Limitations

  • LATTLE underperforms on several data sets, and its computational requirements limited the study to a distilled GPT while model scaling remains unresolved.The authors also note that larger models may perform better, but data memorization and hallucination cannot be ruled out for end-to-end training.Source: §6.1 · p. 12

7 Conclusions

The paper introduces an LLM-based cross-attention method for learning relationships between tabular data sets from different domains. Results show that a lightweight LLM can outperform state-of-the-art deep and transfer-learning methods without shared features or large-scale tabular pretraining.

7 Conclusions

  • LATTLE uses LLM-based cross-attention to learn relationships between tabular data sets from different domains without requiring shared features.Source: §7 · p. 12
  • A lightweight LLM and a single source data set are sufficient to outperform state-of-the-art deep and transfer-learning methods for tabular data.Source: §7 · p. 12

Related papers

Loading 2511.06161v2…