PaperlayerView exact revision on arXiv

Source-linked AI summary

The Illusion of Generalization in Tabular Language Models

Aditya Gorla, Ratish Puduppully

arXiv:2602.04031v2cs.LG

TL;DR

The authors re-examine whether TLM performance reflects tabular generalization or evaluation artifacts. They test Tabula-8B with task-type baselines, contamination probes, and instruction-tuned controls, finding aggregate strength concentrated in quartile classification and substantial performance from instruction-tuning without tabular exposure.

  • Problem

    TLMs are reported to improve on tabular benchmarks, but whether those gains reflect tabular-specific generalization remains in question.

    Source: §1 · pp. 1–2

  • Method

    The authors reevaluate Tabula-8B using a majority-class baseline, targeted contamination probes, and instruction-tuned controls without tabular exposure.

    Source: §3 · p. 3

  • Results

    Quartile classification drives aggregate strength while binary and categorical classification show weak baseline-relative performance; Alpaca achieves 92.2% of Tabula-8B’s mean accuracy.

    Source: §1 · p. 2, §4.1.2 · p. 5, §4.3.1 · p. 7

  • Takeaways & Limitations

    The authors offer seven recommendations for strengthening TLM evaluation and distinguishing evaluation artifacts from substantive advances.

    Source: §1 · p. 2

    The contamination search was not exhaustive, so the analysis can demonstrate contamination’s presence but not its absence.

    Source: §6 · p. 9

Abstract

from arXiv · show

Tabular Language Models (TLMs) have been claimed to achieve strong generalization for tabular prediction. We conduct a systematic re-evaluation of Tabula-8B as a representative TLM, utilizing 165 datasets from the UniPredict benchmark. Our investigation reveals three findings. First, binary and categorical classification achieve near-zero median lift over majority-class baselines and strong aggregate performance is driven entirely by quartile classification tasks. Second, top-performing datasets exhibit pervasive contamination, including complete train-test overlap and task-level leakage that evades standard deduplication. Third, instruction-tuning without tabular exposure recovers 92.2% of standard classification performance and on quartile classification, format familiarity closes 71.3% of the gap with the residual attributable to contaminated datasets. These findings suggest claimed generalization likely reflects evaluation artifacts rather than learned tabular reasoning. We conclude with recommendations for strengthening TLM evaluation.

1. Introduction

The paper questions whether TLMs learn tabular generalization or benefit from other mechanisms, and re-evaluates their reported gains through task-type, contamination, and instruction-tuning analyses.

1. Introduction

  • Tabular data is row- and column-permutation invariant and heterogeneous, properties that distinguish it from text and images.The passage links these properties to the success of gradient-boosted trees and Prior-Fitted Networks.Source: §1 · p. 1
  • TLMs lack explicit tabular-specific inductive biases and serialize heterogeneous table values into sequential text, raising questions about what they learn.The paper asks whether TLMs generalize over tabular data or succeed through other mechanisms.Source: §1 · pp. 1–2
  • Quartile classification drives strong aggregate TLM performance, while binary classification underperforms simple baselines and categorical results are also weak.The authors use these findings to argue that reported performance gains likely do not reflect genuine tabular generalization.Source: §1 · p. 2
  • The investigation reports task-type bias, training-test contamination and competitive performance from instruction-tuning without tabular exposure.The authors conclude with seven recommendations for strengthening TLM evaluation.Source: §1 · p. 2

2. Background

The background reviews tabular prediction and TLM claims, then identifies gaps in evaluation practices; the disclaimer frames the critique as systemic rather than directed at individual authors.

2. Background

  • The paper treats format learning versus genuine understanding as a hypothesis supported by its evidence, while acknowledging that the boundary is difficult to define precisely.This qualification limits how confidently the observed instruction-following performance can be interpreted.Source: §2 · p. 2
  • Prior work reports TLM gains over GBTs and TabPFN on aggregate accuracy, while Tabula-8B is presented as an open, large-scale case study.Tabula-8B was trained on approximately four million tables and evaluated across 329 datasets in five benchmarks.Source: §2 · p. 2
  • TLM evaluations rarely report majority-class baselines or stratify results by task type, making it difficult to distinguish learning from class-frequency exploitation.The surveyed five recent TLM papers also omit chance-corrected metrics such as Cohen’s Kappa.Source: §2 · p. 2
  • The paper notes that tabular contamination can evade row-level deduplication when records recur under changed column names, schemas, or task-level associations.Related work also reports verbatim memorization of popular tabular datasets by GPT-3.5 and GPT-4, with accuracy dropping six percentage points after perturbation.Source: §2 · p. 3

Disclaimer

  • The authors present their critique as addressing recurring evaluation patterns across TLM research, not Tabula-8B’s authors or their work specifically.They say the open T4 corpus, model weights, and evaluation suite enabled independent verification.Source: Disclaimer · p. 3

3. Approach

The study tests Tabula-8B’s generalization with baselines, contamination probes, and instruction-tuned controls without tabular exposure, using a 165-dataset UniPredict subset.

3. Approach

  • The approach tests whether TLMs beat naive baselines, whether contamination explains top performance, and whether general instruction-following suffices without tabular pretraining.The controls include Base Llama, Alpaca fine-tuned on general instructions, and Alpaca+Q with quartile-format examples.Source: §3 · p. 3
  • The evaluation covers 165 UniPredict datasets: 64 binary, 55 categorical, and 46 quartile classification tasks.Twenty quartile datasets were held out for Alpaca+Q training, leaving 26 for evaluation.Source: §3 · p. 3
  • Contamination probes search for distinctive identifiers, row-level value combinations despite column-name variations, and task-level associations that enable memorization-based solutions.The search targets top-performing evaluation datasets, where contamination would be most consequential.Source: §3 · pp. 3–4
  • The reported imbalance-rider metrics include accuracy, majority-class accuracy, lift, Cohen’s Kappa, and aggregate benchmark averages.The majority-class baseline predicts the most frequent test-set class, and lift measures improvement over naive prediction.Source: §3 · p. 3, Table 1 · p. 4

4. Findings

Stratifying Tabula-8B’s results reveals that strong aggregate performance is driven by quartile classification, while binary and categorical classification show marginal lift over majority-class baselines. The highest-performing datasets exhibit contamination, and instruction-tuned controls indicate that format familiarity and general instruction-following account for much of Tabula-8B’s apparent advantage.

4.1.1. HIGH ACCURACY DOES NOT IMPLY LEARNING.

  • 65/165 datasets (39.4%) yield no positive lift over the majority-class baseline, despite cases of high raw accuracy.On stroke prediction, Tabula-8B’s 94.9% accuracy equals the 94.9% majority-class baseline; 12 datasets have negative Cohen’s κ.Source: §4.1.1 · p. 4

4.1.2. TASK-TYPE DECOMPOSITION REVEALS SYSTEMATIC HETEROGENEITY

  • Against the majority-class baseline, binary classification achieves only 2.5 percentage points, while categorical classification exhibits a negative (-0.3) median lift; quartile tasks dominate the upper performance tail.Of the top 20 performing datasets, 11 are stock-price prediction tasks formatted as quartile classification.Source: §4.1.2 · p. 5

4.2.1. COMPLETE TRAIN-TEST OVERLAP

  • All 753 us-womens-labor evaluation rows were found in T4, including their labels, so 99.6% accuracy on that dataset is not evidence of generalization.A test row matched a training record across all fields, despite slight differences in some column names.Source: §4.2.1 · p. 5

4.2.2. DIRECT CONTAMINATION WITH LABEL EXPOSURE

  • Bitcoin records appear in up to four T4 chunks with their target Close values; evidence also implicates Pokémon labels and stock datasets among top performers.The stock-task input can also make prediction easy when Adj Close ≈ Close; the Pokémon search found all 98 examples Alpaca misclassified in T4 with correct labels.Source: §4.2.2 · pp. 5–6

4.2.3. TASK LEAKAGE BEYOND ROW-LEVEL DEDUPLICATION

  • 844 unrelated T4 records encode the date-to-day mapping for Peloton test examples, enabling task leakage despite no exact row matches.This leakage evades row-level deduplication and may inflate results on tasks reducible to widely attested factual associations.Source: §4.2.3 · p. 6, Figure 3 · p. 6

4.2. Data Leakage is Pervasive

  • 8 of the top 10 examined datasets show contamination, and 11 of the top 20 are stock-format quartile tasks with documented direct label exposure.The contamination search targeted top performers, so it demonstrates contamination’s presence, not its absence elsewhere.Source: §4.2.3 · pp. 6–7

4.3.1. CLASSIFICATION: INSTRUCTION-TUNING DOMINATES

  • 92.2%: Alpaca, instruction-tuned without tabular exposure, reaches this share of Tabula-8B’s mean accuracy across classification datasets.The comparison used Base Llama, Alpaca, and Tabula-8B; instruction-tuning accounts for 69% of Tabula-8B’s improvement over Base Llama. The study used identical serialized rows and 4-shot prompting.Source: §4.3.1 · p. 7, §4.3 · p. 7, Table 4 · p. 7

4.3.2. QUARTILE CLASSIFICATION: FORMAT AND CONTAMINATION EXPLAIN THE GAP

  • 71.3% of the quartile-classification gap is closed by Alpaca+Q, which adds quartile-format examples without tabular pretraining.The residual advantage is concentrated in stock/financial datasets with documented contamination; on non-stock datasets, Alpaca+Q has 6.7pp lift versus Tabula-8B’s 6.5pp.Source: §4.3.2 · pp. 7–8, Figure 4 · p. 8

5. Recommendations for TLM Evaluation

The recommendations aim to make TLM evaluations more transparent, task-specific, and resistant to contamination and shortcuts. They also ask reviewers and practitioners to scrutinize whether strong scores demonstrate tabular reasoning.

R1–R2: Baselines and task-stratified reporting

  • Report majority-class and base-model comparisons alongside imbalance-robust metrics, and stratify results by task type with tests for heterogeneity.The recommendations name Cohen’s Kappa or similar measures and suggest ANOVA as an example test.Source: §5 · p. 8

R3–R4: Reproducibility and contamination analysis

  • Release evaluation code, raw predictions, and baselines, and report decontamination methods, corpus-search coverage, and entity-level and fuzzy matching results.The recommendations also urge particular caution about widely replicated sources such as stock-price datasets and Kaggle competitions.Source: §5 · pp. 8–9

R5: Audit task requirements

  • Audit whether tasks require tabular reasoning by checking for memorized associations and excluding tasks solvable through factual recall, text understanding, or simple feature inference.High accuracy alone does not establish tabular reasoning when a task reduces to those shortcuts.Source: §5 · p. 9

R6: Instruction-tuned baselines

  • Compare TLMs with instruction-tuned models that had no tabular exposure; if the TLM does not substantially outperform them, tabular-reasoning claims need further justification.Alpaca and FLAN are given as examples of such baselines.Source: §5 · p. 9

R7: Critical evaluation

  • Reviewers and practitioners should expect baseline comparisons, task-stratified results, and contamination analysis, and scrutinize near-perfect accuracy or tasks reducible to text classification or feature inference.The recommendation assigns responsibility for rigorous evaluation to the broader community, not only authors.Source: §5 · p. 9

6. Limitations

The contamination search was targeted rather than exhaustive, so it establishes contamination’s presence but cannot establish its absence. The re-evaluation also covers a limited dataset subset, task type, model, and training corpus.

Contamination-analysis scope

  • The contamination analysis searched for selected patterns, but an exhaustive search across T4’s roughly 150 archives and millions of parquet files was infeasible.Consequently, the analysis can demonstrate contamination’s presence, not its absence.Source: §6 · p. 9

Detection and causal testing

  • Comprehensive tabular-contamination detection remains an open problem, and current machine-unlearning methods are not reliable at T4’s 100B-token scale.Machine unlearning could in principle provide a causal test of contamination’s effect on accuracy; agentic frameworks are proposed as a possible future route to broader searches.Source: §6 · p. 9

Benchmark and capability scope

  • The evaluation covers 165 datasets from the 329-dataset Tabula-8B benchmark and assesses classification only, not imputation, generation, or multi-table reasoning.The authors describe this as a proof of principle and call for broader re-evaluation of other benchmarks.Source: §6 · p. 9

Model and corpus scope

  • The case study focuses on Tabula-8B and T4; other TLMs trained on different corpora or objectives may show different strengths and failure modes.The authors’ literature survey suggests some evaluation concerns recur beyond this case, but this investigation does not directly establish those patterns for every TLM.Source: §6 · p. 9

7. Conclusions

The authors conclude that Tabula-8B’s strong UniPredict performance likely does not reflect genuine tabular generalization, while treating extension to the broader TLM literature as a structural conjecture. They present seven recommendations to support more trustworthy evaluation.

Scope of the conclusion

  • The investigation directly establishes its findings for Tabula-8B on UniPredict, while broader claims about TLMs remain a structural conjecture grounded in a scorecard of five papers.The authors say the scorecard finds recurring evaluation gaps across those papers.Source: §7 · p. 9

Principal findings

  • Aggregate performance is inflated by task composition, top-performing datasets show pervasive contamination, and instruction-tuning without tabular exposure recovers most reported performance.The passage attributes residual gaps to contamination rather than learned tabular reasoning.Source: §7 · p. 9

Implications

  • The authors offer seven evaluation recommendations intended to help distinguish genuine advances from artifacts and strengthen empirical foundations for TLM research.They frame these recommendations as supporting the field’s progress, not discouraging research on tabular foundation models.Source: §7 · p. 9

Impact Statement

The paper frames its critique as an effort to improve evaluation practices and empirical foundations in TLM research. The appendix passages describe the Alpaca baseline and example prompts used in evaluation.

Impact Statement

  • The authors aim to strengthen TLM evaluation and reduce overestimation that could contribute to inappropriate deployment in high-stakes domains.They describe the critique as addressing systemic patterns rather than targeting individual work.Source: Impact Statement · p. 10

A.1. Alpaca Training

  • The Alpaca baseline fine-tunes base Llama-3-8B on 50K general instruction-following examples with no tabular data, using LoRA for one epoch.The stated setup uses rank r = 8, scaling factor α = 16, learning rate 2 × 10^-4, and batch size 128.Source: §A.1 · p. 13
  • At inference, evaluation examples use the Alpaca instruction template while preserving the serialized rows and few-shot examples from Tabula’s evaluation.The appendix states this preserves the input content so performance differences reflect model capabilities rather than input formatting.Source: §A.1 · p. 13

A.2. Prompt Examples

  • The prompt appendix presents representative examples for each task type using four labeled examples before the test instance.It shows both Tabula’s special-token format and the Alpaca instruction-template format.Source: §A.2 · p. 14

A.2.1. Categorical Classification: Pokémon Type Prediction

  • The categorical example predicts a Pokémon’s primary type from its name and battle statistics, with Type 1 as the target.The displayed example uses Zygarde50% Forme as a labeled example and Sharpedo as the test instance.Source: §A.2.1 · p. 14

A.2.2–A.2.3. Binary and Quartile Classification

  • The binary example predicts labor force participation from demographic features, while the quartile example predicts discretized Bitcoin closing-price bins from market features.The Bitcoin prompt lists Date, Open, High, Low, Adj Close, and Volume as inputs.Source: §A.2.2 · p. 15, §A.2.3 · pp. 15–16

A.3. Alpaca+Q Training

Alpaca+Q tests familiarity with quartile-output formatting using held-out regression datasets, while model comparisons share a few-shot serialized-row protocol and majority-class lift metric.

A.3. Alpaca+Q Training

  • Alpaca+Q adds 10K quartile-format examples from 20 regression datasets held out from evaluation, isolating format familiarity from exposure to evaluation data.Examples use the same serialization as Tabula-8B evaluation.Source: §A.3 · p. 16

A.4. Evaluation Protocol

  • All models use few-shot serialized-row prompting, greedy decoding, and lift over the majority-class baseline as the primary comparison metric.The protocol extracts the predicted class from each generated completion; temperature is 0.Source: §A.4 · p. 17

B.1. Challenges for Tabular Contamination Detection

  • Tabular contamination can evade exact-match or n-gram checks because rows and columns may be reordered, names vary, and records recur across sources.Task-level leakage can also occur when training data contains associations that solve a task even without the evaluation records.Source: §B.1 · p. 17

B.2. Search Procedure

  • The search prioritized top-performing datasets and used distinctive identifiers, rare row-value combinations, and task-level association queries to find possible overlap.Searches were targeted rather than exhaustive because T4 contains millions of parquet files.Source: §B.2 · pp. 17–18

B.3. Limitations

The contamination analysis is explicitly non-exhaustive and focused on top performers; the accompanying task-type tables describe lift summaries and datasets with negative κ.

B.3. Limitations

  • The contamination search cannot establish absence: it targeted patterns, prioritized top performers, and may miss overlap that more advanced detection methods would find.Lower-performing datasets were not systematically examined, and tabular contamination detection lacks established best practices.Source: §B.3 · p. 18
  • The authors characterize their contamination findings as a lower bound on the benchmark’s true contamination extent.Source: §B.3 · p. 18

C. Extended Task-heterogeneity Results

D. Quartile Classification May Be Artificially Easy

Quartile classification can reward numeric shortcuts, balanced-bin prediction, and ordinal reasoning rather than tabular-specific learning; task-type lift comparisons are statistically significant.

D. Quartile Classification May Be Artificially Easy

  • Quartile bins divide continuous targets into four groups, and correlated input features can create numeric shortcuts for predicting the target.For example, Open, High, Low, and Adj Close can help infer a stock’s Close-price bin.Source: D. Quartile Classification May Be… · pp. 18–19
  • All pairwise comparisons of lift over the majority-class baseline across task types are statistically significant at 0.05.Source: Table 8 · p. 19
  • Quartile bins are balanced by construction, unlike naturally imbalanced binary and categorical tasks, and their ordinal structure may invite pretrained numeric reasoning.Source: D. Quartile Classification May Be… · p. 19
  • Figure 5 plots Cohen’s κ distributions by task type against standard interpretation thresholds.Source: Figure 5 · p. 19

E. Extended Contamination Examples

Extended examples document exact labeled overlap, repeated financial records, and task-level date-day leakage; the authors also compare evaluation practices across five other papers.

E. Extended Contamination Examples

F. Alpaca Extended Results

  • The tables report accuracy for bundesliga-seasons and udemy-courses, compare mean and median lift over the majority-class baseline before and after excluding documented-contamination datasets, and give accuracy scores across 165 tabular datasets.The exclusions are pokemon (×2), peloton-data, us-womens-labor, and udemy-courses8. Table 13 names Tabula-8B, Llama-3-8B, and Alpaca variants; its empty cells indicate tasks used to train Alpaca+Q.Source: Table 11 · p. 22, Table 12 · p. 22, Table 13 · p. 22

G. Evaluation Gaps Are Endemic Across TLM Research

G.1.1. UNIVERSAL ABSENCE OF TRIVIAL BASELINES

  • None of the five papers reports a majority-class baseline or chance-corrected metrics, leaving raw accuracy gains without baseline context in examples such as UniPredict.TabLLM uses AUC but does not discuss class distributions in its evaluation datasets.Source: §G.1.1 · p. 26

G.1.2. INCOMPLETE TASK-TYPE STRATIFICATION

  • The reviewed papers do not separate binary from multiclass classification or test task-type heterogeneity; UniPredict aggregates accuracy across all 169 datasets.CARTE separates regression and classification learning curves but still lacks that finer stratification and testing.Source: §G.1.2 · p. 26

G.1.3. CONTAMINATION ANALYSIS IS UNIVERSALLY ABSENT

The review finds shared evaluation gaps across TLM studies and examines how its recommendations carry over, with constraints, to closed-source models and table-synthesis methods.

G.1.3. CONTAMINATION ANALYSIS IS UNIVERSALLY ABSENT

  • All five surveyed papers use public datasets likely present in LLM training corpora, yet none analyze contamination; prior work reports verbatim dataset memorization.Source: §G.1.3 · p. 26

G.1.4. INSTRUCTION-TUNING CONFOUNDED WITH TABULAR LEARNING

  • Prior evaluations leave instruction-following confounded with tabular learning, while the GTL control remains partial because it does not separate these capabilities.Source: §G.1.4 · p. 27

G.1.5. REPRODUCIBILITY

  • CARTE offers code, datasets, and 10-split results with standard deviations, whereas UniPredict lacks code, predictions, uncertainty summaries, and significance tests.Source: Table 14 · p. 27, §G.1.5 · p. 27

G.2. Summary

  • All four comparison papers exhibit the same gaps as the Tabula-8B case study, which the authors characterize as endemic rather than isolated.Source: §G.2 · pp. 27–28, Table 15 · p. 28

G.3. Extending the Framework for Closed-Source TLMs

  • Five of seven recommendations remain usable for closed-source TLMs; contamination checks weaken to suggestive behavioral probes, while code release is out of reach.Source: §G.3 · p. 28

G.4. Extending the Framework for Table-Synthesis Methods

  • Synthetic table generation is excluded because synthesis uses different evaluation primitives; only baseline and contamination checks transfer by analogy, with a full audit left out of scope.Source: §G.4 · p. 28

Related papers

Loading 2602.04031v2…