PaperlayerView exact revision on arXiv

Source-linked AI summary

Tabular foundation models for non-tabular tasks

Goran Nakerst, John Brennan, Wouter Beugeling, Masudul Haque

arXiv:2608.22594v1cs.LG

TL;DR

The paper asks whether tabular foundation models can extend to non-tabular classification tasks. It applies TabPFN v3 to images and language represented as flat table rows, varying context size without fine-tuning; performance rises across tasks and is competitive with specialized methods in some settings.

  • Problem

    The paper investigates whether tabular foundation models can handle tasks not usually regarded as tabular, given that many datasets can be represented as tables.

  • Method

    TabPFN v3 is evaluated on MNIST, French-versus-German word recognition, and Tiny ImageNet by representing examples as rows with missing labels and varying in-context sample counts without weight updates.

  • Results

    Across the three tasks, TabPFN accuracy rises monotonically with context size; it reaches 98.6% on MNIST, 88% on the language task, while Tiny ImageNet remains near chance.

  • Takeaways & Limitations

    TabPFN v3 can achieve similar-order performance to specialized models on some non-tabular tasks despite lacking explicit spatial or sequential structure and task-specific training.

Abstract

from arXiv · show

Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.

I. INTRODUCTION

The paper asks whether tabular foundation models can handle tasks not usually regarded as tabular. It evaluates TabPFN v3 on three such tasks using in-context learning without updating model weights, finding similar-order performance to specialized models in some cases.

  • Motivation: Tabular foundation models aim to generalize across datasets, unlike conventional gradient-boosted models trained from scratch on one dataset type.This motivates testing whether their capabilities extend beyond conventional tabular tasks.
  • Problem framing: MNIST can be represented as rows containing 784 pixel values and a missing digit label, allowing digit recognition to be framed as tabular prediction.The context set contains labeled examples, while query rows have empty labels.
  • Research question: The paper asks to what extent currently available tabular foundation models suit tasks not usually regarded as tabular.
  • Evaluation: The study evaluates TabPFN v3 on MNIST, French-versus-German word recognition, and 100-class Tiny ImageNet while varying the number of context samples.No task-specific weight updates are performed; learning is entirely in-context.
  • Main finding: In some cases, TabPFN v3 reaches accuracies of similar order to specialized models despite receiving no task-specific training.The comparisons use support sets for in-context learning and training sets of the same size for non-foundation models.
  • Implication: The results suggest that TFM capabilities are not necessarily confined to conventionally tabular data and may motivate reconsidering the tabular/non-tabular distinction.The paper also suggests causal-structure pretraining may support generalization across task types.

II. BASICS OF TABULAR FOUNDATION MODELS

Tabular foundation models predict missing target values from context and query rows using a pretrained transformer. Their in-context design supports cross-table generalization but constrains dataset scale, while the paper illustrates evaluation on MNIST raw pixels.

  • Tabular prediction: Tabular machine learning predicts missing values in a target column from known feature columns.
  • Definition: A tabular foundation model is a single pretrained neural network that performs predictive tasks on tabular datasets unseen during training.
  • In-context learning: TFMs load labeled context rows and unlabeled query rows into a transformer to produce predictions through in-context learning.Their pretraining on synthetic datasets generated via causal graphs is intended to capture abstract feature relationships that generalize across tables.
  • Architecture: TabPFN processes tables with alternating column and row attention, capturing feature dependencies and relationships between samples while preserving row and column permutation invariance.
  • Scalability: Context-window limits bound dataset size, creating a trade-off between feature capacity and row capacity even in TabPFN v3.For datasets with millions of rows or tens of thousands of features, gradient-boosted trees remain indispensable.
  • Evaluation example: MNIST is evaluated from raw pixels using TabPFN v3, with accuracy measured as context size varies against a fixed 5,000-image test set.The figure compares TabPFN v3 with a simple CNN and LightGBM trained on identical splits and repetitions.

III. MNIST

The MNIST experiment gives TabPFN v3 flattened raw pixels and asks it to classify digits purely from in-context examples. Accuracy rises sharply with context size and approaches the performance scale of a CNN despite lacking image-specific structure.

  • Setup: TabPFN v3 receives each 28 × 28 MNIST image as a flat row of 784 raw pixel intensities with no normalization, augmentation, or dimensionality reduction.
  • Setup: Classification is purely in-context: support examples are loaded into the context and predictions are produced in one forward pass without gradient updates.
  • Experimental design: The fixed test set contains 500 images per digit, while the number of context examples varies from 1 to 2,000 per digit.This corresponds to 10 to 20,000 in-context rows overall.
  • Comparison: CNN > TabPFN v3 > LightGBM holds across the full context-size curve, with the largest TabPFN-to-CNN gap in the few-example regime.
  • Interpretation: TabPFN v3 reaches similar-order accuracy to the CNN despite lacking access to two-dimensional image structure and using no gradient-descent training.

IV. DIFFERENTIATING FRENCH AND GERMAN WORDS

TabPFN v3 classifies French and German words from minimally processed letter codes using in-context learning, with performance depending on context size and model encoding.

  • Task and setup: TabPFN v3 represents each word as a zero-padded 15-column ordinal vector of letters, without linguistic features, and predicts in one forward pass.Accents and umlauts are stripped, and no gradient updates are used.
  • Evaluation: The comparison uses identical words for all models at each seed, with a fixed disjoint test set of 4,000 words and averages over 10 training resamplings.
  • Results: 62.4% at Ntrain = 50 and 88.2% at Ntrain = 10,000 are the reported TabPFN v3 test accuracies from a 50% chance level.
  • Comparison: TabPFN v3 leads in the few-shot regime at Ntrain ≤100, but both tuned LightGBM encodings overtake it from Ntrain ≥500.
  • Interpretation: The experiment contrasts foundation-model in-context learning with actual model training without pretraining.

V. TINY IMAGENET

On Tiny ImageNet, TabPFN v3 performs above chance but remains weak on raw natural-image pixels; PCA improves results modestly once enough context is available.

  • Task and setup: TabPFN v3 receives each 64 × 64 RGB image as 12,288 raw pixel features and performs classification in one forward pass without gradient updates.A PCA variant reduces the input to 200 features because the raw dimensionality exceeds the ensemble’s default coverage.
  • Raw-pixel results: 1.4% at one example per class and 11.2% at 100 are the raw-pixel accuracies, versus a 1% chance level.The test set contains 5,000 images, with training examples varied from 1 to 100 per class.
  • PCA comparison: From Ntrain ≥ 20, the 200-component PCA variant pulls ahead of raw pixels and reaches about 2 points higher accuracy at 100 examples per class.PCA provides no few-shot gain because components estimated from small training sets are too noisy.
  • Interpretation: Covering every feature recovers only about two points, indicating that the absence of a spatial prior is the larger limitation for these high-dimensional natural images.
  • Limitation: The experiment highlights a size limitation of current-generation TFMs.

VI. CONCLUSION

The paper finds that TabPFN v3 can extend beyond tabular tasks through flat raw-feature representations and in-context learning, though success varies substantially by task.

  • Scope: TabPFN v3 was evaluated on MNIST, French-versus-German word recognition, and Tiny ImageNet using flat rows of minimally processed raw features.
  • Overall pattern: Accuracy rose monotonically with context size and did not saturate at the largest context tested.
  • Results: 98.6% on MNIST placed TabPFN v3 just behind a trained CNN and ahead of gradient-boosted trees, while the language task reached 88% and tuned trees edged past it.
  • Results: Tiny ImageNet was the exception: raw pixels never reached an accuracy significantly over 10%.
  • Few-shot behavior: In the few-shot regime, TabPFN v3 reached 22% from one MNIST example per digit and remained competitive on language recognition without tuning of its own.
  • Interpretation: The authors suggest that pretraining on causal structures may help TabPFN learn relationships between raw features and labels without handcrafted techniques such as convolution.

1. TabPFN v3 configuration

TabPFN v3 uses default preprocessing and ensembles to handle inputs beyond its soft limits, but its feature-sampling cap leaves part of Tiny ImageNet unseen.

  • 1. TabPFN v3 configuration: TabPFN v3 disables input-size checks because all experiments exceed at least one soft limit: 500 features or 10,000 context samples.MNIST and Tiny ImageNet exceed the feature limit, while all tasks exceed the sample limit at their largest context sizes.
  • 1. TabPFN v3 configuration: The default wide-table strategy samples at most 200 features per estimator and increases the ensemble toward feature coverage, capped at 32 estimators.Features are drawn without replacement from a shuffled pool and consumed round-robin across estimators.
  • 1. TabPFN v3 configuration: 48% of Tiny ImageNet pixels are never sampled because 32 estimators provide only 6,400 feature slots for 12,288 columns.The round-robin sampling scheme guarantees that exactly 6,400 distinct raw pixels are selected before the cap is reached.

2. MNIST

The MNIST evaluation compares in-context TabPFN v3 with conventionally trained CNN and LightGBM models under shared data splits and size-dependent training procedures.

  • 2. MNIST: TabPFN v3 receives 784 raw pixel columns without preprocessing and predicts labels purely in-context from training rows in a single forward pass.The MNIST split fixes a 5,000-image test set and draws training examples from the disjoint remainder.
  • 2. MNIST: The CNN uses two convolutional layers, two fully connected layers, 420k parameters, standardized pixels, Adam, dropout, and 100 training epochs.The fixed epoch count is used because test accuracy plateaus once the training set is fit.
  • 2. MNIST: LightGBM uses the same 784 columns with training-size-scaled regularization, fixed boosting settings, and feature and row subsampling that vary by sample size.Its minimum child-sample setting is reduced at very small training sizes so splits remain possible.

3. Differentiating French from German words

French and German words are reduced to normalized ordinal letter sequences, with deliberate accent removal and truncation choices that affect both representation and label information.

  • 3. Differentiating French from German words: Words are lowercased, transliterated, stripped of remaining diacritics, deduplicated within language, and represented using normalized Latin letters.German umlauts and eszett become ae, oe, ue, and ss before other diacritics are removed.
  • 3. Differentiating French from German words: Each word becomes 15 ordinal columns, mapping a→1 through z→26, padding unused positions with 0, and truncating longer words.The length 15 cutoff corresponds to the 95th percentile of cleaned-word lengths.
  • 3. Differentiating French from German words: Removing accents prevents near-trivial French identification from diacritics and makes the reported accuracies a lower bound on language separability.Both models must classify from Latin-letter orthography alone.
  • 3. Differentiating French from German words: Truncation and transliteration affect languages asymmetrically: 7.5% of German words exceed 15 characters versus 0.6% of French words.Padding therefore weakly encodes word length, while accent stripping merges forms and contributes to cross-language collisions removed during deduplication.
  • 3. Differentiating French from German words: TabPFN v3 processes the 15 ordinal columns without further preprocessing, using one estimator because the feature count is below its per-estimator budget.The configuration satisfies ⌈15/200⌉ = 1.
  • 3. Differentiating French from German words: LightGBM is independently tuned at each training size and encoding with 40-trial randomized search and stratified five-fold cross-validation on training data only.The search includes small minimum child-sample values to avoid collapse to the majority class at small sample sizes.

4. Tiny ImageNet

Tiny ImageNet tests TabPFN v3 on 100-class natural images represented as raw pixel tables, comparing direct pixels with a training-set PCA reduction and exposing a small-sample constraint.

  • 4. Tiny ImageNet: The evaluation uses a balanced 100-class subset, with 64 × 64 RGB images encoded as 12,288 raw pixels without normalization or resizing.The 5,000-image test set comes from official validation data and training images come from the disjoint official training split.
  • 4. Tiny ImageNet: TabPFN v3 evaluates raw pixels in-context and also tests PCA-200, which fits 200 components on each training set so every estimator covers all components.Direct raw-pixel processing leaves about 48% of pixels unsampled under the default ensemble cap.
  • 4. Tiny ImageNet: Each image is treated as a 12,288-dimensional row vector, and training images are stacked into an N_train × 12,288 matrix for PCA processing.The representation preserves the raw-pixel table formulation before dimensionality reduction.
  • 4. Tiny ImageNet: PCA-200 retains the eigenvectors associated with the 200 largest covariance eigenvalues, computed once per run from the training set.The selected eigenvectors are stacked as columns of V and used to project test images.
  • 4. Tiny ImageNet: The PCA variant omits the 1-per-class condition because extracting 200 nonzero eigenvectors requires at least 200 training rows.This is a data-size constraint on the PCA variant rather than on the raw-pixel evaluation.
Loading 2608.22594v1…