PaperlayerView exact revision on arXiv

Source-linked AI summary

TabLLM: Few-shot Classification of Tabular Data with Large Language Models

Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, David Sontag

arXiv:2210.10723v2cs.CLcs.AI

TL;DR

TabLLM addresses tabular classification when labeled data are limited by converting rows into natural-language inputs for large language models. It evaluates serialization strategies and few-shot fine-tuning, finding strong zero-shot performance and competitive or superior results against neural and tree-based baselines, especially with very few examples.

  • Problem

    Tabular applications often lack enough labeled data for supervised classification, motivating methods that can exploit prior knowledge encoded in large language models.

  • Method

    TabLLM serializes tabular rows into natural-language strings, combines them with task descriptions, evaluates multiple serialization methods, and fine-tunes the LLM on labeled examples in few-shot settings.

  • Results

    TabLLM outperforms state-of-the-art baseline algorithms like XGBoost and SAINT by over 5 AUC points in the very-few-shot regime while remaining competitive with many samples.

  • Takeaways & Limitations

    TabLLM can exploit prior knowledge in LLMs for non-trivial zero-shot classification and is particularly valuable when only very little labeled data are available.

  • Takeaways & Limitations

    TabLLM has a larger computational footprint than traditional algorithms, and performance may suffer when dense rows exceed the LLM token limit.

Abstract

from arXiv · show

We study the application of large language models to zero-shot and few-shot classification of tabular data. We prompt the large language model with a serialization of the tabular data to a natural-language string, together with a short description of the classification problem. In the few-shot setting, we fine-tune the large language model using some labeled examples. We evaluate several serialization methods including templates, table-to-text models, and large language models. Despite its simplicity, we find that this technique outperforms prior deep-learning-based tabular classification methods on several benchmark datasets. In most cases, even zero-shot classification obtains non-trivial performance, illustrating the method's ability to exploit prior knowledge encoded in large language models. Unlike many deep learning methods for tabular datasets, this approach is also competitive with strong traditional baselines like gradient-boosted trees, especially in the very-few-shot setting.

1 INTRODUCTION

TabLLM applies large language models to data-efficient tabular classification by serializing rows as natural-language text and combining them with task prompts. Across benchmark datasets, it achieves strong zero- and few-shot performance, including competitiveness with tree ensembles.

  • Introduction: The method targets classification when labeled data are scarce, including domains such as healthcare, climate, and finance.Rare diseases illustrate the challenge: 10,000 rare diseases affect very few patients, limiting risk-stratification data.
  • Introduction: TabLLM serializes tabular feature names and values into natural-language strings, combines them with task-specific prompts, and uses LLM verbalizer probabilities for classification.In few-shot settings, labeled examples fine-tune the LLM with T-Few before predictions are made.
  • Introduction: TabLLM studies nine serialization techniques across ten datasets and evaluates T0 models, with additional zero-shot experiments using GPT-3.The framework is presented as a broad evaluation of LLMs for zero- and few-shot tabular classification.
  • Introduction: Simple text serialization with T0 can outperform state-of-the-art neural models and tree ensembles in zero- and few-shot settings.The paper reports particularly strong results when only very few labeled examples are available.
  • Introduction: TabLLM also addresses healthcare claims data with serialization methods designed to handle many input features.This extends the framework beyond small, simple tables to a large real-world healthcare setting.

3 METHODS

The method converts each tabular row into natural-language text, appends a task prompt, and maps LLM outputs to class labels. It compares nine serialization formats and uses labeled examples for parameter-efficient fine-tuning.

  • 3.1 TabLLM for Tabular Data Classification: A dataset is represented as labeled feature vectors with natural-language column names, while k-shot experiments fine-tune on a sampled subset of k examples.The class labels belong to a specified set of classes, and sampling is performed with replacement.
  • 3.1 TabLLM for Tabular Data Classification: The serialize function maps column names and feature values for one row to a textual representation, which is combined with a task-specific prompt.The prompt describes the classification task and follows the serialized input.
  • 3.1 TabLLM for Tabular Data Classification: LLM text outputs are mapped to valid classes through a manually specified verbalizer, while serialized labeled examples provide few-shot fine-tuning data.The verbalizer associates selected LLM output tokens with the discrete class space.
  • 3.2 Our Instantiation of TabLLM: The study compares nine serialization formats, including list and text templates, table-to-text generation, and GPT-3- or T0-generated text.The prompt itself receives no further prompt engineering, isolating serialization choices.
  • 3.2 Our Instantiation of TabLLM: Ablations remove column names or permute their associations with values to test whether correct feature semantics support classification.Permuted values and shortened lists additionally probe the use of fine-grained values and reduced information under input limits.
  • 3.2 Our Instantiation of TabLLM: The main instantiation uses T0 with 11 billion parameters and T-Few for fine-tuning, alongside T0 3B and zero-shot GPT-3 evaluations.T0 has a 1024-token limit, roughly 400 words.

4 EXPERIMENTAL SETUP

The experiments evaluate TabLLM across public tabular and healthcare claims datasets, comparing serializations, language-model settings, and established baselines under varying numbers of labeled examples. Results show strongest benefits in zero- and very-few-shot settings, while serialization differences diminish with more data.

  • Experimental datasets: Nine medium-sized public datasets were selected with at most 50,000 rows and 30 columns, textual feature names, and no derived feature values.The constraints were chosen to manage fine-tuning costs, fit T0’s token limit, and make natural-language serialization meaningful.
  • TabLLM configuration: TabLLM serializes each row into natural language with a task description and evaluates nine serialization methods using T0 models.Prompts encode each class as a textual response, with class probabilities derived from normalized token-sequence probabilities.
  • Baseline models: Baseline comparisons included logistic regression, XGBoost, LightGBM, TabNet, SAINT, NODE, and TabPFN, with hyperparameter tuning for all except TabPFN.Healthcare claims experiments included only logistic regression and LightGBM because of runtime limitations.
  • Serialization effects: Text Template performed best across serialization experiments, improving over List Template at zero shots, while differences vanished by 8 training examples.This suggests that sophisticated serializations may be unnecessary once some labeled data is available.
  • Serialization effects: LLM-generated serializations performed worse overall because they sometimes hallucinated information or failed to include all features faithfully.These issues could bias predictions and explain why they did not match template-based serializations even with many training examples.
  • Healthcare claims: On healthcare claims, List Template slightly outperformed Text Template consistently, with frequent-condition selection performing best.The result held across tasks, while concept-name choices produced no considerable performance difference.
  • Public-dataset results: TabLLM achieved nontrivial zero-shot performance on most public tasks and outperformed or matched tree ensembles through 256 examples except on Calhousing and Jungle.TabPFN remained the strongest baseline, while TabLLM often had a large advantage at lower shot counts.
  • Healthcare claims: Healthcare claims performance ranged from 0.67 AUC for Surgery to 0.71 for LoH in the zero-shot setting, improving with more examples.TabLLM outperformed logistic regression through 16–64 examples and LightGBM through 64–256 examples, depending on the task.

6 DISCUSSION

TabLLM’s sample efficiency and serialization choices matter most when labeled data are scarce, while performance and input-format differences narrow with more examples. The approach trades computational efficiency and some domain or token-limit robustness for strong few-shot performance.

  • For all datasets except Credit-g and Heart, simple List Template and Text Template serializations achieved nontrivial zero-shot performance.This indicates that TabLLM can use prior knowledge encoded in the LLM for classification.
  • When hundreds of training examples were available, less informative serializations approached the best serialization, making input format less relevant.This occurred at 256 examples for tabular datasets and 1,024 for the insurance dataset; permuted column names differed only up to 16 examples.
  • TabLLM often outperformed strong baseline models in the very-few-shot setting, but sample efficiency was highly task-dependent.Performance was worse on Blood, Credit-g, Diabetes, and Heart than on Income and Car.
  • Medical-domain performance may suffer because T0 has limited knowledge of medical procedures, motivating evaluation with medical-domain-specific or medically trained language models.The authors identify substituting T0 with such models as future work for medical prediction tasks.
  • Blood, Diabetes, and Heart may serve as public proxies for medical-domain tabular classification when private healthcare datasets are unavailable.Except in zero-shot and very-few-shot regimes, other baselines generally outperform TabLLM on these datasets.
  • TabLLM has a larger computational footprint, may be constrained by token limits, and depends on pretrained knowledge matching feature semantics and data settings.Dataset shift can make pretraining data poorly reflect the conditions under which a table was aggregated.
  • TabLLM outperformed XGBoost and SAINT by over 5 AUC points in the very-few-shot regime while remaining competitive with many samples.
  • TabLLM does not use unlabeled data, leaving co-training with tree-based models as a proposed direction for combining few-shot and ultimate performance.

8 SOCIETAL IMPACT

Applying TabLLM to sensitive tasks requires care because language models can reproduce historical biases and demand substantial computational resources. These costs may also contribute to an exclusive research environment and environmental impact.

  • Predictions for sensitive tasks such as income or health trajectories require subgroup analyses because LLMs can replicate existing biases and stereotypes.
  • LLM computing requirements risk creating an exclusive research environment and can carry significant environmental impact.

1 ADDITIONAL DATASET DETAILS

The paper details public and healthcare claims datasets, shot-sampling procedures, patient cohorts, serializations, and concept-selection experiments. Most frequent conditions consistently performed best, while alternative concept names showed no consistent improvement.

  • Experimental setup: Datasets used 80/20 train-test splits, class-balanced labeled examples, and shot counts from 0 to 512 plus the full dataset.
  • Public tabular datasets: The public benchmark collection included Jungle, Bank, Blood, California, Car, Credit-g, Diabetes, Heart, and Income classification datasets.These datasets span game outcomes, marketing, donations, housing, cars, credit risk, clinical conditions, coronary disease, and income.
  • Healthcare claims dataset: The healthcare claims data used encounter-level records with visit types, specialties, conditions, and procedures from the OMOP Common Data Model.The dataset came from a large U.S. health insurer and included 14,095 conditions and 21,184 procedures.
  • Healthcare tasks: Healthcare tasks predicted end-of-life mortality, surgical-procedure need, and hospitalization likelihood using six-month outcomes separated by a three-month leakage-prevention gap.The cohorts contained 94,972, 620,382, and 612,656 individuals, respectively.
  • Healthcare serialization: Healthcare serializations began with age, sex, and race, then organized selected concepts with associated visit dates, specialties, visit types, complaints, and hospitalization information.
  • Concept selection: Most frequent conditions consistently outperformed other concept-selection strategies, and conditions were generally more useful than procedures.This strategy even exceeded concept weights from logistic-regression models trained with 256 or 4,096 shots.
  • Alternative concept names: Alternative condition names from ICD, MEDCIN, CHV, and GPT-3 showed no consistent performance improvement, although medical jargon helped TabLLM on EoL and Surgery.

2 RUNTIME ESTIMATES FOR TABLLM

TabLLM required minutes rather than hours for the reported public-dataset experiments, with healthcare claims training taking somewhat longer because of the dataset's larger size.

  • Training on Income took less than 3 minutes for 64 examples, 30 epochs, and batch size 8.
  • Inference on 10,000 Income test examples took 2 minutes, or approximately 12 ms per example, using batch size 16.
  • Healthcare EoL training took nearly 4 minutes for 64 examples and 10 epochs, with similar times reported for the other healthcare tasks.

3 PARAMETER TUNING FOR BASELINES

The baseline models were tuned through cross-validation, parameter searches, and established hyperparameter ranges, with neural baselines selected by validation AUC-ROC.

  • Linear and tree baselines: Logistic regression used cross-validation and parameter tuning over penalty and regularization-strength settings.
  • Linear and tree baselines: Tree-based models used hyperparameter ranges from prior tabular-learning studies, discretized into a complete grid search.
  • Neural baselines: SAINT, TabNet, and NODE used prior recommended setups and ranges, with Optuna selecting parameters that maximized AUC-ROC across folds.

4 COMPARING BASELINE RESULTS TO THE LITERATURE

Comparisons with published baselines generally reproduced the literature's relative performance patterns, while some differences reflected dataset versions, metrics, or more extensive tuning.

  • Public datasets: On Bank, published balanced accuracies were 72.7 for XGBoost, 70.6 for TabNet, and 74.6 for NODE; the experiments found XGBoost better than NODE at 512 shots.
  • Public datasets: On Blood, published balanced accuracies were 62.3 for XGBoost, 64.3 for TabNet, and 50 for NODE, while the experiments found NODE better than XGBoost.
  • Public datasets: For California, published regression mean squared errors were 0.53, 0.21, 0.20, 0.35, 0.28, and 0.23 for the listed linear and neural or tree baselines.The classification experiments found XGBoost better than LightGBM, TabNet and NODE similar, and the linear model much better.
  • Public datasets: Published Car balanced accuracies were 92.4 for XGBoost, 98.7 for TabNet, and 46.1 for NODE; experiments found XGBoost and TabNet similar and NODE slightly inferior.
  • Public datasets: Credit-g experiments at 512 shots followed the published baseline trend, although their AUC results were not directly comparable with published balanced accuracies.
  • Public datasets: Published Diabetes XGBoost performance was AUC 0.828 (0.030), matching the experiments; feature selection and preprocessing raised it to AUC 0.946 (0.020) outside this work's scope.
  • Public datasets: Published Heart results used only the Cleveland cohort, whereas this work combined four sub-cohorts, limiting direct dataset comparability.The published logistic-regression result was AUC 0.923.
  • Public datasets: Published Jungle results reported balanced accuracies of 87.3 for XGBoost and 73.4 for TabNet, matching the trend in the 512-shot experiments.

5 ADJUSTING INCOME DATASET FOR INFLATION

The analysis tests whether inflation-driven distribution shift affects TabLLM’s zero-shot performance on the Income dataset. It compares results without inflation correction against results after correcting dollar values.

  • Inflation adjustment: 1.796 was the assumed inflation rate from 1994 to 2021 for the Income dataset’s dollar-valued label and features.The dataset was collected in 1994, while T0 was trained in 2021.
  • Inflation adjustment: 0.80 (0.01) was the zero-shot result without inflation correction.

6 FEATURE IMPORTANCE ANALYSIS OF TABLLM

The feature-importance analysis estimates which serialized features drive zero-shot TabLLM predictions and compares those estimates with logistic-regression-based importance measures. It applies this procedure to Income and the EoL healthcare task.

  • Importance estimation: Zero-shot TabLLM predictions were modeled with a 4-fold cross-validated L2-regularized logistic regression using serialized features as covariates.For EoL, the inputs included age, sex, race, and conditions, totaling 14,105 features.
  • Baseline comparison: For Income, TabLLM importance scores were compared with coefficients from a logistic-regression model trained on all data.The baseline logistic regression used 4-fold cross validation on an 80% training split for hyperparameter selection before refitting on all training data.
  • Baseline comparison: For EoL, TabLLM feature importance was compared with relative risk computed from feature presence and absence groups.The relative risk used the ratio of the label rate among patients with a concept to the rate among patients without it.

7 EFFECT OF USING DIFFERENT PROMPTS

The prompt-sensitivity analysis evaluates zero-shot TabLLM with multiple prompts expressing the same classification question. Across datasets, the reported prompt effects were relatively small, while some full-dataset results were limited by runtime.

  • Prompt variation: Five prompts containing the same question were constructed for every dataset in the zero-shot setting.An Income example changed a specific question about earning more than 50000 dollars per year into a broader question about earning a lot of money.
  • Prompt variation: Prompt effects were relatively small, ranging from a standard deviation of 0.00 for Jungle Chess.
  • Reported results: The tables report test AUC for competing methods across different k-shot settings, with standard deviations across five random seeds shown as subscripts.
  • Scope caveat: Some full-dataset results were omitted or limited to a single run because of TabLLM runtime limitations.

8 TASK TEMPLATES

The task templates express tabular classification problems as natural-language yes-or-no questions paired with serialized inputs and answer choices. The examples cover healthcare, finance, games, consumer decisions, and end-of-life outcomes.

  • Healthcare tasks: Healthcare templates ask whether a person receives credit, has diabetes, or has heart disease based on patient information.
  • Finance and consumer tasks: Finance and consumer templates ask whether a person earns more than 50000 dollars, subscribes to a term deposit, or considers a house block valuable.
  • Game task: The Jungle Chess template asks whether the white player wins a two-pieces endgame.
  • End Of Life Task: End-of-life templates ask whether a patient dies or needs surgery in the next nine months.
  • Likelihood of Hospitalization Task: The hospitalization template asks whether a patient is admitted to the hospital in the next nine months.

9 EXAMPLE SERIALIZATIONS

This section presents example natural-language serializations of tabular records from banking, blood donation, California housing, car evaluation, and credit datasets. The examples vary from lists and field-by-field templates to table-to-text and GPT-generated descriptions.

  • Bank Dataset: Bank records are serialized with labeled fields, including age, occupation, marital status, finances, loans, contact details, and campaign history.The examples include a 69-year-old retired, single person with tertiary education, a 2144-euro balance, and a successful previous campaign.
  • Serialization Variability: Some table-to-text outputs introduce distorted or mismatched descriptions, whereas other prose outputs preserve the record attributes more coherently.For example, the Bank Text T0 passage retains several banking attributes in prose, while table-to-text passages contain malformed statements about values and meanings.
  • Blood Dataset: Blood-donation records are expressed as lists, templated sentences, or concise prose describing recency, donation frequency, monetary volume, and time since first donation.One example states one donation, 250 c.c. donated, and 23 months for both recency and time since first donation.
  • California Dataset: California housing records are serialized using labeled numeric fields or prose covering income, age, rooms, bedrooms, population, households, latitude, and longitude.The examples represent a location with median income 3.2377, median age 32, 6597 rooms, and coordinates 34.15 and -118.01.
  • Car Dataset: Car-evaluation records use categorical field lists, templated statements, or prose describing price, doors, maintenance, passenger capacity, safety, and trunk size.The record is characterized by low buying price and maintenance costs, three doors, more than four persons, and medium safety and trunk-size ratings.
  • Credit-g Dataset: Credit-g records are serialized with categorical and numeric attributes such as account status, duration, credit history, purpose, amount, savings, employment, installment rate, residence, property, and age.The list example includes a credit amount of 1577, savings of at least 1000 DM, employment under one year, installment rate 4, and age 20.
Loading 2210.10723v2…