Source-linked AI summary
Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey
Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socolinsky, Srinivasan Sengamedu, Christos Faloutsos
TL;DR
LLM research on tabular data spans prediction, synthesis, question answering, and table understanding, but the field lacks a comprehensive comparison of its techniques, metrics, datasets, models, and optimization approaches. This survey consolidates that literature through taxonomies, task-focused comparisons, references, and future research directions, while identifying robustness and broader methodological limitations.
Problem
Research on LLMs for tabular data covers diverse tasks, but lacks a comprehensive review comparing techniques, metrics, datasets, models, and optimization approaches.
Method
The survey builds taxonomies and comparisons of tabular-data LLM methods, metrics, datasets, and models across prediction, synthesis, question answering, and table understanding.
Results
The survey consolidates recent progress and reports that table information, prompt design, inference strategies, and data curation affect LLM performance across tabular tasks.
Takeaways & Limitations
The review provides task-specific recommendations, benchmark references, code and dataset links, and an overview of open research directions for tabular-data LLMs.
Takeaways & Limitations
Current LLM research on tabular data remains primarily surface-level, with limited understanding of class imbalance, order invariance, hallucination, fairness, preprocessing, and interpretability.
Abstract
from arXiv · showhide
Recent breakthroughs in large language modeling have facilitated rigorous exploration of their application in diverse tasks related to tabular data modeling, such as prediction, tabular data synthesis, question answering, and table understanding. Each task presents unique challenges and opportunities. However, there is currently a lack of comprehensive review that summarizes and compares the key techniques, metrics, datasets, models, and optimization approaches in this research domain. This survey aims to address this gap by consolidating recent progress in these areas, offering a thorough survey and taxonomy of the datasets, metrics, and methodologies utilized. It identifies strengths, limitations, unexplored territories, and gaps in the existing literature, while providing some insights for future research directions in this vital and rapidly evolving field. It also provides relevant code and datasets references. Through this comprehensive review, we hope to provide interested readers with pertinent references and insightful perspectives, empowering them with the necessary tools and knowledge to effectively navigate and address the prevailing challenges in the field.
1 Introduction
This section introduces tabular data, its structural challenges, and the growing use of LLMs for prediction, synthesis, and table understanding. It positions the survey as a comprehensive taxonomy of methods, metrics, datasets, and future research directions.
- LLM applications: Researchers are applying LLMs to tabular prediction, quantitative reasoning, data generation, question answering, and table understanding.These applications extend earlier efforts to integrate tabular data with neural networks for NLP and data management tasks.
- Survey contributions: The survey organizes LLM techniques into application taxonomies and systematically compares datasets, metrics, methodologies, and models across tasks.It also provides recommendations, code and dataset references, benchmark methods, and future directions concerning bias, hallucination, numerical representations, interpretability, and evaluation.
- Tabular data characteristics: Tabular data organizes heterogeneous features into rows and columns and is widely used across finance, medicine, business, agriculture, and education.Its features may be categorical, numerical, binary, or textual, with dense or sparse representations.
- Tabular data characteristics: Real-world tabular datasets present sparsity, missing values, class imbalance, preprocessing dependencies, feature correlations, order invariance, and limited prior structural knowledge.Preprocessing may involve scaling, encoding, imputation, and outlier removal; improper choices can cause information loss or other distortions.
- Traditional and deep learning methods: Traditional GBDT ensembles remain state-of-the-art for tabular classification, while deep learning offers advantages for implicit representation learning and sequential data.Tree-based models are also noted for high performance, training efficiency, ease of tuning, and interpretability.
- Future directions: Future research directions include mitigating bias and hallucinations, improving numerical representations, capacity, benchmarks, interpretability, workflows, fine-tuning, and downstream performance.The survey frames these directions as responses to remaining challenges in LLM-based tabular data modeling.
2 Key techniques for LLMs’ applications on tabular data
LLM applications for tabular data use shared techniques including serialization, table manipulation, prompt engineering, and context management, with fine-tuning treated as application-specific. These choices affect performance, cost, scalability, and robustness across tabular tasks.
- 2.1 Serialization: Tabular data must be converted into text, embeddings, or other serialized representations before sequence-to-sequence LLMs can process it.Common text formats include JSON, lists, delimited values, HTML, XML, and human-readable sentences; table encoders provide numerical representations.
- 2.1 Serialization: LLM performance depends on serialization format: markup formats often improve table understanding, while DFLoader and JSON benefit fact-finding and transformation tasks.HTML and XML can outperform X-separated formats for GPT models but consume more tokens.
- 2.1 Serialization: Traditional list and text templates outperform LLM-generated sentence serialization for few-shot classification, partly because generated descriptions may hallucinate data or feature values.Among evaluated LLM serializers, larger and more complex models performed better, but remained disadvantaged on this task.
- 2.2 Table Manipulations: Large tables require compaction or retrieval because context limits, quadratic self-attention cost, degraded information access, and API pricing constrain long prompts.Proposed responses include truncation, predefined constraints, and retrieval of relevant tables, rows, columns, or cells.
- 2.2 Table Manipulations: Adding schemas, statistics, references, and term explanations can improve accuracy, but table size may help little and header hierarchy can hurt performance.Documentation disambiguating column names, value consistency, coverage, and granularity improved GPT-4 accuracy by 28.9%.
- 2.2 Table Manipulations: LLMs are sensitive to table structure and prompting: they show structural bias under transposition, vulnerability to adversarial manipulation, and substantial gains from carefully selected examples.Removing one-shot examples decreased overall accuracy by 30.38%, while manually curated examples exceeded random selection by 14.7 F1 points.
3 LLMs for predictions
This section surveys LLM-based tabular prediction methods across preprocessing, fine-tuning, target augmentation, and applications including time series and medical prediction. It emphasizes serialization and prompting choices, benchmark datasets, inference-only approaches, and reported limitations.
- Methodology: LLM-based prediction methods are organized around preprocessing, fine-tuning, and target augmentation for feature-based and time-series tabular data.Preprocessing modifies inputs, target augmentation modifies outputs, and fine-tuning adapts the model.
- Preprocessing: Text-based serialization commonly converts tables into structured strings, human-readable sentences, or schema-based prompts containing summaries and examples.Prompt engineering may add task-specific cues, relevant samples, dataset background, and task descriptions.
- Prompt engineering: Naturally occurring instructions significantly enhance TABLET’s zero-shot F1 performance, although LLMs may ignore instructions and still produce prediction failures.The instructions were paired with examples for a medical diagnosis task.
- Prompt engineering: LLM-generated instructions do not significantly improve performance, while TabLLM reports that LLM-generated serialization is not faithful and may hallucinate.TABLET instead revises templates generated from a simple model’s important features.
- Inference-only prediction: Inference-only prediction uses LLMs without fine-tuning, but TABLET reports that an XGBoost-weighted KNN approach surpasses Flan-T5 11b with similar examples and instructions.Summary Boosting is effective on smaller datasets but performs poorly with many continuous variables.
- Medical prediction: Medical LLM predictors capture long-term dependencies and unstructured clinical notes, while CPLLM reports significant improvements over the state of the art across tested disease-prediction tasks.The surveyed medical models nevertheless focus on a small fraction of ICD codes, while Meditab aims to create a medical foundation model.
4 LLMs for tabular data generation
This section categorizes LLM-based tabular synthesis into causal and masked language-modeling methods, describing their training, prompting, and evaluation strategies. The methods target data augmentation, missing-value imputation, class rebalancing, privacy, and data quality.
- Motivation: Tabular synthesis supports augmentation, missing-value imputation, class rebalancing, privacy protection, and preservation of statistical properties.LLMs are used to capture patterns and relationships in serialized tabular data.
- Method taxonomy: Most surveyed synthesis methods fine-tune LLMs on corresponding tables, whereas CLLM uses GPT4 knowledge to augment low-data training without fine-tuning.CLLM filters generated samples using confidence and uncertainty metrics based on classifier learning dynamics.
- Causal language modeling: GReaT serializes tabular records into meaningful text, fine-tunes GPT-2 or GPT-2 distill, and uses feature-order permutation before generation.This is described as the first CLM-based table generative method in the survey.
- Causal language modeling: TAPTAP combines table pretraining and machine-learning-generated labels to produce higher-quality training samples than GReaT.TabuLa instead starts from a randomly initialized model and iteratively fine-tunes it on successive tabular data.
- Generation process: CLM generation uses feature-name, one name-value pair, or related preconditioning strategies to prompt new samples, while MLM supports arbitrary prompts through masking.MLM’s masking procedure also addresses missing data by learning from masked values.
- Evaluation: Synthetic-data evaluation spans low-order statistics, high-order fidelity and diversity, privacy preservation through DCR, and machine-learning utility through MLE.The survey identifies these as four evaluation dimensions.
5 LLMs for table understanding
The survey reviews datasets, models, and methods for tabular QA, fact verification, table reasoning, numerical QA, and related table-understanding tasks. It reports that large LLMs can perform complex reasoning, while model scale, fine-tuning, and prompting materially affect results.
- Scope: The section covers QA, fact verification, and table reasoning, excluding database manipulation, management, and integration tasks not typically classified as QA.The survey organizes this coverage around datasets, trends, and methods.
- Datasets: Popular table QA datasets include FetaQA, WikiTableQuestions, HybridQA, and SQA, with FetaQA requiring deeper reasoning and free-form answer generation.WikiTableQuestions emphasizes retrieving short-form answers, whereas FetaQA integrates retrieved information into generated responses.
- Datasets: Table-understanding resources span pretraining, conversational QA, table classification, Text2SQL, table NLG, table NLI, and domain-specific numerical reasoning.Examples include TaBERT, TAPAS, HybriDialogue, FEVEROUS, Spider, WikiSQL, BIRD, ToTTo, InfoTabs, TabFact, AIT-QA, and TAT-QA.
- Models and methods: GPT-3.5 and GPT-4 are widely used and often perform complex table reasoning effectively with prompt engineering such as chain-of-thought.These models were not specifically optimized for table-based tasks.
- Numerical QA: GPT-4 with chain-of-thought significantly outperforms other LLMs on DOCMATH-EVAL, while open-source models lag behind.The comparison includes LLaMA-2, Vicuna, Mistral, Starcoder, MPT, Qwen, and other open-source systems.
- Model scale: On WebTableQuestions, the 6.7B GPT-3 model achieved only half the scores of the 175B model, while models up to 6.7B were nearly random on TabFact.These findings indicate a strong association between model size and performance in the reported evaluations.
- Fine-tuning: Fine-tuning improves performance across tabular tasks, with the largest gains in zero-shot settings; PASTA scored 93.00% on TabFact versus 85.60% for GPT-3 Codex.PASTA was pretrained on 1.2 million synthesized WikiTables items for sentence–table cloze tasks.
- Fine-tuning: Inference-based GPT-4 methods DIN-SQL and DAIL-SQL surpassed previous fine-tuned smaller models on Spider.Fine-tuning is less common in QA than in prediction and generation, partly because general LLMs can perform QA off the shelf.
6 Limitations and future directions
The survey identifies limitations involving numerical and categorical representation, benchmarking, tabular-specific behavior, fairness, hallucination, interpretability, accessibility, and adaptation strategy. It proposes more specialized representations, standardized evaluation, practical pipelines, and deeper investigation of these issues.
- Numerical representation: LLM tokenization can fragment integers and make basic mathematical operations depend on memorization rather than algorithmic processing.Digit-level tokenization may improve numerical understanding but greatly increases input dimensionality, limiting practicality for large datasets and many features.
- Categorical representation: Excessive categorical columns can exceed context limits, increase cost, prune data, and hinder interpretation when feature encoding or metadata is poor.The survey calls for better categorical feature encoding.
- Standard benchmarks: Reported performance differs across papers even for the same dataset, method, and task, preventing reliable cross-method classification benchmarks.For Blood, TabLLM performance is reported as 0.70 by GTL and 0.66 by UniPredict.
- Tabular-specific challenges: Research on LLMs for tabular data remains surface-level, with limited understanding of behaviors on class-imbalanced datasets and interactions between training and inference data.The survey prioritizes deeper investigation of tabular-specific behaviors.
- Bias and fairness: LLMs can inherit social biases that widen fairness gaps between subgroups, while flipping in-context labels narrows gaps at the cost of predictive performance.The survey notes that prompting alone may not sufficiently mitigate inherent bias.
- Hallucination: Hallucinations can reduce reasoning performance and have critical consequences in medical tabular prediction, while auditing and reflection-based correction are iterative and difficult to deploy.The survey calls for more efficient and practical hallucination-handling methods.
- Model interpretability: LLM-based tabular systems often lack interpretability, and the effectiveness of Shapley-based or other explanations remains unexplored for prediction and table understanding.Only a few systems, such as TabLLM, expose justifications for outputs.
- Ease of use: Most tabular LLM tools require fine-tuning or serialization, while existing preprocessing-heavy approaches remain difficult to use and deploy.The survey suggests unified pipelines with automated preprocessing and serialization as a future direction.
7 Conclusion
The survey synthesizes LLM-based modeling of heterogeneous tabular data across prediction, synthesis, question answering, and table understanding. It compares current solutions, identifies limitations and research gaps, and outlines future directions for navigating these challenges.
- The survey systematically compares datasets, methodologies, metrics, and models across major tabular-data tasks.It covers prediction, data synthesis, question answering, and table understanding.
- It examines the steps required for LLMs to ingest tabular data, including serialization, table manipulation, and prompt engineering.
- The review provides recommendations for selecting datasets and models for specific tabular-data modeling tasks.These recommendations target both machine-learning researchers and practitioners.
- Current approaches remain limited by hallucination, fairness concerns, preprocessing intricacies, and interpretability challenges.The survey discusses future research directions in response to these limitations.
- The survey aims to equip readers with references, perspectives, and knowledge for addressing challenges in this rapidly developing field.