Source-linked AI summary
TableBench: A Comprehensive and Complex Benchmark for Table Question Answering
Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Guanglin Niu, Tongliang Li, Zhoujun Li
TL;DR
Real-world table question answering requires more complex reasoning than existing academic benchmarks capture. The paper introduces TableBench and TableInstruct, evaluates more than 30 models, and finds that TABLELLM is comparable to GPT-3.5 while GPT-4 still trails humans.
Problem
Existing academic benchmarks do not adequately capture the complex reasoning required for table question answering in industrial scenarios.
Method
The paper constructs TableBench with 886 samples spanning 18 capabilities and develops TableInstruct with three reasoning methods to train TABLELLM.
Results
GPT-4 outperforms other models on numerous TableBench tasks, TABLELLM achieves performance comparable to GPT-3.5, and humans surpass all evaluated LLMs.
Takeaways & Limitations
TableBench indicates that current open-source and proprietary LLMs still have substantial room to improve on complex tabular reasoning in real-world scenarios.
Takeaways & Limitations
The study focuses mainly on reasoning complexity, does not extensively examine the inherent complexity of tables, and excludes tabular data in image formats.
Abstract
from arXiv · showhide
Recent advancements in Large Language Models (LLMs) have markedly enhanced the interpretation and processing of tabular data, introducing previously unimaginable capabilities. Despite these achievements, LLMs still encounter significant challenges when applied in industrial scenarios, particularly due to the increased complexity of reasoning required with real-world tabular data, underscoring a notable disparity between academic benchmarks and practical applications. To address this discrepancy, we conduct a detailed investigation into the application of tabular data in industrial scenarios and propose a comprehensive and complex benchmark TableBench, including 18 fields within four major categories of table question answering (TableQA) capabilities. Furthermore, we introduce TableLLM, trained on our meticulously constructed training set TableInstruct, achieving comparable performance with GPT-3.5. Massive experiments conducted on TableBench indicate that both open-source and proprietary LLMs still have significant room for improvement to meet real-world demands, where the most advanced model, GPT-4, achieves only a modest score compared to humans.
Introduction
Existing table-question-answering benchmarks do not fully capture the complex reasoning demands of real-world tabular applications. TableBench addresses this gap with a broad benchmark, new instruction corpus, and systematic evaluation of language models.
- Motivation: LLMs have improved tabular-data interpretation through structured prompts, reasoning methods, and external programming languages, but existing benchmarks may not reflect practical complexity.These developments include in-context learning, structure-aware prompting, chain-of-thought reasoning, and SQL or Python assistance.
- TableBench: TableBench covers 18 TableQA subcategories across fact-checking, numerical reasoning, data analysis, and visualization.The benchmark is designed from real-world table applications and user demands.
- Modeling: TableInstruct covers textual, symbolic, and program-of-thought reasoning, and TABLELLM achieves performance comparable to GPT-3.5.Symbolic reasoning uses programming commands, while program-of-thought generates executable code as reasoning steps.
- Evaluation: Evaluation of more than 30 models shows that both open-source and proprietary LLMs still require substantial improvement for real-world tabular demands.GPT-4 performs strongly in many tasks, yet humans still outperform all evaluated language models.
- Resources: TableBench contains 886 human-annotated samples, while TableInstruct provides a large instruction corpus for tabular reasoning.The benchmark supports evaluation, and the instruction corpus supports model training across multiple reasoning approaches.
Construction of TableBench
TableBench is constructed from real-world tabular applications to cover diverse TableQA demands and higher reasoning complexity than existing datasets. Its benchmark and instruction corpus use systematic categorization, question generation, and multi-agent annotation.
- Scope and categories: TableBench comprises 886 samples spanning four major question categories and 18 subcategories, including fact-checking, numerical reasoning, data analysis, and visualization.The benchmark emphasizes data analysis and chart generation capabilities that previous datasets notably lack.
- Complexity: The construction process analyzes real-world tabular applications and defines task complexity by the number of reasoning steps required for problem solving.Annotation follows detailed guidelines for defining and decomposing these steps.
- Data selection: TableBench uses moderately complex tables collected from existing datasets while focusing complexity on the questions rather than table structure or size.Sources include WTQ, SQA, TabFact, FeTaQA, FinQA, and AIT-QA.
- Question generation: A self-inspiration mechanism uses seed questions, category definitions, and GPT-4-based generation to create constrained questions that are then manually annotated.The process limits initial outputs to five questions and iteratively identifies new patterns.
- Answer annotation: Three reasoning agents using TCoT, SCoT, and PoT generate candidate answers, whose agreement is checked by voting before manual review and modification.The final annotations include answers and associated reasoning details.
- Complexity: TableBench’s overall reasoning complexity is significantly greater than existing datasets, especially for data analysis and visualization questions.The comparison is presented through the benchmark’s reasoning-step analysis.
TABLELLM
The paper frames TableQA as generating answers from semi-structured tables and evaluates multiple reasoning strategies for producing those answers. TABLELLM is fine-tuned on question-answer data organized by reasoning method.
- Problem definition: TableQA maps a semi-structured table T and question Q to an answer A containing one or more values or entities.The table is represented with R rows and C columns, and A contains k answer elements.
- Reasoning setup: In-context learning supplies a task-specific instruction and examples to an LLM, enabling assessment of its tabular reasoning capabilities.The formulation treats the model, instruction, and examples as inputs for generating practical outputs.
- TCoT: TCoT derives intermediate reasoning steps or sub-goals through textual prompts before producing the final answer.The reasoning chain consists of textual inferential steps.
- SCoT: SCoT repeatedly analyzes the next move, generates Python commands, and simulates their results until reaching a conclusion.Its reasoning trace records analysis, program-generation, and result-simulation steps.
- PoT: PoT separates computation from reasoning by generating executable programming commands and using an interpreter to produce results.Unlike SCoT, PoT emphasizes actually executing the generated code in a programming environment.
- Training: TABLELLM is trained by fine-tuning all baseline-model parameters on TableInstruct, whose datasets are indexed by reasoning method and contain table questions and answers.The training objective aggregates examples across the reasoning-method datasets.
Experiments
The experiments evaluate 34 models and multiple reasoning methods on TableBench, using standardized prompting, output parsing, and answer evaluation. GPT-4 leads many tasks, while humans remain stronger and complex tabular reasoning remains difficult.
- 34 models spanning 7B to 110B parameters are evaluated, including general, code, open-source, proprietary, and supervised-finetuned models.
- GPT-4 outperforms other models across numerous tasks, while TABLELLM reaches performance comparable to GPT-3.5.
- Humans surpass all evaluated LLMs, despite advanced proprietary models showing potential on TableBench.
- Category Analysis: Fact-based reasoning is comparatively strong, whereas numerical reasoning, data analysis, and especially chart generation impose greater reasoning or coding demands.
- Reasoning Methods Analysis: Reasoning-step methods outperform direct-answer methods; TCoT is stable, PoT is strong for numerical computation but limited by code execution, and SCoT is balanced but modest.For GPT4-Turbo, the executable code ratio in PoT is 78.67% under single-generation execution without code correction.
- The ROUGE-L evaluation may not fully capture flexible LLM outputs, so GPT-4 and human judgments are used to test agreement among evaluation methods.
Further Analysis
Further analyses show that reasoning-method behavior depends on model size and output parsing, while TableInstruct remains effective even when substantially reduced. Smaller models favor direct prompting, and reasoning methods retain improvement potential as parsing succeeds more often.
- Instruct Following Analysis: Small models perform better with Direct Prompting, while PoT and SCoT are harder because of insufficient code generation and iterative symbolic reasoning demands.
- Instruct Following Analysis: At low parsing ratios, some models score higher as parsing decreases, indicating strong table understanding alongside weaker instruction-following.
- Instruct Following Analysis: Direct Prompting faces a performance ceiling, whereas reasoning-based methods, especially PoT, show potential to improve with higher parsing rates.
- Data Efficiency of TableInstruct: With 60% of TableInstruct samples, the model retains over 90% of full-dataset performance.
- Data Efficiency of TableInstruct: Fewer than 4,000 samples enable Llama-3-8B to surpass Qwen1.5-70B on the dataset, while the full corpus reaches performance comparable to GPT-3.5.
Related Work
Earlier TableQA benchmarks established semantic and factual question answering, but many rely on table-cell content or limited query forms. Later datasets broadened the task toward free-form and numeric reasoning to better reflect real-world questions.
- WTQ, SQA, and TabFact use question-answer pairs based on HTML tables from Wikipedia and focus heavily on specific cell content.
- These earlier datasets do not fully represent the multidimensional queries found in real-world scenarios.
- ToTTo, OTTQA, and FeTaQA introduce free-form answers that go beyond explicit table content.
- FinQA and AIT-QA extend the literature toward numeric reasoning tasks.
Conclusion
The paper introduces TableBench to evaluate broad tabular skills across realistic, complex capabilities and supplies TableInstruct for training TABLELLM. Although TABLELLM approaches ChatGPT-level performance, GPT-4 remains substantially behind humans, underscoring persistent real-world difficulty.
- TableBench contains 886 question-answer pairs spanning 18 distinct tabular capabilities to bridge academic benchmarks and real-world applications.
- TableInstruct enables TABLELLM to achieve performance comparable to ChatGPT.
- GPT-4 still lags significantly behind human performance on TableBench, highlighting the difficulty of real-world tabular tasks.
Limitations
The study focuses on the reasoning complexity of table questions, leaving the inherent complexity of tables less extensively explored. It also excludes tabular data presented in image formats.
- The study mainly examines table-question reasoning complexity rather than the inherent complexity of the tables themselves.The data collection adopts moderate table complexity instead of specifically designing for structural or large-table complexity.
- Tabular data in image formats, despite appearing in real-world applications, is outside the paper’s scope.