Source-linked AI summary
Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, Yongbin Li
TL;DR
Table-based reasoning degrades on huge tables and complex questions with scattered information. The paper uses LLMs to decompose evidence and questions, and reports stronger results across three benchmarks, including performance above humans on TabFact.
Problem
Existing table-based reasoning methods struggle with huge tables and complex questions whose required information is scattered across different places.
Method
DATER uses LLMs to extract relevant table rows and columns and applies parsing-execution-filling with SQL to separate logical steps from numerical computation.
Results
DATER outperforms competitive baselines on TabFact, WikiTableQuestion, and FetaQA, and exceeds human performance on TabFact; with Codex, it reaches 85.6% TabFact accuracy.
Takeaways & Limitations
Decomposed evidence and questions support effective table-based reasoning while making returned results tractable through generated sub-evidence and sub-questions.
Takeaways & Limitations
Evidence decomposition extracts row and column indexes as a whole without modeling fine-grained alignment with the question's chain-of-thought.
Abstract
from arXiv · showhide
Table-based reasoning has shown remarkable progress in combining deep models with discrete reasoning, which requires reasoning over both free-form natural language (NL) questions and structured tabular data. However, previous table-based reasoning solutions usually suffer from significant performance degradation on huge evidence (tables). In addition, most existing methods struggle to reason over complex questions since the required information is scattered in different places. To alleviate the above challenges, we exploit large language models (LLMs) as decomposers for effective table-based reasoning, which (i) decompose huge evidence (a huge table) into sub-evidence (a small table) to mitigate the interference of useless information for table reasoning; and (ii) decompose complex questions into simpler sub-questions for text reasoning. Specifically, we first use the LLMs to break down the evidence (tables) involved in the current question, retaining the relevant evidence and excluding the remaining irrelevant evidence from the huge table. In addition, we propose a "parsing-execution-filling" strategy to alleviate the hallucination dilemma of the chain of thought by decoupling logic and numerical computation in each step. Extensive experiments show that our method can effectively leverage decomposed evidence and questions and outperforms the strong baselines on TabFact, WikiTableQuestion, and FetaQA datasets. Notably, our model outperforms human performance for the first time on the TabFact dataset.
1 Introduction
Table-based reasoning combines natural-language and tabular reasoning but struggles with huge tables and complex questions. DATER addresses these challenges by decomposing evidence and questions, with strong benchmark results and improved interpretability.
- Table-based reasoning supports fact verification and question answering but requires textual, numerical, and logical reasoning across natural language and tables.
- Large tables can exceed LLM token limits and introduce irrelevant information that interferes with reasoning.
- DATER decomposes huge tables into relevant small sub-evidence by predicting related row and column indexes with an LLM and prompts.
- Its parsing-execution-filling strategy decomposes complex questions into logical and numerical sub-questions using intermediate SQL.
- The framework's generated sub-evidence and sub-questions make returned results tractable to some extent.
2 Related Work
Related work spans executable-language methods, table pre-training, large-language-model reasoning, and question decomposition. These approaches improve table reasoning but leave challenges in semantic coverage, scalability, and generalization across tasks.
- Traditional methods use executable languages such as SQL and SPARQL, but struggle with text semantics inside tables and free-form cell content.
- Table pre-training methods encode tables and text through objectives such as masked-cell recovery, SQL-execution imitation, and reasoning-oriented pre-training.
- DATER is presented as a framework overview in which an LLM probes sub-evidence and sub-questions before a reasoner produces the final answer.
- Large language models show arithmetic, commonsense, and symbolic reasoning abilities, while chain-of-thought uses intermediate steps for complex tasks.
- Question decomposition has evolved from manually designed rules to neural models, with rule-based methods requiring expert-crafted features and rules.
3 Problem Formulation and Notations
The paper formalizes each table-based reasoning instance with a table, a natural-language question, and an answer. It considers fact verification and question answering as the two target tasks.
- Each table-based reasoning instance consists of a table T, a natural-language question Q, and an answer A.
- A table T contains Row_T rows and Col_T columns, with v_i,j denoting the content of cell (i,j).
- In fact verification, A is a Boolean value indicating whether the input statement is true or false.
- In question answering, A is a natural-language sequence answering the question described by the input statement.
4 Method
DATER uses in-context learning to decompose both tables and complex questions before final table-based reasoning. Its pipeline extracts relevant rows and columns, converts logical sub-questions into executable SQL, and fills numerical answers reliably.
- 4.1 In-context Learning: In-context prompting examples provide task demonstrations for evidence decomposition, question decomposition, and final reasoning.The prompts are manually constructed small sets of examples used during in-context learning.
- 4.2 Evidence Decomposer: DATER predicts relevant row and column indexes to reduce a complete table to question-specific sub-evidence.This aims to exclude irrelevant table content and focus reasoning on essential evidence.
- 4.3 Question Decomposer: The parsing-execution-filling strategy decomposes complex natural-language questions into logical and numerical sub-questions using SQL as an intermediate bridge.It extends chain-of-thought decomposition by separating logical steps from numerical computation.
- 4.3.1 Reliable Sub-questions: Abstract logic questions mask numerical spans, which are converted into SQL queries and executed on the evidence to obtain reliable values for filling placeholders.For example, execution produces the values “6” and “8” for Minnesota home and away games.
- 4.4 Jointly Reasoning: The final reasoner combines decomposed sub-evidence with reliable sub-questions to predict the final answer.The framework uses these intermediate representations in a subsequent joint reasoning step.
5 Experimental Setup
The evaluation covers three table-based reasoning benchmarks spanning fact verification, compositional table question answering, and free-form table question answering. Metrics are selected according to each task’s answer format.
- 5.1 Datasets: DATER is evaluated on TabFact, WikiTableQuestion, and FetaQA test or validation sets without fine-tuning on their training data.The datasets contain 2,024, 2,381/4,344, and 2,003 samples in the reported evaluation splits, respectively.
- 5.2 Evaluation Metrics: TabFact measures binary classification accuracy, while WikiTableQuestion uses denotation accuracy to verify exact agreement with gold answers.TabFact evaluates whether statements are true or false; WikiTableQuestion evaluates predicted answer denotations.
- 5.2 Evaluation Metrics: FetaQA evaluates complete long-form answers using BLEU, ROUGE-1, ROUGE-2, and ROUGE-L.Its free-form answer format differs from the short-phrase outputs of TabFact and WikiTableQuestion.
5.3 Implementation Details
The implementation uses GPT-3 Codex with task-specific prompting examples and self-consistency decoding to obtain consistent in-context reasoning results.
- 5.3 Implementation Details: GPT-3 Codex (code-davinci-002) serves as the large language model in the experiments.The final reasoning step uses 4, 2, and 6 prompting examples for TabFact, WikiTableQuestion, and FetaQA, respectively.
5.4 Baselines
DATER is compared with both fine-tuning methods requiring task-specific training and LLM-based in-context methods that do not require fine-tuning. The baselines include representative table reasoning and program-generation systems.
- 5.4 Baselines: The baseline suite contains fine-tuning methods and LLM-based in-context learning methods.The two categories differ in whether they require task-specific downstream training.
- 5.4 Baselines: Fine-tuning baselines include Table-BERT, LogicFactChecker, ReasTAP, PASTA, TableFormer, and TaCube.These methods use approaches such as table linearization, semantic parsing, synthetic pretraining, positional encoding, and precomputed numerical results.
- 5.4 Baselines: LLM-based baselines include Codex, which directly generates final answers, and Binder, which generates programming-language programs.These baselines perform in-context learning without fine-tuning.
5.5 Main Results
Dater achieves strong results across table-based reasoning benchmarks, with gains over competitive baselines and human performance surpassed on TabFact.
- 85.6% accuracy with Codex as reasoner, 13.0% above Codex without evidence and question decomposition.
- 93.0% versus 90.8% accuracy when Dater decomposition is injected into PASTA, a 2.2% improvement.
- Dater outperforms human performance for the first time on TabFact.
- 65.9% test accuracy on WikiTableQuestion, surpassing Binder by 4.0%.
- Dater achieves better results than T5 and Codex on FetaQA.
5.6 Ablation Study
The ablation and case studies examine how Dater’s evidence and question decomposition contribute to table-based reasoning and interpretability.
- Ablation Study: The ablation study removes evidence or question decomposition across TabFact, WikiTableQuestion, and FetaQA, including simple and complex questions.
- Evidence Decomposition: Dater accurately selects relevant columns by using semantic and commonsense knowledge from a powerful LLM.
- Question Decomposition: Dater decomposes complex questions into simpler sub-questions using parsing-execution-filling and generates SQL queries that retrieve correct evidence information.
- Question Decomposition: Without question decomposition, Codex produces reasoning inconsistent with the evidence when handling complex questions.
5.8 Analysis of Sub-evidence
Evidence decomposition substantially reduces table size while maintaining better performance, narrowing the information presented to the reasoner.
- 3x smaller sub-evidence than original evidence across the three datasets, while achieving better performance.
- 56 versus 164 average table cells on WikiTableQuestion after evidence decomposition.
- Similar reductions in average table cells occur on TabFact and FetaQA.
6 Conclusion
Dater uses LLM-based decomposition of evidence and questions to improve table-based reasoning, with strong benchmark results and a stated limitation in evidence alignment.
- Dater decomposes structured evidence into relevant sub-evidence by predicting row and column indexes with an LLM and prompts.
- The parsing-execution-filling strategy uses SQL to separate complex questions into logical and numerical computation.
- Dater achieves significantly better results than competitive fine-tuning-based and LLM-based baselines across three benchmark datasets.
- Dater outperforms human performance for the first time on TabFact.
- Evidence decomposition extracts row and column indexes as a whole without modeling the question’s chain-of-thought characteristics.