PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models are few(1)-shot Table Reasoners

Wenhu Chen

arXiv:2210.06710v2cs.CL

TL;DR

The paper addresses whether LLMs can perform table reasoning through few-shot in-context learning, a capability not established by prior text-reasoning results. It evaluates prompting-based LLMs across table QA, generation, and fact verification tasks, finding strong reasoning performance and a useful generic baseline, while recognizing limitations from table size, model scale, cost, and remaining performance gaps.

  • Problem

    The paper asks whether LLMs can perform complex table-based reasoning with few-shot in-context learning despite not being optimized for table tasks.

  • Method

    The paper evaluates direct prediction, chain-of-thought, and self-consistency prompting on WikiTableQA, FetaQA, TabFact, and FEVEROUS instead of fine-tuning.

  • Results

    LLMs achieve strong cross-task performance with 1 or 2 demonstrations, including 48.8% on WikiTableQuestions, 78.8% on TabFact, and over 30% gains over T5-large on selected FetaQA human-evaluation aspects.

  • Takeaways & Limitations

    LLMs with chain-of-thought prompting can serve as a simple, generic baseline for table-related research and can produce useful intermediate reasoning forms.

  • Takeaways & Limitations

    The approach remains below state-of-the-art, is costly because superior performance requires scaling, and smaller models remain weak at table reasoning.

Abstract

from arXiv · show

Recent literature has shown that large language models (LLMs) are generally excellent few-shot reasoners to solve text reasoning tasks. However, the capability of LLMs on table reasoning tasks is yet to be explored. In this paper, we aim at understanding how well LLMs can perform table-related tasks with few-shot in-context learning. Specifically, we evaluated LLMs on popular table QA and fact verification datasets like WikiTableQuestion, FetaQA, TabFact, and FEVEROUS and found that LLMs are competent at complex reasoning over table structures, though these models are not pre-trained on any table corpus. When combined with `chain of thoughts' prompting, LLMs can achieve very strong performance with only a 1-shot demonstration, even on par with some SoTA models. We show that LLMs are even more competent at generating comprehensive long-form answers on FetaQA than tuned T5-large. We further manually studied the reasoning chains elicited from LLMs and found that these reasoning chains are highly consistent with the underlying semantic form. We believe that LLMs can serve as a simple yet generic baseline for future research. The code and data are released in https://github.com/wenhuchen/TableCoT.

1 Introduction

The paper asks whether LLMs can perform complex table reasoning through few-shot in-context learning despite lacking table-specific pretraining. Across table QA and verification tasks, prompting—especially chain-of-thought—yields strong performance and supports a generic baseline.

  • Research motivation: Traditional SQL- or SPARQL-based approaches impose rigid table assumptions and struggle with textual content and irregular web tables.Understanding web tables requires both structured reasoning and textual reasoning.
  • Research motivation: The paper investigates whether LLMs can solve complex table-based reasoning tasks with few-shot in-context learning.The motivation is that LLM reasoning successes on text tasks had not established their ability on tables.
  • Approach and evaluation: The experiments evaluate direct prediction, Chain of Thoughts, and Chain of Thoughts with self-consistency on WikiTableQA, FetaQA, TabFact, and FEVEROUS.The models receive demonstrations rather than task-specific fine-tuning.
  • Results: 48.8% on WikiTableQuestions and 78.8% on TabFact are achieved with only 1 or 2 demonstrations, on par with some near-SoTA models.These results are reported for LLM-based prompting across table reasoning benchmarks.
  • Results: More than 30% improvement over T5-large is observed for GPT-3 on FetaQA correctness and adequacy in human evaluation.The paper studies long-form answer quality beyond automatic BLEU scoring.
  • Analysis and limitations: Correct predictions have rationales highly consistent with ground-truth semantic forms, while errors include failures on huge tables and simple symbolic operations.The models perform operations such as maximum, minimum, counting, comparison, addition, and difference.
  • Implication: The authors propose LLMs with CoT as a simple and general baseline for future table-related research.They also suggest direct prediction or auxiliary annotation of complex intermediate forms.

2 Related Work

Prior table reasoning systems commonly synthesize executable logical forms, while newer text-to-text and few-shot LLM approaches seek greater generality. Chain-of-thought prompting extends LLM reasoning to complex table tasks without downstream fine-tuning.

  • Traditional table reasoning: Semantic-parsing systems traditionally synthesize SQL or SPARQL to interact with tables.These approaches require values within columns to follow consistent data types, an assumption often violated by web tables.
  • Traditional table reasoning: Irregular web tables containing unnormalized free-form text motivate methods that combine table reasoning with language understanding.The paper frames textual reasoning inside tables as necessary for robust web-table understanding.
  • LLM-based approaches: BINDER uses Codex to synthesize SQL and execute logical forms, falling back to language-model answers when execution fails.This contrasts with the paper’s more direct few-shot language-model approach.
  • LLM-based approaches: Few-shot prompting lets LLMs predict task outputs from natural-language descriptions and a small number of examples without fine-tuning.Scaling model size, data, and computation is described as crucial for this ability.
  • LLM-based approaches: Chain-of-thought prompting provides reasoning-chain exemplars so LLMs can solve difficult unseen reasoning tasks.Self-consistency later samples diverse reasoning paths and selects the majority answer.

3 Method

The method linearizes tables and combines them with few-shot demonstrations in prompts for table reasoning. It compares direct prediction, chain-of-thought reasoning, and self-consistency decoding under a bounded input context.

  • Prompt construction: Tables are linearized and concatenated with a few demonstrations that show the language model how to predict an unseen test output.The prompt format is described in Figure 2.
  • Prompting variants: The method compares Direct Prediction, Chain of Thoughts, and Chain of Thoughts with self-consistency decoding.These variants differ in whether intermediate rationales and multiple sampled paths are used.
  • Prompting variants: Self-consistency generates five diverse reasoning paths and selects the answer receiving the most votes.Majority voting is applied after sampling the paths from the LLM.
  • Input constraints: Input tables are truncated to the first 22 rows and 8 columns, with each cell limited to its first 10 words.This keeps the input within 2000 tokens and enables analysis of input-token-length effects.

4 Experimental Results

Across four table-reasoning datasets, few-shot LLM prompting achieves strong results without task-specific fine-tuning, especially with chain-of-thought reasoning. Performance reflects both broad reasoning competence and clear sensitivity to table structure and size.

  • WikiTableQuestions: GPT-3 achieves 26% EM on WikiTableQuestions with direct prediction, rising to roughly 46% with two demonstrations and over 48.8% with Codex.These results indicate diverse table reasoning despite no table-specific training.
  • FetaQA: GPT-3 outperforms fine-tuned T5-large by more than 30% in correctness, adequacy, and faithfulness on FetaQA.Human evaluation also places GPT-3’s output almost on par with average human performance.
  • TabFact: On TabFact, GPT-3 reaches 72% accuracy directly, over 77% with CoT, and 78.8% with Codex, only 2% below TAPAS.LLM-generated intermediate rationales require no training annotations.
  • FEVEROUS: FEVEROUS performance is weaker than TabFact because its irregular structure contains many segments and subtables that challenge GPT-3.The best-performing model falls roughly between UnifiedSKG-base and UnifiedSKG-large.
  • Model Scaling: 6.7B GPT-3 achieves only half the WikiTableQuestions performance of the 175B model, while models with 6.7B or fewer parameters are nearly random on TabFact.The authors interpret this scaling pattern as evidence that table reasoning ability emerges with model scale.
  • Table Size and Limitations: GPT-3’s accuracy decreases almost monotonically as table size grows and degrades nearly to random guessing beyond 1000 tokens.The paper identifies large-table generalization as a major limitation, alongside unpredictable randomness.

5 Conclusion

The paper finds that current LLMs can perform complex table reasoning despite not being optimized for table-based tasks, particularly with chain-of-thought prompting.

  • LLMs can perform complex table reasoning despite not being optimized for table-based tasks.

Limitations

The approach remains far from state-of-the-art performance and becomes costly because strong table-reasoning ability requires scaling to larger models.

  • The approach remains far from state-of-the-art performance, limiting its use as an alternative.
  • Strong performance requires scaling up, while smaller models remain weak at table reasoning.
Loading 2210.06710v2…