PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding

Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, Tomas Pfister

arXiv:2401.04398v2cs.CL

TL;DR

Table reasoning requires models to interpret interactions between free-form questions and semi-structured data, while textual reasoning chains are not ideally suited to complex tables. Chain-of-Table represents intermediate reasoning steps as evolving tables produced by iterative operations, achieving state-of-the-art performance across WikiTQ, TabFact, and FeTaQA with multiple LLMs.

  • Problem

    Textual or code-based reasoning chains are not ideally suited to complex tables, motivating better ways to leverage tabular data during reasoning.

  • Method

    Chain-of-Table prompts LLMs to iteratively generate table operations, transform the latest table, and use the evolving operation chain and table states as intermediate reasoning steps.

  • Results

    Chain-of-Table achieves state-of-the-art performance on WikiTQ, TabFact, and FeTaQA across PaLM 2, GPT-3.5, and LLaMA 2; with PaLM 2, it improves End-to-End QA by 8.69% on TabFact.

  • Takeaways & Limitations

    Evolving tables provide structured intermediate information that helps LLMs dynamically plan tabular reasoning and generate more accurate, reliable answers.

  • Takeaways & Limitations

    The evaluation does not compare Chain-of-Table with Dater using Codex because Codex is no longer publicly available.

Abstract

from arXiv · show

Table-based reasoning with large language models (LLMs) is a promising direction to tackle many table understanding tasks, such as table-based question answering and fact verification. Compared with generic reasoning, table-based reasoning requires the extraction of underlying semantics from both free-form questions and semi-structured tabular data. Chain-of-Thought and its similar approaches incorporate the reasoning chain in the form of textual context, but it is still an open question how to effectively leverage tabular data in the reasoning chain. We propose the Chain-of-Table framework, where tabular data is explicitly used in the reasoning chain as a proxy for intermediate thoughts. Specifically, we guide LLMs using in-context learning to iteratively generate operations and update the table to represent a tabular reasoning chain. LLMs can therefore dynamically plan the next operation based on the results of the previous ones. This continuous evolution of the table forms a chain, showing the reasoning process for a given tabular problem. The chain carries structured information of the intermediate results, enabling more accurate and reliable predictions. Chain-of-Table achieves new state-of-the-art performance on WikiTQ, FeTaQA, and TabFact benchmarks across multiple LLM choices.

1 INTRODUCTION

Table understanding matters because tables encode information through row–column interactions, yet generic text or code reasoning can struggle with complex tabular structure. Chain-of-Table addresses this by iteratively applying operations that evolve the table and preserve intermediate reasoning results.

  • Table understanding supports question answering and fact verification but is difficult because meaning depends on interactions between rows and columns.
  • Generic reasoning and program-aided reasoning often represent steps as text or code, which can fail on complex tables requiring accurate parsing.
  • CHAIN-OF-TABLE extends Chain-of-Thought by representing intermediate reasoning steps as transformed tables.
  • At each step, the LLM dynamically generates an operation and arguments, which are executed to enrich or condense the table.
  • Experiments evaluate CHAIN-OF-TABLE on WikiTQ, TabFact, and FeTaQA using PaLM 2, GPT-3.5, and LLaMA 2.
  • The framework achieves state-of-the-art performance on WikiTQ, TabFact, and FeTaQA.

2 RELATED WORK

Prior work improves table understanding through specialized pretraining, prompting-based reasoning chains, and executable programs, but difficult tables remain challenging for single-pass methods. Chain-of-Table differs by dynamically adapting table operations and reasoning chains to each input.

  • Fine-tuned models learn tabular structure through specialized embeddings, attention mechanisms, or table-cell reconstruction objectives.
  • Prompting methods such as Chain-of-Thought, Least-to-Most, and DecomP construct reasoning chains whose later steps use earlier results.
  • Program-aided methods generate executable Python, SQL, or hybrid programs, but single-pass generation leaves them reasoning over static tables in difficult cases.
  • Unlike Dater’s fixed row and column selections, CHAIN-OF-TABLE dynamically generates broader operation chains based on each table and question.

3 CHAIN-OF-TABLE REASONING

CHAIN-OF-TABLE treats table transformations as explicit reasoning steps: an LLM selects operations and arguments using the question, current table, and operation history, then executes them iteratively. The resulting table contains intermediate reasoning information used to produce the final answer.

  • Problem Formulation: Given a table-question pair, the framework predicts an answer for question-answering or fact-verification tasks.
  • Framework: CHAIN-OF-TABLE dynamically plans atomic operations such as adding columns, selecting rows or columns, grouping, and sorting.
  • Overview: Each iteration selects an operation from the table, question, and operation history, generates its arguments, executes it, and records the operation.
  • Overview: The table is updated until an ending tag is generated, after which the resulting table is queried for the final answer.
  • Dynamic Planning: DynamicPlan uses the latest intermediate table, prior operation chain, and question to select the next operation from an operation pool.
  • Argument Generation: GenerateArgs produces arguments for the selected operation from the current table, operation, and question before programmatic execution creates the next intermediate table.
  • Reasoning Chain: The operation chain acts as a proxy for tabular reasoning, with intermediate tables storing and presenting results for the final query.

4 EXPERIMENTS

The experiments evaluate CHAIN-OF-TABLE across three table-understanding benchmarks, multiple LLM backbones, operation-chain lengths, table sizes, and query budgets. It consistently outperforms comparison methods while degrading more gracefully on complex inputs and using fewer generated samples.

  • Experimental Setup: The evaluation covers WikiTQ, FeTaQA, and TabFact using PaLM 2, GPT-3.5, and LLaMA 2.WikiTQ and FeTaQA assess table-based question answering, while TabFact assesses binary fact verification.
  • Overall Results: CHAIN-OF-TABLE significantly outperforms generic and program-aided reasoning methods on TabFact and WikiTQ across PaLM 2, GPT-3.5, and LLaMA 2.The comparison follows the official evaluation pipeline and includes methods such as Chain-of-Thought, Text-to-SQL, Binder, and Dater.
  • Overall Results: 8.69% on TabFact and 6.72% on WikiTQ are the reported PaLM 2 End-to-End QA improvements from CHAIN-OF-TABLE.Vanilla Chain-of-Thought instead decreases WikiTQ performance in this setting.
  • Operation-Chain Analysis: CHAIN-OF-TABLE surpasses Chain-of-Thought and Dater across all operation-chain lengths, with margins reaching 11.6% and 7.9%, respectively.Performance generally declines as required operation count increases, but CHAIN-OF-TABLE declines more gradually, including from four to five operations.

5 CONCLUSION

CHAIN-OF-TABLE uses tabular structure to express intermediate thoughts during table-based reasoning. It dynamically plans operation chains according to each input table and question.

  • CHAIN-OF-TABLE leverages tabular structure to express intermediate thoughts for table-based reasoning.
  • The framework dynamically plans an operation chain based on the input table and its associated question.

A.1 INTRODUCTION

The framework uses a set of five commonly used table operations, with operations for adding results and selecting relevant rows or attributes. These operations support constructing tabular reasoning chains.

  • CHAIN-OF-TABLE adopts five table operations commonly used in SQL and DataFrame development.The framework can accommodate additional operations, which are left for future work.
  • f_add_column() adds a new column for storing intermediate reasoning or computational results.
  • f_select_row() selects relevant rows to locate context needed for the question.
  • f_select_column() selects relevant columns corresponding to attributes needed to answer the question.

A.2 ABLATION STUDY

The ablation study removes each predefined operation in turn to assess its contribution. Every operation contributes to performance, with task-dependent differences in which operations matter most.

  • Five leave-one-out variants each remove one predefined operation from the operation pool.The remaining four operations are available for constructing operation chains.
  • Removing any operation decreases performance, indicating that all five operations contribute to CHAIN-OF-TABLE’s final state-of-the-art performance.
  • f_select_row() and f_select_column() have the largest contribution on TabFact, while f_group_by() contributes most on WikiTQ.
  • Different tasks require different operations, supporting dynamically customized operation chains.

B EXPERIMENTS OF CHAIN-OF-TABLE ON FETAQA

On FeTaQA, CHAIN-OF-TABLE improves free-form question-answering performance across all reported metrics, although the measured gains are marginal. Low ROUGE scores can nevertheless coincide with correct generated answers.

  • The improvement over baseline methods is marginal on FeTaQA.
  • Low ROUGE scores can occur even when generated FeTaQA answers are correct.The paper attributes this mismatch to ROUGE’s n-gram text-similarity design and its limited sensitivity to some in-context-learning improvements.

C INFERENCE PARAMETERS AND NUMBER OF DEMO SAMPLES OF CHAIN-OF-TABLE

CHAIN-OF-TABLE uses a shared set of in-context demonstrations across datasets, with all demonstrations drawn from training data and held out from testing.

  • 29 annotated samples are used across the different datasets and operations.
  • The same demonstration samples can introduce an operation across different datasets, such as f_add_column in DynamicPlan.
  • Tables 7, 8, and 9 report the LLM parameters and demonstration counts for WikiTQ, TabFact, and FeTaQA, respectively.

D TABULAR FORMAT ENCODING COMPARISON

The study adopts PIPE encoding to isolate tabular Chain-of-Thought gains, then compares it with HTML, TSV, and Markdown representations on WikiTQ.

  • D TABULAR FORMAT ENCODING COMPARISON: PIPE encoding is adopted to decouple tabular Chain-of-Thought gains from table-formatting choices.
  • D TABULAR FORMAT ENCODING COMPARISON: Additional experiments compare PIPE with HTML, TSV, and Markdown using End-to-End QA on WikiTQ with PaLM 2.
  • D TABULAR FORMAT ENCODING COMPARISON: Table 10 reports the outcomes of the tabular format encoding comparison on WikiTQ with PaLM 2.
  • E.1 DynamicPlan: DynamicPlan receives the latest intermediate table, its question, and the operation history as prompting context.
  • E.1 DynamicPlan: DynamicPlan generates a complete operation chain but uses only the first generated operation, because later operations may not reflect the updated table.
  • E.2 GenerateArgs: GenerateArgs uses the latest intermediate table, its question, and the selected operation to generate that operation's arguments.
  • E.2 GenerateArgs: The listed atomic operations include adding columns, selecting rows or columns, grouping, and sorting.
  • E.3 Query: Query uses the resulting table from Chain-of-Table with the question to generate the answer.

F IMPLEMENTATION DETAILS OF BASELINE METHODS

Baseline implementations use official code or prompts where available, while the appendix documents prompting examples for table operations and QA settings.

  • Text-to-SQL and Binder use official open-source code and prompts, while Dater uses its official open-source implementation and prompts.
  • The baseline implementations replace OpenAI Codex with publicly available GPT 3.5, PaLM 2, and LLaMA 2 backbones.
  • End-to-End QA, Few-Shot QA, and Chain-of-Thought prompts are documented through Figures 16, 17, and 18.
  • DynamicPlan prompts demonstrate atomic operations, complete-chain generation, and generation for an input table and question.
  • GenerateArgs parses LLM-generated arguments for selected operations using predefined templates and regular expressions.
  • f_select_row retains rows relevant to the question, while f_select_column filters out useless columns.
  • f_add_column adds columns, f_group_by groups values, and f_sort_by orders numerical, date, or string columns in either direction.
  • The documented prompts provide table content and task-specific instructions before requesting an answer.
Loading 2401.04398v2…