PaperlayerView exact revision on arXiv

Source-linked AI summary

Tab-CoT: Zero-shot Tabular Chain of Thought

Ziqi Jin, Wei Lu

arXiv:2305.17812v1cs.CL

TL;DR

Zero-shot prompting makes it challenging to inject explicit structure into reasoning. Tab-CoT prompts LLMs to generate a table as they reason, and the authors report that it outperforms traditional CoT in zero- and few-shot settings.

  • Problem

    Injecting explicit structure into the reasoning process is challenging under zero-shot prompting.

    Source: §2 · p. 2

  • Method

    Tab-CoT uses a table-filling procedure with rows representing reasoning steps and columns contributing details within each step.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    Tab-CoT outperforms traditional CoT techniques in zero- and few-shot settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Tab-CoT uses a two-dimensional table structure to represent step-by-step reasoning more explicitly than a linear chain of thought.

    Source: §1 · p. 2

    The approach applies to language models pretrained with tables, and its effectiveness may depend on the task and the level of structured reasoning required.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The chain-of-though (CoT) prompting methods were successful in various natural language processing (NLP) tasks thanks to their ability to unveil the underlying complex reasoning processes. Such reasoning processes typically exhibit implicitly structured steps. Recent efforts also started investigating methods to encourage more explicitly structured reasoning procedures to be captured. In this work, we propose Tab-CoT, a novel tabular-format CoT prompting method, which allows the complex reasoning process to be explicitly modelled in a highly structured manner. Despite its simplicity, we show that our approach is capable of performing reasoning across multiple dimensions (i.e., both rows and columns). We demonstrate our approach's strong zero-shot and few-shot capabilities through extensive experiments on a range of reasoning tasks.

1 Introduction

Tab-CoT introduces table-filling as a structured chain-of-thought approach, using tables to represent reasoning across rows and columns. The authors report that it outperforms traditional CoT in zero- and few-shot settings.

1 Introduction

  • Tab-CoT models structured reasoning through a table-filling procedure, generating the table while reasoning and extracting the answer from it.Unlike work that extracts information from existing tables, Tab-CoT generates its table as part of the reasoning process.Source: §1 · pp. 1–2
  • The table's two-dimensional structure supports inference across both rows and columns, unlike conventional one-dimensional text.The introduction describes information flowing in two dimensions and reasoning horizontally and vertically at the same time.Source: §1 · p. 2
  • In the example, zero-shot CoT generates 140 words, while Tab-CoT generates 28.The authors describe Tab-CoT's output as more organized and concise in this example.Source: §1 · p. 2
  • The authors present Tab-CoT as the first method to use tables in a chain-of-thought process, to their knowledge.It combines tabular structured reasoning with large language models to generate answers.Source: §1 · p. 2
  • Tab-CoT outperforms traditional CoT techniques in zero- and few-shot settings.The introduction reports this as the outcome of extensive experiments.Source: §1 · p. 2

2 Related Work

Prior work supplied reasoning structures through few-shot examples, staged prompting, and external tools. The authors identify the difficulty of injecting structure in zero-shot prompting as motivation for generating table-structured outputs.

2 Related Work

  • Few-shot chain-of-thought examples provide a solution format that models can follow and decompose difficult tasks into smaller steps.The cited account describes this as improving performance across multiple datasets.Source: §2 · p. 2
  • Program Aided Language Models use few-shot examples containing executable Python code so LLMs can use a Python shell for mathematical reasoning.This lets the model focus on mathematical reasoning rather than numerical calculations.Source: §2 · p. 2
  • Least-to-Most prompting decomposes a complex problem into subquestions and solves them sequentially using earlier answers.The passage contrasts its more restricted reasoning structure with zero-shot CoT and notes that tools such as calculators or Python shells can aid arithmetic computation.Source: §2 · p. 2
  • The authors identify zero-shot prompting as making it challenging to inject structure into generation, motivating a scheme for structured table outputs.The related-work passage frames this as the motivation for Tab-CoT.Source: §2 · p. 2

3 Tab-CoT

Tab-CoT prompts an LLM to fill a table as it reasons, with rows and columns organizing the process. The method separates table generation from answer extraction and illustrates reasoning across both dimensions.

Tab-CoT

  • Replacing the table’s pipe delimiter with commas disrupted the model’s pattern capture, indicating that table formatting matters for this reasoning setup.The paper reports that the model failed to capture patterns after the delimiter change.Source: §3 · p. 3
  • A demonstration shows the model handling vertical reasoning despite an incorrect entry in the same row, and combining horizontal and vertical reasoning when the generated value is 13 rather than 11.The paper presents the latter as evidence of simultaneous reasoning across both dimensions.Source: §3 · pp. 3–4

Table Generation Prompt

  • Tab-CoT uses a table-generation prompt to make the model complete a table row by row, with each row representing a reasoning step.The prompt supplies column headers, and the columns contribute details within each step.Source: §3 · p. 4

Answer Extraction Prompt

  • After generating the table, Tab-CoT adds an answer-extraction prompt because the final answer may not be in the last cell.The prompt “the answer is” asks the model to extract the final answer from the table.Source: §3 · p. 4

Structure-Promoting Table Scheme

  • The main header assigns columns to steps, subquestions, reasoning processes, and intermediate results.The scheme is written as “|step|subquestion|process|result|”.Source: §3 · p. 4

4 Experimental Setup

The experiments evaluate two GPT-3-family models on arithmetic, symbolic, and commonsense reasoning tasks. The arithmetic results use zero-shot evaluation and a shared answer-extraction prompt across methods.

Experimental Setup

  • The arithmetic results compare methods in the zero-shot setting using the same answer-extraction prompt.Source: Table 2 · p. 5

Tasks and Datasets

  • Six arithmetic reasoning datasets are used: SingleEq, AddSub, MultiArith, GSM8K, AQUA-RAT, and SVAMP.The paper describes these as standard datasets widely used in the community.Source: §4 · p. 5
  • Additional evaluations cover the symbolic reasoning tasks Last letter and Coin Flip, and the commonsense tasks CommonsenseQA and StrategyQA.The paper also examines how column-name specificity affects performance in an ablation study.Source: §4 · p. 5

5 Results

Tab-CoT outperformed conventional CoT in most zero-shot arithmetic tasks, while its results varied with the table scheme, model, and reasoning task. Additional experiments and ablations examine those differences across zero- and few-shot settings.

5.1 Main Results

  • Tab-CoT with the code-based model outperformed the best conventional CoT model on 5 of 6 arithmetic tasks, with a 2.2% average improvement.Within Tab-CoT, code outperformed text by 4.0% on average, leading to the best performance among the tested configurations.Source: §5.1 · p. 5

5.2 Importance of Scheme Design

  • Removing any column from the four-column scheme reduced performance; removing step caused the largest drop, while removing subquestion lowered average performance by 5.4%.The step column orders rows sequentially, and subquestion structures the vertical sequence of instructions.Source: Table 4 · p. 6, §5.2 · p. 6

5.4 Few-shot Tab-CoT

  • Few-shot Tab-CoT outperformed few-shot CoT by 1% on average, with a 6.5% difference on SVAMP.The authors suggest the larger SVAMP improvement may relate to the selection of few-shot examples.Source: Table 5 · p. 6, §5.4 · p. 6

5.3 Effectiveness of Self-Consistency

  • Majority voting across three prompts using different table schemes improved overall performance, even though one prompt was expected to perform worse.The authors describe this as evidence that combining table schemes helps improve the approach’s overall robustness.Source: §5.3 · p. 6

5.5 Case Studies

  • Case studies found code produced more concise, straightforward reasoning than text, whose outputs included extra steps, ambiguous subquestions, and an incorrect reasoning order.Text generated five steps in one example, compared with three for code.Source: Table 6 · p. 7, §5.5 · p. 7

5.6 Additional Experiments

  • In symbolic reasoning, task-specific schemes beat zero-shot CoT on both tasks, but greater scheme specificity did not always improve accuracy.A task-specific Coin Flip scheme largely disabled vertical reasoning, while the general scheme enabled both vertical and horizontal reasoning and led to significantly better results.Source: Table 7 · p. 8, §5.6 · p. 8, Figure 4 · p. 8
  • Tab-CoT had the highest average accuracy on the commonsense tasks, but did not significantly improve over standard prompting in few-shot settings; on GPT-3.5, it beat conventional zero-shot CoT.Many GPT-3.5 responses to “Let’s think step by step” already contained a table.Source: Table 8 · p. 8, §5.6 · p. 8, Table 9 · p. 8

5.7 Ablation Studies

  • The 13B code-cushman-001 model did not reach 10% performance except on AQUA; removing any column from the proposed scheme also lowered performance.The size comparison was with the 175B code-davinci-002 model.Source: §5.7 · p. 9, Table 11 · p. 9

6 Discussion

Tab-CoT models reasoning in a table, making the process more structured and supporting reasoning across rows and columns. The discussion also highlights its readable output and the relative ease of crafting table prompts.

6 Discussion

  • Tab-CoT’s generated table is described as organized and as making the reasoning process easier to follow.The discussion points to Table 6 for this observation.Source: §6 · p. 9
  • Conventional zero-shot CoT tables under GPT-3.5 mostly organize question-related information, while Tab-CoT tables model reasoning steps.The authors report this distinction from their observations.Source: §6 · p. 9
  • Tab-CoT explicitly models a more structured reasoning process, with horizontal reasoning across rows and vertical reasoning across columns.The table structure supports both dimensions.Source: §6 · p. 9
  • Crafting a table-generation prompt involves choosing concise header names without needing to address grammar.The authors say this is less cumbersome than selecting among diverse natural-language prompts.Source: §6 · p. 9
  • The authors argue that table schemes are natural prompts suited to zero-shot learning under current state-of-the-art LLMs.Source: §6 · p. 9

7 Conclusion

The conclusion reports Tab-CoT’s effectiveness across zero-shot and few-shot reasoning settings and describes its structured table output as a way to inspect intermediate thought processes. It identifies scheme automation and integrations with calculators or task-specific supervision as future directions.

7 Conclusion

  • Tab-CoT shows competitive results on arithmetic reasoning tasks in both zero-shot and few-shot settings.Source: §7 · p. 9
  • The authors report comprehensive experiments across reasoning tasks and settings that identify method benefits and an optimal way to use it.Source: §7 · p. 9
  • Future work includes automating scheme selection and integrating external calculators or task-specific supervision in zero-shot and few-shot settings.Source: §7 · p. 9
  • Tab-CoT provides a structured decomposition of intermediate thought that may help people observe how LLMs decompose complex problems.The authors believe it may help reveal mechanisms associated with complex LLM behaviors.Source: §7 · p. 9

Limitations

The stated limitations concern the model pretraining required by Tab-CoT and variation in its effectiveness across task types.

Limitations

  • Tab-CoT applies to language models pretrained with tables, a capability that may not be included in all models.Source: Limitations · p. 10
  • The authors note that table pretraining may be absent in some language models, especially smaller ones.Source: Limitations · p. 10
  • Limited improvement on commonsense reasoning suggests Tab-CoT’s effectiveness may depend on the task and the level of structured reasoning required.Source: Limitations · p. 10

A One-shot Reasoning on Symbolic Reasoning

The one-shot evaluation covers Coin Flip and Last Letter. The reported results say that adding one few-shot sample significantly boosts performance on both tasks with a general table scheme.

A One-shot Reasoning on Symbolic Reasoning

B Additional Few-shot Results

The additional few-shot results report a slight average-accuracy advantage on commonsense reasoning. Additional arithmetic case studies present errors made by the method.

B Additional Few-shot Results

  • Under few-shot evaluation on commonsense reasoning tasks, the method performs slightly better in average accuracy.The results are reported in Table 12.Source: B Additional Few-shot Results · p. 11

C Additional Case Studies

D Few-Shot Samples

The section provides few-shot samples for arithmetic and other reasoning tasks, along with one-shot samples for symbolic reasoning. The arithmetic examples organize solutions into subquestions, processes, and results.

D Few-Shot Samples

Loading 2305.17812v1…