PaperlayerView exact revision on arXiv

Source-linked AI summary

Faithful Chain-of-Thought Reasoning

Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, Chris Callison-Burch

arXiv:2301.13379v3cs.CL

TL;DR

Chain-of-Thought reasoning can produce explanations that do not faithfully reflect how a model reaches its answer. Faithful CoT translates queries into symbolic reasoning chains executed by deterministic solvers, achieving new state-of-the-art performance on 7 of 10 datasets while providing faithful explanations.

  • Problem

    Chain-of-Thought explanations may not faithfully reflect model behavior, risking misleading interpretations of how answers are produced.

  • Method

    Faithful CoT translates natural-language queries into interleaved natural-language comments and symbolic programs, then uses a deterministic external solver to derive answers.

  • Results

    Faithful CoT achieves new state-of-the-art performance on 7 of the 10 reasoning datasets while providing faithful explanations for final answers.

  • Takeaways & Limitations

    The results provide empirical evidence that faithful interpretability need not reduce performance and can instead coincide with strong accuracy.

  • Takeaways & Limitations

    The Translation stage remains opaque, so the method guarantees faithfulness for Problem Solving but not self-interpretability of reasoning-chain generation.

Abstract

from arXiv · show

While Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of complex reasoning tasks, the generated reasoning chain does not necessarily reflect how the model arrives at the answer (aka. faithfulness). We propose Faithful CoT, a reasoning framework involving two stages: Translation (Natural Language query $\rightarrow$ symbolic reasoning chain) and Problem Solving (reasoning chain $\rightarrow$ answer), using an LM and a deterministic solver respectively. This guarantees that the reasoning chain provides a faithful explanation of the final answer. Aside from interpretability, Faithful CoT also improves empirical performance: it outperforms standard CoT on 9 of 10 benchmarks from 4 diverse domains, with a relative accuracy gain of 6.3% on Math Word Problems (MWP), 3.4% on Planning, 5.5% on Multi-hop Question Answering (QA), and 21.4% on Relational Inference. Furthermore, with GPT-4 and Codex, it sets the new state-of-the-art few-shot performance on 7 datasets (with 95.0+ accuracy on 6 of them), showing a strong synergy between faithfulness and accuracy.

1 Introduction

Faithful CoT addresses standard CoT’s unreliable explanations by separating reasoning into LM-based Translation and deterministic Problem Solving. The framework guarantees that executable reasoning chains support the final answer while improving performance across diverse reasoning domains.

  • Motivation: Standard CoT does not guarantee faithfulness: its generated reasoning chain may fail to support the final answer and can misrepresent the model’s reasoning.Such unfaithful explanations may appear plausible and encourage over-trust in high-stakes applications.
  • Method: Faithful CoT decomposes reasoning into Translation and Problem Solving, with an LM producing an NL–SL chain and a deterministic solver executing it.Natural-language components break queries into simpler subproblems, while task-dependent symbolic languages include Python, Datalog, and PDDL.
  • Interpretability: Faithful CoT guarantees that its reasoning chain faithfully explains how the final answer is produced, though Translation itself is not claimed to be interpretable.The method can expose the reasoning process behind both correct and incorrect answers.
  • Empirical results: 10 reasoning datasets from 4 diverse domains were used to evaluate Faithful CoT, which empirically improves correctness over standard CoT.The reported domains include Math Word Problems, Planning, Multi-hop Question Answering, and Relational Inference.
  • Empirical results: 95.0+ accuracy was achieved on 6 of 7 datasets with GPT-4 and Codex, indicating that interpretability need not come at the cost of performance.The paper describes this as evidence of a strong synergy between interpretability and accuracy.

2 Related Work

This section defines faithfulness as accurately reflecting a model’s reasoning process, distinguishes it from human-perceived plausibility, and situates Faithful CoT among CoT-style prompting methods and related symbolic-program approaches.

  • Faithfulness: Faithfulness requires an explanation to accurately represent the reasoning process behind a model’s prediction, unlike plausibility, which concerns how convincing it is to humans.The section notes that faithfulness is also called fidelity or reliability, while plausibility is also called persuasiveness or understandability.
  • Chain-of-Thought-style prompting: Standard CoT prompts a language model to generate a reasoning chain and final answer for a complex question using in-context exemplars.Pure natural-language reasoning chains may appear plausible without guaranteeing that the final answer causally follows from them.
  • Chain-of-Thought-style prompting: Existing CoT-style prompting methods are categorized as all-at-once, ensemble-based, or modularized.All-at-once methods produce the reasoning chain and answer as one continuous string without dependencies or constraints between them; examples include Scratchpad, standard CoT, and “Let’s think step by step.”
  • Related symbolic approaches: Concurrent work generates Python programs as symbolic reasoning chains, whereas this approach is motivated by interpretability and uses structured natural-language problem decomposition.The structured decomposition is intended to be more understandable and potentially interactive for users without programming backgrounds.
  • Related symbolic approaches: The framework recasts Planning, Multi-hop QA, and Relational Inference into symbolic representations and supports multiple symbolic languages and domains.The section also highlights a more comprehensive analysis than previous work, including human evaluation of reasoning-chain correctness.

3 Method

Faithful CoT translates natural-language queries into interleaved natural-language comments and symbolic programs, then uses a task-specific deterministic solver to derive the answer. The framework applies this pipeline to math word problems, multi-hop QA, planning, and relational inference.

  • Core pipeline: Faithful CoT uses two stages: an LM translates Q into an interleaved reasoning chain C, and a deterministic solver derives A.C combines natural-language decomposition with symbolic-language programs; solving uses tools such as Python interpreters, Datalog executors, or PDDL planners.
  • Core pipeline: The framework decomposes complex queries into interdependent subproblems, represents each in a task-dependent symbolic language, and executes the resulting chain.Supported symbolic languages include Python, Datalog, and PDDL.
  • Math Word Problems: Math word problems use subquestions, dependency graphs, and rationales in CNL to generate Python code whose execution produces the numerical answer.For the example with 3 initial cars and 2 arrivals, the Python interpreter derives 5 cars.
  • Multi-hop QA: Multi-hop QA converts natural-language subquestion answers into Datalog statements and executes the program to determine the final Boolean or string answer.In the pear example, density statements yield the conclusion that a pear would not sink in water.
  • Planning and Relational Inference: Planning translates user requests into PDDL symbolic goals, which a PDDL planner solves into an action plan.Relational inference instead recursively applies transitivity rules to CSL, deriving answers such as Vincent being Gabrielle’s grandson.

4 Experimental setup

The evaluation spans math word problems, multi-hop QA, planning, and relational inference, using few-shot prompts with task-specific exemplars and official splits where available. Performance is measured primarily by final-answer accuracy against several prompting baselines, including standard CoT and Least-to-Most.

  • Evaluation protocol: The evaluation uses 6–10 exemplars per task, with official training, validation, and test splits used for selection, tuning, and evaluation unless otherwise stated.The few-shot prompts follow the exemplar count used by Wei et al. (2022).
  • Datasets: The MWP evaluation covers GSM8K, SVAMP, MultiArith, ASDiv, and AQuA, with natural-language questions and task-specific mathematical answer formats.The same 8-shot prompt is used across MWP datasets except AQuA.
  • Datasets: Planning uses SayCan, where models map natural-language household-task queries to plans composed of predefined robot actions, using 7 manually written exemplars.SayCan provides no training set for exemplar selection.
  • Datasets: Relational inference uses CLUTRR, prompting with 8 exemplars at K ∈{2, 3} and testing on remaining examples with up to 10 intermediate steps.The dataset contains multiple splits based on the number of intermediate steps required to reach the answer.
  • Metrics: Final-answer accuracy is the main metric, with exact-match rules based on nearest-integer conversion for most MWP datasets and Boolean evaluation for StrategyQA and Sports Understanding.These evaluation definitions follow previous work.
  • Baselines and models: The method is compared with standard prompting, CoT, and Least-to-Most under greedy and alternative decoding strategies, using OpenAI Codex in Section 5.Standard prompting includes only question–answer demonstrations; CoT adds a natural-language reasoning chain, while Least-to-Most decomposes the question in natural language without symbolic language.

5 Results

Faithful CoT outperforms the baselines across most datasets and domains, with especially large gains on harder reasoning tasks. The method also shows consistent gains across alternative language models, while StrategyQA remains a notable limitation.

  • Overall results: Faithful CoT outperforms standard, CoT, and Least-to-Most prompting across the vast majority of datasets and domains under both decoding strategies.Table 1 evaluates the methods on 10 reasoning datasets from 4 domains using code-davinci-002.
  • Greedy decoding: 8 of 10 datasets: Faithful CoT outperforms all baselines with greedy decoding.Relative improvements reach 14.2% on MWP, 3.4% on Planning, 6.5% on Date Understanding from Multi-hop QA, and 21.4% on Relational Inference.
  • Dataset difficulty: Faithful CoT generally achieves larger gains on harder datasets, whereas gains are less evident when CoT already performs decently on simpler datasets.The paper illustrates this contrast with MultiArith and AsDiv within MWP.
  • Limitations: StrategyQA remains far behind the baselines, while Sports Understanding is nearly saturated with Faithful CoT and Least-to-Most both reaching 99+ accuracy.Datalog prompting also makes Codex produce relevant subquestions, retrieve knowledge, and derive valid reasoning less effectively, at 17%, 10%, and 10%, respectively.
  • Generalization: 7 datasets: GPT-4 establishes new few-shot state-of-the-art results, with 95.0+ accuracy on four of five MWP and two of three Multi-hop QA datasets.Experiments with four alternative LMs show consistent gains over the baselines, supporting the empirical benefit of faithfulness.

6 Analysis

The analysis finds that Faithful CoT depends strongly on the external solver for several tasks, remains robust to exemplar selection, and usually produces valid reasoning chains when answers are correct. However, human evaluation also identifies occasional cases where the model reaches correct answers through invalid reasoning.

  • Prompt ablations: On CLUTRR, removing the nudge line causes a 31.3-point accuracy drop, whereas NL comments contribute little or slightly hurt performance on GSM8K, Date Understanding, and SayCan.The CLUTRR result compares the No NL but nudge and No NL variants; removing rationale, nudge, or subquestions each reduces accuracy.
  • Prompt ablations: Without the external solver, accuracy declines by -50.8 on GSM8K, -22.9 on Date Understanding, and -19.4 on CLUTRR, but improves by 2.9 on SayCan.SayCan’s three predefined actions may make it relatively easy, with all variants reaching around 90% accuracy.
  • Exemplar robustness: Across five random exemplar samples, mean accuracy stays close to the original, ranging from -1.5 to +1.2, while remaining 7 to 17 points above baselines except on SayCan.The reported standard deviation ranges from 1.3 to 2.9.
  • Reasoning-chain plausibility: Human evaluators find reasoning chains fully correct in more than 90% of cases for Sports, SayCan, and MWP, compared with 66.7% for StrategyQA, 87.9% for Date, and 88.0% for CLUTRR.The study evaluates chains that produce correct answers and labels them completely correct, incorrect NL, incorrect SL, flawed question, or confused.
  • Reasoning-chain plausibility: Faithful CoT generates valid reasoning chains for the vast majority of correct answers, but occasionally reaches the right answer through an incorrect chain.The paper notes that such exceptions can involve unnecessary steps or chance agreement, especially for True/False questions in StrategyQA.

7 Conclusion

Faithful CoT decomposes complex reasoning into Translation and Problem Solving, guaranteeing faithful explanations while achieving new state-of-the-art performance on 7 of 10 datasets across four reasoning problem types.

  • 7 Conclusion: Faithful CoT decomposes complex reasoning into Translation and Problem Solving, guaranteeing that the reasoning chain faithfully explains the final answer.The framework separates symbolic-chain translation from answer generation through problem solving.
  • 7 Conclusion: 7 of 10 datasets achieved new state-of-the-art performance with Faithful CoT, alongside faithful explanations for the final answers.The approach was evaluated on Math Word Problems, Multi-hop QA, Planning, and Relational Inference.

Limitations

The study’s limitations include the loss of reproducibility caused by OpenAI’s discontinuation of codedavinci-002 and incomplete faithfulness guarantees because Translation remains opaque. The role of natural-language comments also requires further study, as they may have limited performance impact but improve interpretability.

  • Reproducibility: On March 23rd, 2023, OpenAI discontinued codedavinci-002, rendering part of the results unreplicable for researchers without continued access.The discontinuation was unexpected and raises questions about using closed-source models for academic research.
  • Faithfulness scope: Faithfulness is guaranteed for Problem Solving, but the Translation stage remains opaque because how the LM generates reasoning chains is not self-interpretable.The interpretability of LM generation remains an under-explored question.
  • Faithfulness scope: Recent studies show promising early progress toward improving interpretability of LM generation, which might strengthen faithfulness in Translation.The passage cites Yin and Neubig (2022) and Sarti et al. (2023) as potential foundations.
  • Natural-language comments: Natural-language comments require further exploration because their inclusion often has little performance effect, especially on non-knowledge-intensive datasets, while improving interpretability.They can make reasoning chains more structured and understandable to end users.

Ethics Statement · A Implementation Details · B Extended Results

The section warns that plausible LM reasoning can accompany incorrect conclusions, then documents the experimental setup and notes that additional results are provided beyond the main text.

  • Ethics Statement: Plausible LM reasoning chains can accompany incorrect final answers, creating a reliability risk when LM outputs are used for reasoning.The ethics discussion frames the work as a step toward more trustworthy LM use.
  • A Implementation Details: The experiments used OpenAI GPT-3 and Codex models through the Python API from September 2022 to January 2023.The models included text-davinci-001, text-davinci-002, code-davinci-001, and code-davinci-002.
  • A Implementation Details: $0 was the inference cost per example for Codex models, while GPT-3 cost $0.01–$0.03 depending on the dataset.Codex models were in a limited beta period.
  • A Implementation Details: 2–15 seconds was the Codex inference time per example under a 150,000 tokens/minute rate limit.This timing is reported for the experimental setup described in the implementation details.
  • A Implementation Details: Temperature was 0.0 for greedy decoding and 0.4 for self-consistent decoding, with n set to 1 and 40 respectively.The remaining listed hyperparameters were max_tokens 1000, frequency_penalty 0, and presence_penalty 0.
  • A Implementation Details: Unspecified hyperparameters used the default values from OpenAI’s completions API documentation.The cited documentation was beta.openai.com/docs/api-reference/completions/create.
  • B Extended Results: The extended-results section presents additional results that did not fit into the main text.This section is explicitly framed as supplementary to the main presentation.

B.1 Examples of Unfaithful CoT Output … C Extended Analysis

The appendix illustrates why standard CoT reasoning chains may be unfaithful, then compares Faithful CoT with few-shot state-of-the-art and concurrent program-based methods. Faithful CoT achieves strong benchmark performance while emphasizing interpretable structured comments as its central contribution.

  • B.1 Examples of Unfaithful CoT Output: In StrategyQA, the chain identifies that a hummingbird outweighs a pea but still answers “yes,” while SayCan’s plan includes unnecessary “pick” and “put” operations.The SayCan chain only mentions visiting the table and counter, illustrating a mismatch between the displayed reasoning and the resulting plan.
  • B.1 Examples of Unfaithful CoT Output: Figure 7 shows unfaithful CoT outputs on GSM8K, StrategyQA, and SayCan, where answers do not follow from the displayed reasoning chains.The examples are explicitly selected as apparently unfaithful outputs; cases without known inconsistency still do not establish faithfulness because the model’s ground-truth reasoning is unknown.
  • B.1 Examples of Unfaithful CoT Output: Because CoT does not guarantee faithfulness, its reasoning chain may differ drastically from the model’s actual decision process and remain difficult for users to debug.This challenges the claim that CoT necessarily provides an interpretable window into model behavior.
  • B.2 Comparison with Few-shot SOTA: With Codex and GPT-4, Faithful CoT reaches new few-shot state-of-the-art performance on 7 of 10 datasets and exceeds 95.0 accuracy on 6.The comparison uses optimal Faithful CoT results with the best-performing language model for each dataset.
  • B.3 Empirical Comparison with Concurrent Work: PoT and PAL generate Python programs, whereas Faithful CoT additionally uses structured natural-language comments to decompose problems into inter-dependent subproblems.The empirical comparison reimplements the concurrent methods with the noNL prompt, code-davinci-002, and greedy decoding.
  • B.3 Empirical Comparison with Concurrent Work: On 6 of 10 datasets, PAL/PoT and Faithful CoT differ by less than 2.0 accuracy; PAL/PoT leads on AQuA, while Faithful CoT leads on StrategyQA, Sports Understanding, and CLUTRR.The reported pattern may indicate an advantage for Faithful CoT on tasks requiring extensive external knowledge or longer symbolic reasoning.
  • B.3 Empirical Comparison with Concurrent Work: Faithful CoT’s key contribution is interpretability: structured natural-language comments make reasoning more understandable and may let nonprogrammers debug through the decomposed subproblems.Users could potentially add, remove, or edit a natural-language subproblem instead of modifying a program.

C.1 Ablation Study … C.4 Enforcing Constraints

The appendices examine ablations, prompt-phrasing robustness, translator-model sensitivity, and generation constraints. Faithful CoT remains robust across prompt variations and models, while most added constraints reduce performance.

  • C.1 Ablation Study: The full ablation-study results accompanying Figure 5 are reported in Table 4.
  • C.2 Robustness to Prompt Phrasing: Three prompt variations permute independent subquestions, change variable names, or revise the nudge line.
  • C.2 Robustness to Prompt Phrasing: Across four applicable datasets, performance remains stable under all prompt variations and stays above every baseline.
  • C.3 Model Sensitivity: 16.1%, 11.0%, 9.4%, and 4.6% are Faithful CoT’s average relative accuracy gains over the best baseline method across the four alternative translator LMs, respectively.The evaluated models are code-davinci-001, text-davinci-002, text-davinci-003, and gpt-4, using greedy decoding on five MWP datasets.
  • C.3 Model Sensitivity: Faithful CoT outperforms all baselines on most datasets for each underlying LM and remains within <2.0 difference of the best method on the rest.
  • C.3 Model Sensitivity: 95.0+ accuracy is achieved on 4 of 5 MWP datasets with GPT-4, surpassing previous few-shot SOTA on GSM8K, SVAMP, and ASDiv.
  • C.4 Enforcing Constraints: The constraint study evaluates graph validity, no over-dependency, and no under-dependency for structured reasoning generations on MWP datasets.Graph validity requires a DAG; the other constraints respectively prohibit unused or require all mentioned variables.
  • C.4 Enforcing Constraints: Graph validity causes little to no performance change, whereas the other constraints mostly decrease and become more unstable across datasets; combining constraints lowers performance further.The effects are measured under self-consistent decoding against original results without constraints.

D Human Evaluation Details … F.1 GSM8K

The appendix details human evaluation, dataset preparation, splits, and error analysis for Faithful CoT. On GSM8K, errors are dominated by wrong natural-language subquestions, followed by wrong code and semantic misunderstandings.

  • D Human Evaluation Details: 100 reasoning chains per domain were sampled from code-davinci-002 using greedy decoding, with equal representation across datasets and programming-experienced annotators.Annotators used Python for MWP, CLUTRR, Sports, and Date, and Scala for StrategyQA.
  • D Human Evaluation Details: At least three annotators labeled each of 100 questions per domain, after receiving domain-matched examples and attention checks; majority labels formed the final analysis labels.The study used 100 annotators, compensated at $16/hr, averaging 2 minutes per question.
  • E.1 Statistics: 10 exemplars were used to rerun the Wei et al. (2022) baseline, resolving an inconsistency between the main text and appendix.The main text reported 10 exemplars, while the appendix reported 6.
  • E.2 URLs and Licenses: The evaluation used GSM8K, SVAMP, MultiArith, ASDiv, AQuA, StrategyQA, Date Understanding, Sports Understanding, and CLUTRR, with dataset licenses and sources documented.StrategyQA used the open-domain question-only setting, while Date and Sports Understanding came from BIG-Bench.
  • E.3 Data Cleaning: Manual cleaning corrected annotation issues in ASDiv, Date Understanding, Sports Understanding, and SayCan before rerunning all baselines on cleaned test sets.For ASDiv, all answer values were extracted and compared as a set rather than using only the first value.
  • E.4 Dataset Splits: GSM8K validation was formed by randomly sampling 1,000 training examples, while other datasets used their official or adapted splits for prompting and testing.SayCan used 7 manually written few-shot exemplars, and CLUTRR used training-fold exemplars with concatenated test folds.
  • F Error Analysis: 100 errors from each of four datasets were manually categorized to investigate remaining failure modes.Error samples were diversified by embedding them with text-davinci-002, spectral clustering, and importance sampling.
  • F.1 GSM8K: 49% of GSM8K errors were Wrong Subquestion, 24% Wrong Code, 12% Semantic Understanding Error, and 7% Generation Cutoff.Semantic errors included incorrect polarity for pounds lost versus pounds gained and failures to interpret quantities or temporal phrases.

F.2 StrategyQA … G Prompts

The paper analyzes Faithful CoT errors across StrategyQA, Date Understanding, SayCan, and CLUTRR, and documents dataset-specific prompting procedures, including a two-step conversion for AQuA’s multiple-choice outputs.

  • F.2 StrategyQA: Syntax (29%) is the most frequent StrategyQA error, producing invalid Datalog programs through issues such as relation redefinition, invalid comparisons, or ungrounded variables.The authors believe further Datalog pretraining could overcome these errors to a reasonable extent.
  • F.2 StrategyQA: Infinite Loop (24%) occurs when the LM generates philosophical chains of subquestions, while Wrong Subquestions (17%) introduces irrelevant subquestions.One example generated 40 subquestions, and such loops occurred much less often when prompts contained only natural-language subquestions.
  • F.2 StrategyQA: 10% of StrategyQA errors involve Wrong Reasoning, and another 10% involve Wrong Knowledge, where correct subquestions are paired with faulty inference or retrieved facts.The Wrong Knowledge example incorrectly states that Albuquerque is known as Yootó, whereas Santa Fe is known as Yootó.
  • F.3 Date Understanding: Date Understanding has 69 errors among 359 test examples, with ambiguous phrasing appearing among error categories and no observed generation errors in the annotated samples.All 69 errors were annotated.
  • F.4 SayCan: 64% of SayCan errors are Additional Subgoals that mislead the planner, while 36% are Wrong Object errors involving incorrect objects or object types.The latter includes generating actions with water instead of Pepsi when asked to bring an opened Pepsi can.
  • F.5 CLUTRR: CLUTRR errors include Inversed Relation (41%), Wrong Relation (30%), Wrong Path (12%), Wrong Gold Label (13%), and Nonexistent Relation (4%).The analysis groups cases by K, the number of steps in the gold reasoning chain, and importance-samples 100 examples.
  • G Prompts: The prompts section presents one exemplar per dataset, with full prompts available in the repository.Exemplars are provided for AQuA, GSM8K, SVAMP, MultiArith, ASDiv, StrategyQA, Date Understanding, Sports Understanding, SayCan, and CLUTRR.
  • G Prompts: AQuA prompts first produce a numerical answer, then prompt the LM to convert it into the closest multiple-choice answer choice.This differs from other math datasets because AQuA answers use a multiple-choice format instead of integers.
Loading 2301.13379v3…