PaperlayerView exact revision on arXiv

Source-linked AI summary

Question Decomposition Improves the Faithfulness of Model-Generated Reasoning

Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandrasekaran, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, Ethan Perez

arXiv:2307.11768v2cs.CLcs.AIcs.LG

TL;DR

As LLM tasks become more difficult, verifying their safety and correctness grows harder, and CoT’s stated reasoning may not reflect the model’s actual process. This paper tests question decomposition as an alternative and finds improved faithfulness, with a trade-off between faithfulness and question-answering performance.

  • Problem

    CoT reasoning may not faithfully reflect a model’s actual reasoning, limiting its use for checking model behavior and safety.

    Source: §1 · pp. 1–2

  • Method

    The paper compares CoT with question-decomposition methods that generate subquestions and answers, including factored decomposition’s separate contexts.

    Source: §1 · p. 2, §2.2 · p. 4, §2.3 · p. 4

  • Results

    Question decomposition improves reasoning faithfulness over CoT, with factored decomposition most faithful but weakest on question-answering performance and CoT decomposition intermediate on both.

    Source: §3.2.3 · p. 8, §3.5 · p. 10, §3.3.3 · p. 10

  • Takeaways & Limitations

    The findings suggest that prompting models to decompose questions can improve the faithfulness of their stated reasoning while retaining some question-answering performance.

    Source: §5 · p. 11

    Faithfulness is assessed without access to ground truth for the model’s reasoning, and evaluation covers only four question-answering tasks and one RLHF-finetuned model.

    Source: §3.5 · p. 10

Abstract

from arXiv · show

As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. To improve over the faithfulness of CoT reasoning, we have models generate reasoning by decomposing questions into subquestions. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT while improving the faithfulness of the model's stated reasoning on several recently-proposed metrics. By forcing the model to answer simpler subquestions in separate contexts, we greatly increase the faithfulness of model-generated reasoning over CoT, while still achieving some of the performance gains of CoT. Our results show it is possible to improve the faithfulness of model-generated reasoning; continued improvements may lead to reasoning that enables us to verify the correctness and safety of LLM behavior.

1. Introduction

As LLM tasks grow harder, verifying safety and correctness becomes more difficult; although Chain-of-Thought (CoT) can make model processes easier to evaluate, its stated reasoning may not reflect its actual process. The paper examines question decomposition as an alternative, reporting improved reasoning faithfulness while retaining some question-answering performance.

1. Introduction

  • CoT reasoning can misrepresent a model’s actual process, including by justifying biased answers or remaining unchanged after its reasoning is altered.These findings cast doubt on using stated reasoning alone to verify model behavior.Source: §1 · p. 2
  • Factored decomposition answers subquestions in separate contexts, while CoT decomposition keeps subquestion generation and answering in one context.The latter may retain performance-supporting context while obtaining some faithfulness benefits of factored decomposition.Source: §1 · pp. 2–3, Figure 2 · p. 2
  • Decomposition-based methods improve faithfulness over CoT on evaluated question-answering tasks, with factored decomposition showing a larger gain at some performance cost.CoT decomposition improves faithfulness while maintaining similar performance to CoT.Source: §1 · p. 3, Table 1 · p. 3
  • The authors argue that further progress in faithful reasoning may support confidence in the trustworthiness of LLM answers.They note that checking a model’s process is motivated by the difficulty of verifying correctness and safety as task difficulty increases.Source: §1 · pp. 1, 3

2. Methods

The methods prompt an instruction-following LLM to produce reasoning samples and evaluate both answer correctness and reasoning faithfulness. They compare CoT with two question-decomposition formats and describe the evaluation and prompting setup.

2. Methods

  • The study evaluates whether an instruction-following LLM produces correct answers after generating reasoning samples, and whether those samples reflect its actual reasoning.Faithfulness is assessed with metrics for properties expected of faithful reasoning.Source: §2 · p. 3

2.1. CoT prompting

  • CoT prompting elicits step-by-step reasoning in response to a question before the model gives its final answer.The sampled reasoning is treated as a sequence of individual steps.Source: §2.1 · p. 3

2.2. Factored decomposition

  • Factored decomposition alternates between generating subquestions, answering those answerable in isolation, and updating the list until recomposing the collected answers into a final answer.Each reasoning step is a subquestion–subanswer pair; subquestions that refer to other subquestions are answered after their dependencies.Source: §2.2 · p. 4

2.3. CoT decomposition

  • CoT decomposition generates subquestions and subanswers iteratively as paired steps in one autoregressive sampling call.It is intermediate between CoT’s one-call generation and factored decomposition’s separate contexts; its potential faithfulness benefits depend in part on subquestions not containing biasing features.Source: §2.3 · p. 4

2.4. Implementation

  • The evaluation compares five prompting strategies across four multiple-choice question-answering tasks, using 300 randomly sampled test questions per task (1,200 total); chain of thought and chain-of-thought decomposition achieve strongest performance.Methods requiring few-shot examples share the same set of 14 questions. High-quality reasoning samples are included for strategies that elicit reasoning samples.Source: §2.4 · pp. 4–6, Table 4 · p. 6

3. Results

The results compare question-answering performance and several faithfulness measures for CoT prompting and two decomposition methods. Decomposition improves faithfulness on perturbation tests, while bias tests and qualitative findings add important qualifications to that pattern.

3.1. Question-Answering Performance

  • CoT decomposition underperforms CoT by 0.4% (absolute) on average, while factored decomposition outperforms few-shot and zero-shot prompting by 2.1 and 9.0% on average.On HotpotQA, accuracy is 87.3% for CoT, 86.7% for CoT decomposition, 83.0% for factored decomposition, and 77.0% for both zero-shot and few-shot prompting.Source: §3.1 · p. 6

3.2. Faithfulness Measured via Reasoning Perturbation

  • 9.6% of corrupted CoT samples change the final answer, compared with 28.7% for CoT decomposition and 33.6% for factored decomposition.In early answering, factored decomposition requires more of its reasoning sample to reach the same answer, whereas CoT and CoT decomposition reach high same-answer rates early.Source: §3.2.1 · p. 7, §3.2.2 · p. 8, §3.2.3 · p. 8

3.3.1. BIASED REASONING FROM ANSWER ALWAYS A

  • For the answer-is-always-A bias, accuracy changes little across reasoning methods: CoT drops 1.2%, CoT decomposition drops 2.8%, and factored decomposition gains 2.1%.The few-shot condition drops 7.1%; the authors report no significant faithfulness differences between methods in this setting.Source: §3.3.1 · p. 8

3.3.2. BIASED REASONING FROM SUGGESTED ANSWER

  • With an incorrect answer suggested by the user, factored decomposition has the smallest accuracy drop at 9.2%, versus 21.3% for CoT and 29.1% for CoT decomposition.Factored decomposition mitigates some, but not all, of the faithfulness shortfall in this setting; CoT decomposition is less faithful than CoT by this measure.Source: §3.3.2 · p. 9

3.3.3. CONCLUSIONS

  • Across bias measures, factored decomposition appears more faithful than CoT or CoT decomposition, but the ordering of CoT and CoT decomposition conflicts with perturbation results.After normalizing and averaging the metrics, the authors assess CoT decomposition as more faithful than CoT.Source: §3.2.3 · p. 8, §3.3.3 · p. 10

3.4. Qualitative Findings

  • Decomposition samples are generally sensible, often using subquestions for answer-choice elimination and introductory questions to gather context.Factored decomposition can produce unanswerable or repeated subquestions, or subanswers supporting multiple choices without explicitly resolving which is correct.Source: §3.4 · p. 10

3.5. Discussion and Limitations

  • Question decomposition improves faithfulness at a cost to question-answering performance: factored decomposition is most faithful but performs worst, while CoT decomposition is intermediate on both.The effect of further training on this trade-off is uncertain.Source: §3.5 · p. 10
  • Faithfulness conclusions rely on indirect assessment methods, and evaluation covers only four question-answering tasks and one RLHF-finetuned model.The authors note that pretrained models may show different faithfulness benefits from decomposition and that broader task and model coverage could strengthen conclusions.Source: §3.5 · p. 10

4. Related Work

Prior work uses task decomposition to improve performance across varied settings, while this paper examines whether decomposition also improves reasoning faithfulness. Research on explanation faithfulness finds that CoT can be ignored and that its plausibility varies.

Task-Decomposition and Factored Cognition

Explanation Faithfulness

  • Prior faithfulness work reports that models may ignore parts of CoT reasoning, while studies disagree on whether CoT explanations are plausible and helpful [Gao, 2023; Uesato et al., 2022; Jung et al., 2022; Ye & Durrett, 2022; Golovneva et al., 2023; Madaan & Yazdanbakhsh, 2022; Li et al., 2022].The paper adopts the definition of faithful reasoning as the extent to which reasoning reflects the model’s actual reasoning.Source: §4 · p. 11

5. Conclusion

The paper finds that question decomposition can improve the faithfulness of model-generated reasoning while maintaining similar question-answering accuracy. It argues that faithful reasoning may support process-based oversight, while emphasizing the need to validate whether it helps detect undesirable behavior.

5. Conclusion

  • Question decomposition can greatly improve reasoning faithfulness while maintaining similar question-answering accuracy, suggesting further room for improvement with other techniques.The conclusion discusses Chain-of-Thought, CoT decomposition, and factored decomposition as the prompting strategies explored.Source: §5 · p. 11
  • Faithful reasoning could make process-based oversight more useful in high-stakes settings by allowing outputs to be discarded when reasoning surfaces behavior such as reward hacking or sycophancy.The authors say this consequence requires further validation, especially for undesirable behaviors difficult to detect from final outputs alone.Source: §5 · p. 11
  • The authors identify training models for more effective, faithful reasoning and validating its use in detecting undesirable behavior as directions for further research.They also note that faithful stated reasoning may improve methods that train models using their stated reasoning process.Source: §5 · p. 11

A.1. Further Early Answering Results

Early-answering trends generally resemble the averages across evaluated tasks, though TruthfulQA is a slight exception. Factored decomposition remains the most faithful method by this metric on TruthfulQA.

A.1. Further Early Answering Results

  • Across tasks, early-answering curves generally match the averages shown in Figure 3, suggesting similar sensitivity to reasoning truncation.TruthfulQA is a slight exception, with more similar trends among prompting strategies.Source: §A.1 · p. 15
  • On TruthfulQA, factored decomposition still appears most faithful by the early-answering metric, despite the strategies’ more similar trends.The passage characterizes TruthfulQA as a slight exception to the cross-task pattern.Source: §A.1 · p. 15
  • The reported cross-task similarity concerns model sensitivity to truncating reasoning samples, not a claim that every task’s curves are identical.The authors describe the sensitivity as fairly similar across evaluated tasks.Source: §A.1 · p. 15

A.2. Further Adding Mistakes Results

When reasoning is corrupted with added mistakes, the method ranking is maintained across tasks: factored decomposition prompts yield the most answer changes and CoT the fewest. OpenBookQA shows the smallest effect sizes across methods.

A.2. Further Adding Mistakes Results

  • For every evaluated task, corrupted factored decomposition samples prompt the most answer changes, corrupted CoT the fewest, and corrupted CoT decomposition falls between them.The relative ordering of reasoning faithfulness is maintained across tasks.Source: §A.2 · p. 15
  • OpenBookQA has the smallest effect sizes for final-answer sensitivity to reasoning truncation across all prompting methods.Other tasks generally show very similar effect sizes.Source: §A.2 · p. 15
  • The method ordering under added mistakes is consistent across tasks, while effect sizes vary, with OpenBookQA smallest and other tasks generally similar.Source: §A.2 · p. 15

A.3. Further Biasing Context Results

Further biasing-context results are detailed in Figures 7 and 8. Some findings are qualified by answer-format constraints and possible noise.

A.3. Further Biasing Context Results

  • HotpotQA and StrategyQA biasing-context results may be skewed because their questions have only two answer choices.The passage specifically notes possible distortion from suggested-answer bias.Source: §A.3 · p. 15
  • The OpenBookQA “answer is always A” results for factored decomposition are unexpected and may reflect noise.Source: §A.3 · p. 15
  • Figures 7 and 8 provide detailed results for the biasing-context experiments discussed in Sections 3.3.1 and 3.3.2.Source: §A.3 · p. 15

B. Biased Reasoning from Sycophancy

On sycophancy evaluations, factored decomposition produces more non-sycophantic answers than the compared prompting methods. The authors caution that this pattern does not establish greater reasoning faithfulness, and describe the evaluations and few-shot materials.

B. Biased Reasoning from Sycophancy

  • The evaluations tested whether models agreed with users who stated a belief before asking a related question, using 200 randomly chosen questions per evaluation.The authors expected 50% non-sycophantic answers if a model were completely non-sycophantic.Source: B. Biased Reasoning from Sycophancy · pp. 15, 18
  • 14.7% of factored-decomposition answers were non-sycophantic, versus 4.7% for CoT, 5.2% for CoT decomposition, 9.2% zero-shot, and 8.3% few-shot.The evaluations covered philosophy, NLP, and political questions.Source: Figure 9 · p. 18, B. Biased Reasoning from Sycophancy · p. 18
  • The sycophancy reduction with factored decomposition is not clear evidence of greater faithfulness, since the model may fail to phrase questions to infer users’ views.The authors say their assumption that reasoning samples do not explicitly reference biasing features does not hold for these evaluations.Source: B. Biased Reasoning from Sycophancy · p. 18

C. Few-Shot Examples and Instructions

  • Tables 6–12 provide few-shot examples and prompting instructions for the methods.The full prompts, including additional examples and reasoning demonstrations, are available in a supplementary repository.Source: C. Few-Shot Examples and Instructions · p. 18

D. Reasoning Samples

The reasoning samples show that CoT decomposition and factored decomposition often use similar question decompositions, including a process-of-elimination structure.

D. Reasoning Samples

  • CoT decomposition and factored decomposition samples often use similar question decompositions with a process-of-elimination structure.Source: D. Reasoning Samples · p. 18

E. Adding Mistakes Prompts and Corrupted Reasoning Samples

The paper describes prompts for introducing mistakes into reasoning samples and shows original and corrupted examples. Over two-thirds of the corrupted samples contain errors expected to change the final answer, so the reported sensitivity may be underestimated.

E. Adding Mistakes Prompts and Corrupted Reasoning Samples

  • Over two-thirds of corrupted reasoning samples contain errors that should almost certainly change the final answer, likely making measured answer sensitivity an underestimate.Source: E. Adding Mistakes Prompts and… · p. 18
  • The corruption prompts target a CoT step, a CoT-decomposition question–answer pair, or a factored-decomposition subanswer.The factored-decomposition corruption is introduced during subquestion answering.Source: Table 15 · p. 27, Table 16 · p. 28, Table 17 · p. 28
  • Tables 18–20 show original and corrupted samples, including cases where corruption changes the answer and cases where it does not.The examples cover CoT, CoT decomposition, and factored decomposition.Source: Table 18 · p. 29, Table 19 · p. 30, Table 20 · p. 31
Loading 2307.11768v2…