PaperlayerView exact revision on arXiv

Source-linked AI summary

Faithful Reasoning Using Large Language Models

Antonia Creswell, Murray Shanahan

arXiv:2208.14271v1cs.AIcs.CL

TL;DR

LM answers often conceal the assumptions and intermediate steps needed for verification, especially in multi-step reasoning. The paper develops Selection-Inference, which chains fine-tuned selection and inference models, adds halting and value-guided beam search, and evaluates the approach on two reasoning datasets. The resulting models achieve higher final-answer accuracy than baselines and produce more accurate, interpretable traces, while requiring a provided context and retaining no guarantee that every step is logically correct.

  • Problem

    LM question-answering answers are often opaque because their assumptions and intermediate reasoning are hidden, limiting verification and debugging.

  • Method

    Selection-Inference chains fine-tuned selection and inference LMs with causal structure mirroring logical validity, supplemented by halting and value-guided beam search.

  • Results

    The model achieves higher final-answer accuracy than baselines on both Proof Writer and EntailmentBankQA, with stronger reasoning-trace accuracy and less hallucination.

  • Takeaways & Limitations

    Faithful reasoning traces let users inspect whether conclusions follow from context while the system improves answer and trace quality across evaluated tasks.

  • Takeaways & Limitations

    The approach assumes access to a context, although most real-world settings are unlikely to provide one without retrieval.

Abstract

from arXiv · show

Although contemporary large language models (LMs) demonstrate impressive question-answering capabilities, their answers are typically the product of a single call to the model. This entails an unwelcome degree of opacity and compromises performance, especially on problems that are inherently multi-step. To address these limitations, we show how LMs can be made to perform faithful multi-step reasoning via a process whose causal structure mirrors the underlying logical structure of the problem. Our approach works by chaining together reasoning steps, where each step results from calls to two fine-tuned LMs, one for selection and one for inference, to produce a valid reasoning trace. Our method carries out a beam search through the space of reasoning traces to improve reasoning quality. We demonstrate the effectiveness of our model on multi-step logical deduction and scientific question-answering, showing that it outperforms baselines on final answer accuracy, and generates humanly interpretable reasoning traces whose validity can be checked by the user.

1. Introduction

The paper addresses the opacity of LM answers by developing faithful reasoning whose causal structure mirrors logical validity and exposes interpretable traces. Its Selection-Inference model uses fine-tuned selection and inference LMs, with halting and search methods that improve performance across evaluations.

  • LM question-answering systems hide assumptions and intermediate reasoning, preventing users from verifying answers, debugging errors, and trusting responses.
  • Faithful reasoning mirrors logical validity and exposes assumptions and reasoning steps so users can assess whether conclusions are justified.
  • Selection-Inference chains fine-tuned selection and inference LMs whose causal structure mirrors logical validity, guaranteeing answers follow from context under assumptions.
  • A halter terminates reasoning and returns Unknown when traces exceed a step limit, while a learned value function guides beam search over traces.
  • The model outperforms baselines on final-answer accuracy across Proof Writer and EntailmentBankQA, while improving trace quality, context use, and resistance to hallucinated facts.

2. Defining a Valid Reasoning Trace

The paper defines valid forward reasoning through structured traces composed of connected, logically correct reasoning steps. Its architecture is designed to satisfy these structural requirements under assumptions about step correctness.

  • A reasoning step is a pair consisting of a selection set of statements and an inference statement.
  • A reasoning trace pairs a context with a sequence of reasoning steps.
  • A trace is connected when every selected statement comes from the original context or an earlier inference.
  • A trace is valid when it is connected and every inference logically follows from its selected statements.
  • The architecture mirrors these validity requirements, guarantees connectedness by design, and encourages—but cannot guarantee—logical correctness of every LM-generated step.

3. Components of a Faithful Reasoning Model

The model combines step-wise Selection-Inference reasoning with a halter and value-guided beam search. Selection and inference construct traces from context, halting converts sufficient inferences into answers, and search favors higher-quality traces.

  • Selection-Inference: Selection-Inference repeatedly selects context statements and predicts an entailment, adding each inference to the context to form a multi-step reasoning trace.
  • 3.1.1. Selection: The Selection model uses sentence labels to select only context statements, preventing it from hallucinating facts outside the context.
  • 3.1.2. Inference: The Inference model receives only the selected statements, which prevents direct answer prediction from the question and encourages entailment-based reasoning.
  • 3.2. Halting: When to Stop Reasoning?: The Halter checks whether the current inference suffices, returns an answer when it does, and otherwise continues reasoning or returns Unknown after a limit.
  • 3.3. Search: Finding the Best Trace: A value function scores candidate reasoning steps, and beam search retains the highest-scoring traces as Selection-Inference expands the trace tree.

4. Experimental Setup and Evaluation of Components

The evaluation uses Proof Writer and a modified question-answering version of EntailmentBank, with 7B Chinchilla models trained and evaluated across the system’s components. Results assess reasoning, halting, and trace-quality mechanisms, including performance when unknown answers are filtered.

  • Datasets: The experiments use Proof Writer and a modified, more challenging question-answering version of EntailmentBank.The reformulated EntailmentBankQA task predicts an answer and reasoning trace from the question, context, and choices.
  • Model configuration: Each system component uses a 7B-parameter Chinchilla language model.
  • Selection-Inference: Selection models predict context-statement labels, while inference models predict entailments from selected statements.Training examples are derived from individual reasoning steps; label-based selection prevents the model from making up facts absent from the context.
  • Halter: The Halter is trained to identify whether an inference contains enough information to answer and then to predict the answer from the inference and choices.Intermediate reasoning steps use a negative target, while final steps use a positive target; a separate answer datapoint uses the ground-truth answer.
  • Halter evaluation: 88.8% accuracy is achieved by the EntailmentBank Halter when evaluated on ground-truth proofs.The Proof Writer Halter performs almost perfectly under the same independent evaluation procedure.
  • Filtered-answer evaluation: 87.5% and 83.7% accuracy are obtained on EntailmentBank Tasks 1 and 2 after filtering problems where the model does not know the answer.On Proof Writer, filtering unknown answers yields nearly perfect accuracy across all reasoning depths.

5. Experiments and Results

Across Proof Writer and EntailmentBankQA, SI outperforms baselines in final-answer accuracy while producing more faithful, connected reasoning traces. Search improves performance most on deeper problems and distractor-heavy contexts, while the halter supports abstention when answers are unknown.

  • 88.1% and 78.1% final answer accuracy are achieved on Proof Writer and EntailmentBankQA, respectively, significantly outperforming baseline models.
  • SI often produces higher-quality reasoning traces than baselines, especially on challenging Proof Writer depth-5 and EntailmentBankQA Task 2 problems.
  • Baseline models can predict intermediate outputs yet select incorrect supporting statements, limiting their final-answer accuracy and trace validity.
  • The halter identifies cases where the model does not know the answer, enabling filtering of Unknown outputs to increase precision.
  • Search improves both SI and baseline performance, with the largest gains on Proof Writer depth-5 and EntailmentBankQA Task 2 with distractors.
  • SI makes up facts in under 1% of EntailmentBankQA problems, compared with up to 60% for Proof + Answer and up to 40% for Proof Only models.

6. Related Work

Prior work has improved answer accuracy or exposed selected evidence, but often does not produce valid, faithful traces showing how knowledge is combined. This paper distinguishes its approach through validity-grounded traces and step-level search.

  • Reasoning traces: Recent methods often improve final answer accuracy without producing valid, human-interpretable reasoning traces.
  • Reasoning traces: The paper focuses on answering questions with faithful reasoning traces rather than post-hoc explanations.
  • Reasoning traces: Unlike evidence-extraction approaches, this work shows how multiple knowledge pieces are combined over several iterations into a valid reasoning trace.
  • Validity: The paper relates faithful reasoning more precisely to logical validity than earlier informal uses of the concept.
  • Search: Its value function evaluates partial reasoning traces for step-level beam search, rather than verifying only complete traces.
  • Search: Unlike exhaustive pairwise selection, the paper uses beam search and permits selection of more than two statements, supporting rules conditioned on multiple statements.

7. Limitations

The model’s architecture mirrors the requirements for connected and valid reasoning traces, but individual reasoning steps are not guaranteed to be logically correct. The study also assumes a supplied context, leaving retrieval for future work.

  • The architecture guarantees connected reasoning traces by design and mirrors the requirements for valid traces.
  • Individual reasoning steps may still be incorrect because the system relies on language models.
  • A selection of unrelated statements can lead the inference model to draw a nonsensical conclusion.
  • The learned value function filters poor reasoning traces but cannot guarantee correctness for every step.
  • The approach assumes access to a context and leaves retrieval to future work, limiting its scope in settings without provided context.

8. Discussion

The discussion characterizes faithful reasoning as logically valid traces and presents Selection-Inference as an architecture that mirrors this definition. Across two tasks, the model improves answer accuracy and several measures of reasoning quality, while illustrating algorithmic prompting as a broader design pattern.

  • Selection-Inference mirrors the structure of logical validity and produces valid reasoning traces when individual reasoning steps are correct.
  • The model achieves higher final answer accuracy than baseline models on both Proof Writer and EntailmentBankQA.
  • Search benefits both Proof Only and SI, while the Halter and search components provide compounding performance improvements.
  • Compared with baselines, the model is less likely to hallucinate facts, pays more attention to context, and more often uses its reasoning trace to answer questions.
  • The approach exemplifies algorithmic prompting, in which model queries and prompt construction are composed into algorithms using sequence, choice, and iteration.

Supplementary Materials

The supplementary formalization describes reasoning over a question and context through alternating selection and inference steps. The context accumulates inferred statements, while a halting model decides when to stop and supplies the answer.

  • The problem is represented as a question q paired with an initial context C0 containing statements sufficient to predict the correct answer.
  • The Selection model samples a selection s_k from the question q and current context C_k.
  • The Inference model samples an inference i_k from the selected statements s_k.
  • After each inference, the context is updated by accumulating inferences from previous reasoning steps.
  • The Halter LM decides whether reasoning should stop and answers the question when the model is ready.

B. Training Details

The supplementary materials describe dataset selection, algorithm inputs, and training-data construction for the Selection, Inference, Halter, and value-function components. They also reference qualitative Halter results and examples of Selection-Inference training pairs.

  • Training data: The Proof Writer setup uses a subset of the Open World Assumption dataset because it provides proofs for both True and False answers, while proofless cases are Unknown.
  • Algorithm inputs: Algorithm 1 takes Selection and Inference LMs, a halt function, a question, an initial context, choices, and a maximum number of reasoning steps as inputs.
  • Model training: Figures 11 and 12 provide training samples used to fine-tune the Selection and Inference LMs.
  • Model training: Figure 13 shows how training data is constructed for the Halter model.
  • Model training: The value function is trained with targets labeled correct or incorrect, and Figures 14 and 15 show example data points.
  • Halting: The same Halter LM determines whether the model can answer and produces the answer, using different prompts for the two functions.
  • Evaluation: Figure 16 presents qualitative results from the Halter model trained on EntailmentBankQA.

C.2. Reasoning Trace Accuracy

The evaluation examines whether reasoning traces are accurate at the selection, inference, and full-trace levels, while also checking whether models introduce facts absent from context.

  • Fact faithfulness: Proof + Answer and Proof Only do not make up facts on Proof Writer, while EntailmentWriter and SI are designed not to hallucinate facts.The comparison measures the proportion of problems containing facts absent from the context.
  • Trace accuracy: SI and models using halting and search outperform baseline models on ordered intermediate-inference Rouge-1 scores, especially as reasoning depth increases.EntailmentWriter performs well on single-step Task 1 problems but declines quickly on multi-step problems.
  • Inference accuracy: The inference model achieves almost perfect accuracy on Proof Writer when evaluated by exact string match.Table 7 uses lowercased exact matching to determine whether predicted and reference statements are identical.

D.1. Proof Writer: SI + Halter + Search

Proof Writer examples show SI + Halter + Search producing ranked reasoning traces that may repeat valid steps, while the traces distinguish valid from invalid reasoning and derive answers through chained implications.

  • Trace examples: SI + Halter + Search displays the top ten Proof Writer reasoning traces ranked by a learned value-function score.The examples receive no additional filtering.
  • Trace examples: The displayed traces combine each selection and inference into one line, with invalid reasoning marked in red.High-scoring examples can contain repeated steps because Proof Writer proofs often repeat steps.
  • Trace examples: The Proof Writer examples include multi-step derivations that conclude answers by applying rules to stated facts and previously derived inferences.For example, the traces derive the bald eagle’s greenness through successive claims about roughness, eating the cow, and being green.
  • Trace examples: The examples also include negative conclusions and binary answer judgments supported by explicit proof chains.One trace derives that the bald eagle is not kind from the fact that the cat eats the bald eagle.

D.2. EntailmentBankQA Task 1 - depth 1: SI + Halter + Search

For depth-one EntailmentBankQA questions, SI + Halter + Search produces concise traces that apply a single relation or fact chain to select the correct answer.

  • Depth-one examples: Depth-one traces answer questions by applying one-step entailments, such as inferring that a fly has six legs from its being an insect.The predicted proof and target proof reach the same answer.
  • Depth-one examples: The examples correctly infer an ice cube’s solid state from the statements that solids are states of matter and ice cubes are solids.The target answer is solid.
  • Depth-one examples: The traces answer comparison questions by combining matching facts, such as mountains on both the Moon and Earth.The target answer is mountains.
  • Depth-one examples: Other depth-one examples correctly connect material properties to answers, selecting steel for fastest sound transmission and a drinking straw for flexibility.These traces use the provided category and property statements directly.

D.3. EntailmentBankQA Task 1 - depth 2: SI + Halter + Search

Deeper EntailmentBankQA examples show the system chaining two or three reasoning steps to answer scientific questions, while preserving the intermediate conclusions in its proofs.

  • Depth-two examples: Two-step reasoning identifies increasing the number of airport runways as the most effective way to reduce air-traffic congestion.The proof connects runways with pathways and then connects more pathways with reduced congestion.
  • Depth-two examples: Two-step reasoning classifies wave and tidal energy as both renewable by combining their individual renewable-resource statements.The target answer is Both are renewable.
  • Depth-two examples: The convex-lens example chains refraction and light bending to select a convex lens for demonstrating bending through a substance.The proof first places light at the focal point, then uses refraction’s definition as bending.
  • Depth-three examples: Three-step traces derive that equatorial ocean air is moist and warm by combining ocean humidity with sunlight, temperature increase, and warmth rules.The proof preserves each intermediate inference before producing the answer.
  • Depth-three examples: Another multi-step trace explains a rock’s slowing on ice through friction, using object categories before applying friction’s effect on speed.The target answer is the force of friction.

D.5. EntailmentBankQA Task 1 - depth 5: SI + Halter + Search

The section presents high-value SI reasoning traces on five-step and single-step EntailmentBankQA problems, including both successful proofs and recovery from an early mistake.

  • Five-step reasoning traces: Five-step traces show SI deriving answers through chained statements that connect context facts and rules to the target answer.Examples include identifying forest-adapted animals, predicting harm to fish from toxic chemicals, and classifying horse plowing as non-instinctive behavior.
  • Five-step reasoning traces: In one five-step example, the model makes a mistake on the first reasoning step but recovers without using the incorrect inference later.The example explicitly reports recovery after an incorrect initial inference.
  • Five-step reasoning traces: Additional traces answer questions about specialized cells protecting the body and wood boards being made from a renewable natural resource.The proofs connect specialized cells to chemicals destroying microbes and wood boards to renewable natural resources.
  • Single-step reasoning traces: The section also introduces top-ranked single-step EntailmentBankQA traces selected by the learned value function.These examples are ranked by value and are described as not cherry picked.

E.1. EntailmentWriter on Proof Writer

The section shows that EntailmentWriter can reach correct answers with reasoning traces that contain inferences not logically supported by the context, unlike SI's question-independent inference step.

  • EntailmentWriter behavior: EntailmentWriter examples produce correct final answers while using incorrect reasoning and fabricated inferences that help solve the question.The examples are presented as proofs in which the final conclusion answers the question despite unsupported intermediate steps.
  • EntailmentWriter behavior: EntailmentWriter has direct access to the question during inference, creating an incentive to generate helpful conclusions even when they do not follow logically.The passage contrasts this with SI, whose inference step does not access the question.
  • Proof Writer examples: Proof Writer examples include chains that derive conclusions about the dog, cow, and cat through stated rules and intermediate assertions.The displayed examples illustrate the types of reasoning traces evaluated in this section.
  • Evaluation materials: The accompanying figures document Value LM training examples, qualitative EntailmentBankQA outputs, and exact-string evaluation of Proof Writer reasoning traces.Another figure reports Rouge scores between ground-truth and predicted intermediate inferences in order.
Loading 2208.14271v1…