Answering Questions by Meta-Reasoning over Multiple Chains of Thought

1 Introduction

• Self-consistency can lose useful intermediate facts and offers no single reasoning chain as an explanation.When sampled chains produce different answers, a significant majority may not form; the intermediate steps can nevertheless contain relevant information.Source: §1 · p. 1
• MCR uses a separate meta-reasoner to combine information from multiple reasoning chains and produce a final answer with an explanation.It concatenates the chains’ intermediate steps into a unified context rather than using the chains only for their predictions.Source: §1 · p. 2
• MCR explanations are rated high quality for over 82% of examples, while fewer than 3% are judged unhelpful.The paper reports these findings from its explanation-quality analysis.Source: §1 · p. 3
• MCR outperforms all baselines, including self-consistency, on all 7 multi-hop open-domain QA benchmarks.The introduction reports that MCR beats self-consistency by up to 5.7%.Source: §1 · pp. 2–3

2 Background

• Many prompting approaches plan a reasoning chain and solve intermediate steps, often using retrieval to reduce hallucination.The related work describes this as a common approach to multi-hop questions.Source: §2 · p. 3
• Prior methods may incorporate multiple reasoning chains by using their answers to derive a final answer.MCR’s related-work discussion distinguishes this answer-combination stage from its own focus.Source: §2 · p. 3
• MCR focuses on using multiple chains to collect relevant evidence for question answering, rather than merely combining their answers.This is the stated distinction from prior work in the section.Source: §2 · p. 3

3 Method

• MCR’s core contribution is a second LLM that meta-reasons over multiple generated chains, collecting evidence facts for its explanation and final answer.The chains are used as context rather than sampled for their predicted answers.Source: §3 · p. 3, §3.2 · p. 3

3.1 Generating Reasoning Chains

• For each chain step, decomposition generates an intermediate question, retrieval finds relevant evidence, and decomposition generates an answer; evidence is prepended to the chain.The authors report that prepending evidence during answer generation improves accuracy for all baselines.Source: §3.1 · p. 3, Figure 3 · p. 3

3.2 Reasoning over Reasoning Chains

• The meta-reasoner receives the original question and the sampled chains’ intermediate question-answer pairs as a multi-chain context.It is few-shot prompted to answer step by step; its generated reasoning chain followed by the answer serves as the explanation.Source: §3.2 · p. 4
• With multiple chains available, the meta-reasoner can aggregate facts across them and select the most relevant facts for its explanation.The paper presents this as the basis for MCR’s accuracy and interpretability compared with past multi-chain approaches.Source: §3.2 · p. 4

4.1.1 Datasets

• Evaluation spans implicit reasoning that requires inferred strategies, explicit multi-hop reasoning, and FEVEROUS fact verification using sentence or table evidence.The experiments use development examples and also evaluate on official STRATEGYQA and FERMI test sets.Source: §4.1.1 · p. 4

4.1.2 Models

• Main models use retrieval-augmented code-davinci-002 with in-context exemplars; MCR gives its meta-reasoner five chains, while SCR uses one greedy chain.Google Search retrieves top-1 evidence, and greedy-decoded chains are more accurate than sampled chains.Source: §4.1.2 · p. 5

4.2 Main Results

• MCR outperforms all baselines on seven multi-hop QA benchmarks spanning experiments with two LLMs and retrievers.On datasets shared with recent approaches, its performance is comparable with the best results, though the comparison is not apples-to-apples.Source: §4 · p. 4, Table 2 · p. 6, §4.2 · p. 6
• Against SC@5, MCR gains +1.4% on STRATEGYQA, +0.6% on FERMI, +4.0% on QUARTZ, and +5.7% on HOTPOTQA.Table 2 reports code-davinci-002 experiments across seven datasets; results average three runs, or five for BAMBOOGLE.Source: §4.2 · p. 6, Table 2 · p. 6
• Against SC@5, MCR gains +2.5% on 2WIKIMQA, +1.5% on BAMBOOGLE, and +1.5% on FEVEROUS.These are the remaining dataset comparisons in the seven-benchmark results.Source: §4.2 · p. 6
• MCR+SC@3 consistently outperforms SC@15, and five-chain MCR beats SC@15 on every dataset except STRATEGYQA; Figure 5 tracks performance at 1, 5, and 15 chains.MCR+SC@3 uses 15 chains across three MCR runs.Source: Table 3 · p. 6, §4.2 · p. 6, Figure 5 · p. 6
• On official test sets, MCR beats SC at equal chain counts on STRATEGYQA, while both methods perform similarly on FERMI.The evaluation uses the official STRATEGYQA and FERMI test sets.Source: §4.2 · p. 6, Table 4 · p. 7

4.3 Open-source Models

• With ColBERTv2, code-davinci-002 MCR beats SC@5 across all seven datasets; Vicuna-13B MCR also beats SC@5 on all seven.Vicuna-13B MCR beats all baselines on five datasets, and MCR+SC@3 beats SC@15; results suggest greater gains when both models are larger.Source: §4.3 · p. 7

When are Multiple Chains Helpful?

• MCR’s gains over SCR are highest when its explanations are less similar to the greedy chain, a trend observed across all datasets.On STRATEGYQA, MCR outperforms SCR when it uses a chain other than the greedy one.Source: §5 · p. 7, Figure 7 · p. 8

Combining Reasoning Chains

• MCR explanations combine chains in 20% of STRATEGYQA and 25% of FERMI examples; manual review found these combinations better than any single chain in 10%.Detection looked for sentences drawn from different chains; the remaining 90% of combinations paraphrased an individual chain.Source: §5 · p. 8

Explanation Quality

• Meta-reasoner explanations were rated for relevance to answering the question; they were highly relevant in 82% of cases (87% excluding FERMI, the most challenging), and irrelevant in less than 3%.Source: §5 · p. 8

Faithfulness

• MCR predictions matched human answers in 90% of cases (95% excluding FERMI) when people answered from explanations scored as high quality.The authors attribute part of the gap to implicit reasoning tasks, where humans lead by five points on average.Source: §5 · pp. 8–9

Error Analysis

• Four datasets had over 20% valid errors; decomposition errors exceeded 24% on implicit datasets, while other error types varied by reasoning style.Excluding valid examples, explanation and answer errors were 50% on implicit versus 23% on explicit tasks; retrieval or contradicting-fact errors were 30% versus 66%, respectively.Source: §5 · p. 9, Table 7 · p. 9

Reasoning in LLMs

• LLM reasoning work includes scratchpads, chain-of-thought prompting, learned verifiers, selection-inference, and bootstrapping.The passage cites Nye et al. (2022), Wei et al. (2022), Zhou et al. (2022), Cobbe et al. (2021), Creswell et al. (2022), and Zelikman et al. (2022).Source: §6 · p. 9

Self-consistency and meta-reasoning

• Self-consistency selects the majority answer across chains; later work increases chain diversity, verifies intermediate steps, or reranks chains using retrieved evidence.MCR also reviews LLM-generated chains, but focuses on meta-reasoning over multiple chains.Source: §6 · p. 9

Question answering over retrieved facts

• Related QA research addresses reasoning over multiple retrieved facts, including explicit, implicit, and multimodal questions, as well as retrieval-augmented LLMs for open-domain questions.The cited work spans multi-step reasoning and open-domain QA approaches.Source: §6 · p. 9

• MCR outperforms previous approaches on all evaluation benchmarks across seven multi-hop QA datasets requiring implicit or explicit reasoning in an open-domain setting.Source: §7 · p. 9

• The study tests a prompted LLM rather than a fine-tuned meta-reasoner, and leaves further improvements to meta-reasoner context for future work.Because of inference costs, evaluation uses code-davinci-002; results with an open-source LLM and retriever are also provided for reproducibility.Source: §8 · p. 10

A.1 Generating Unknown as the Answer

• Binary-choice abstentions or null predictions receive 0.5, while Unknown or null open-ended predictions receive 0.For STRATEGYQA test submissions, abstentions or null predictions are assigned a random Yes or No label; Self-Ask and SC also receive special handling for null answers.Source: §A.1 · p. 13

A.2 FERMI

• FERMI questions include the gold answer’s measurement units for every model and baseline, yet final-answer prediction remains highly challenging.Examples of units include meters, cubes, and litres.Source: §A.2 · p. 13

B.1 Retrieval

• Google Search retrieval returns the top result as an evidence snippet and prepends its page title; snippets may include answer boxes or tables.Figure 8 illustrates a retrieved snippet for an intermediate question.Source: §B.1 · p. 13, Figure 8 · p. 13

B.2 Implementation Details

• Original-question retrieval lowers FERMI scores for SA, SC, SCR, and MCR, so the authors omit it for FERMI evaluation.The authors hypothesize MCR may be somewhat more robust than the other models to irrelevant evidence.Source: §B.2 · pp. 13–14
• Listing true and false intermediate facts accompanies lower FEVEROUS scores for MCR and SCR; the authors hypothesize repeated true facts may bias predictions toward Yes.The reported scores decrease from 69.4±1.0 to 66.4±0.7 for MCR and from 65.1±0.4 to 62.9±0.3 for SCR.Source: §B.2 · p. 14

B.3 Empirical Comparison to Recent Approaches

• Recent-method comparisons are not apples-to-apples across evaluation sets, retrieval corpora, hyperparameters, prompts, and language models.MCR’s stated focus is the potential of reasoning over chains; task-specific improvements are described as orthogonal to that focus. The use of updated Wikipedia evidence may also help explain the high share of MCR predictions labeled valid in the error analysis.Source: §B.3 · p. 14

B.4 Reasoning on Retrieved Evidence

• Meta-reasoning over retrieved question-evidence pairs is less effective than over question-answer pairs, especially on implicit reasoning tasks.The authors suggest retrieved evidence may be less relevant on average; Table 9 compares the two context variants.Source: §B.4 · p. 14, Table 9 · p. 15

C.1 When are Multiple Chains Helpful?

• Across datasets, MCR’s gains over SCR are larger when its explanation is less similar to the greedy chain.Similarity is measured with ROUGE-1 precision; the analysis groups examples into Low, Medium, High, and Identical categories.Source: §C.1 · p. 15

C.2 Combining Reasoning Chains

• Figure 10 provides additional examples of combining facts from multiple reasoning chains.Source: §C.2 · p. 15

C.3 Explanation Quality Analysis

• Four graduate-student authors annotated meta-reasoner explanations by rating explanation quality and answering the associated question based on each explanation; full annotator instructions and results are provided in Fig. 11 and Tab. 10.Source: §C.3 · p. 15

C.4 Error Analysis

• Retrieval errors involve missing relevant information and a hallucinated intermediate answer; decomposition errors occur when retrieved evidence is ignored and an incorrect answer is hallucinated.The authors say the error categories mainly correspond to MCR components and could help guide future improvements.Source: §C.4 · p. 15

D.1 Prompt Details

• The prompt appendix gives decomposition and meta-reasoner examples for 2WIKIMQA and STRATEGYQA, using random exemplars and minimal prompt engineering.The number of exemplars varies slightly across datasets and models.Source: §D.1 · pp. 15–16

D.2 Prompt Statistics

• Prompt statistics include decomposition inputs and outputs, retrieved evidence, and meta-reasoning prompts; generating one chain takes multiple LLM calls and is generally longer than one extra meta-reasoning step.The lengths are token counts from the code-davinci-002 tokenizer, averaged over multiple runs corresponding to Table 2.Source: §D.2 · p. 16

D.3 Robustness to Choice of Prompt

• With different random exemplars, MCR still outperforms all baselines on 500 examples each from STRATEGYQA and HOTPOTQA.Decomposition performance is more affected by exemplar choice, but the performance trend remains the same, with MCR on top; Table 13 compares these results with the original prompts.Source: §D.3 · p. 16, Table 13 · p. 18
• The appendix includes prompt exemplars for decomposition and meta-reasoning on 2WIKIMQA and STRATEGYQA, including retrieved-evidence variants.Figures 13–18 contain the listed instructions and exemplars.Source: Figure 13 · p. 20, Figure 14 · p. 21, Figure 16 · p. 23
• The appendix also supplies examples of questions, multi-chain contexts, and answers used with the prompts.These examples include cases involving film directors, jellyfish, tonsure, arithmetic, and family relations.Source: §D.3 · pp. 22–23