Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks

• Complex knowledge-intensive tasks require multi-step reasoning, yet LLMs struggle with compositional, long-tail, and factual knowledge, while context-only outputs lack traceability.Source: §1 · p. 1
• Introducing IR can break LLM reasoning, mislead it with incorrect retrieved information, and leave prior methods unable to dynamically change reasoning direction.Source: §1 · pp. 1–2
• SearChain has the LLM plan a global Chain-of-Query (CoQ), with each node containing an IR-oriented query, an answer, and a flag for additional knowledge needs.IR verifies node answers and corrects inconsistencies when retrieval confidence is high; it supplies missing knowledge indicated by the flag.Source: §1 · p. 2
• After interaction, SearChain traces the reasoning and marks supporting documents, while IR interaction creates a Tree-of-Reasoning that lets the LLM modify its reasoning direction.Source: §1 · p. 2
• The paper frames SearChain as improving knowledge-reasoning and providing needed knowledge and document references, with reasoning-path changes enabled by node-identifying depth-first search.Source: §1 · p. 2
• Experiments report that SearChain outperforms state-of-the-art baselines on multi-hop Q&A, slot filling, fact checking, and long-form Q&A.Source: §1 · p. 2

2.1 Chain-of-Thought Prompting

• CoT methods provide intermediate reasoning, while decomposition methods often solve local subquestions step by step; CoQ instead plans a global query-answer chain [24] [48] [15].Source: §2.1 · p. 3

2.2 Retrieval-augmented Language Models

• Prior LLM–IR methods can interrupt continuous reasoning and provide information without correcting errors; SearChain verifies answers, supplies missing knowledge, and supports branching reasoning.Source: §2.2 · p. 3

3 OUR METHOD

• Each interaction round builds a Chain-of-Query of query-answer nodes, lets IR verify or complete them, and uses its feedback to generate a new chain.After interaction, SearChain traces the reasoning process and marks references to supporting documents.Source: §3 · p. 3

3.1 Comparison with Previous Methods

• SearChain plans a global reasoning chain per generation, verifies and completes nodes instead of directly inserting retrieved information, and can redirect reasoning on a tree.The comparison contrasts this with methods that reason over local subquestions, risk being misled by incorrect retrieval, and cannot dynamically change direction.Source: §3.1 · p. 3

3.2 Chain-of-Query Generation

• Each Chain-of-Query node pairs an IR-oriented query with an answer; an [Unsolved Query] flag marks missing knowledge for later rounds.The global chain plans a complete reasoning path rather than answering directly or solving only local subquestions; its broader perspective can prompt further decomposition or rewriting when a subquestion is difficult.Source: §3.2 · p. 4

3.3 Interaction with Information Retrieval

• IR retrieves a Top-1 document per query and corrects an inconsistent answer when the Reader is sufficiently confident, while supplying retrieved knowledge for unsolved queries regardless of that threshold.For short-form tasks, consistency checks whether the extracted answer appears in the LLM answer; for long-form tasks, they compare ROUGE with a threshold.Source: §3.3 · pp. 4–5
• SearChain traces the correct reasoning path and marks references to supporting documents for the corresponding generated sub-fragments.The final answer is prompted from query-answer pairs along the path, and the approach retrieves document support without supervised citation-annotation data or LLM training.Source: §3.3 · p. 5

4.1.1 Datasets and Evaluation Metric.

• The evaluation uses multi-hop QA, slot filling, fact checking, and long-form QA; it measures ELI5 with ROUGE-L and other tasks with cover-EM.Baselines include reasoning-focused methods and methods that combine IR with reasoning approaches.Source: §4.1.1 · p. 5, §4.1.2 · p. 5

4.2 Main Results

• Across complex knowledge-intensive tasks, SearChain without IR outperforms the CoT-based baselines, and SearChain with IR outperforms all baselines.The comparison covers multi-hop QA, slot filling, fact checking, and long-form QA; the supplied passages do not provide table values.Source: Table 1 · p. 6, §4.2 · p. 6
• SearChain without IR outperforms Self-Ask without IR and Least-to-Most, supporting global-chain construction over stepwise sub-question answering.The authors also report stronger performance than CoT, Auto-CoT, CoT-SC, and Recite-and-answer.Source: §4.2 · p. 6

4.3.1 Knowledge Decoupling.

• IR verifies LLM knowledge, corrects inconsistent answers when confident, and supplies missing knowledge; on StrategyQA, it supplies less because the LLM has memorized most retrievable knowledge.Knowledge-source shares are counted over Tree-of-Reasoning nodes.Source: §4.3.1 · p. 7, Table 2 · p. 7

4.3.3 CoQ vs Baselines in Reasoning.

• On Musique, SearChain uses more reasoning steps, and its average step count increases with question hops; it can also rewrite an unsolved sub-question and continue reasoning.The case study contrasts this behavior with CoT and Self-Ask stopping at the difficult sub-question.Source: Table 4 · p. 7, §4.3.3 · pp. 7–8, Figure 4 · p. 7

4.3.2 Positive and Negative Effects of IR on LLM.

• On questions selected for IR assistance, SearChain without IR performs worse than on the full set, while IR helps on those questions and mitigates newly incorrect answers on questions LLM already answers correctly.IR uses its confidence together with CoQ information to decide whether to correct an answer or provide missing knowledge.Source: §4.3.2 · p. 7, Table 3 · p. 7

4.3.4 SearChain vs New Bing in Tracing.

• SearChain scores 2.882 versus New Bing’s 1.143 on knowledge coverage, and 0.80 versus 0.45 on reference-position accuracy.The measures are Scope of Knowledge Coverage and Accuracy of Marking Position, respectively; the position metric uses human evaluation.Source: §4.3.4 · p. 8
• SearChain significantly improves task performance with no significant increase in time consumption, and most baselines also use multiple IR–LLM interaction rounds.The efficiency analysis considers input and output text lengths, interaction rounds, and overall running time.Source: Table 5 · p. 8, §4.3.5 · p. 8

5 CONCLUSION

• SearChain uses IR to provide needed external knowledge and marks supporting documents for knowledge in generated content, improving accuracy, credibility, and traceability.The paper describes its reasoning path as node-identify Depth-first Search on a tree, allowing the LLM to dynamically modify reasoning direction.Source: §5 · p. 8
• SearChain transforms the reasoning path from a chain to node-identify Depth-first Search on a tree, enabling dynamic changes in reasoning direction.The description continues across the supplied conclusion passages.Source: §5 · pp. 8–9
• On complex knowledge-intensive tasks, SearChain performs better than all baselines.Source: §5 · p. 9

A.1 Case Study

• Baselines focus on local sub-questions, whereas SearChain plans a global chain for solving the complex question.Source: §A.1.1 · p. 10
• When a sub-question cannot be solved, SearChain can further decompose or rewrite it and continue without IR assistance.Source: §A.1.1 · p. 10
• The Figure 6 case study shows SearChain continuing to reason while baselines stop.The authors connect this behavior to trying more when intermediate difficulties arise.Source: §A.1.1 · p. 10

A.1.1 Case Study for CoQ vs Baselines in Reasoning.

• The case study compares SearChain and New Bing on reference marking for generated content.Source: §A.1.2 · p. 10
• SearChain marks references for each knowledge item involved in the reasoning process, corresponding to each correct CoQ node.Source: §A.1.2 · p. 10
• The case study describes SearChain’s reference marking as more fine-grained.Source: §A.1.2 · p. 10

A.1.2 Case Study for SearChain vs New Bing in Tracing.

• SearChain decomposes complex knowledge-intensive tasks into single-step reasoning problems and marks supporting documents for each step.The paper says this organization lets supporting documents cover every item of knowledge in the generated content, enhancing traceability.Source: §A.1.2 · p. 10
• New Bing’s references may leave knowledge uncovered, and in some cases it cannot find the knowledge.The paper contrasts this with SearChain’s step-by-step reference marks.Source: §A.1.2 · p. 10
• Figure 7 presents a case study comparing SearChain and New Bing in tracing.Its caption identifies the comparison but does not report a specific outcome.Source: Figure 7 · p. 11

A.2 Performance on Vicuna-13B

• SearChain outperforms DSP on Vicuna-13B in the reported comparison.The passage identifies DSP as a competitive baseline; the table’s caption is not among this request’s evidence.Source: §A.2 · p. 11
• The comparison evaluates SearChain and DSP using Vicuna-13B.The passage describes Vicuna-13B as a strong open-source large model trained by Stanford.Source: §A.2 · p. 11
• In this section, experimental results in Table 6 show SearChain again outperforming competitive baseline DSP on Vicuna-13B 5, a strong open-source large model trained by Stanford.Source: §A.2 · p. 11

A.3.1 Threshold Selection.

• The confidence threshold is set to 1.5 after the highest F1-score was observed on mixed open-domain QA datasets.The threshold was initialized at 1.0 and increased in 0.1 steps; F1 measures recall and precision for judging whether a passage can answer the question.Source: §A.3.1 · pp. 11, 13
• For ELI5, performance change is not obvious when the ROUGE threshold ranges from 0.3 to 0.5.The threshold was selected by manually observing ROUGE relationships between generated text and ground truth in a few in-context examples.Source: §A.3.1 · p. 13
• SearChain achieves the best performance with fewer in-context learning examples than competitive baselines.The passage refers to prompt-example counts across datasets but provides no numerical counts here.Source: §A.3.2 · p. 13

A.3.3 Prompts in Experiment.

• Prompts are shown for Chain-of-Query generation across several datasets and retrieval settings.The supplied captions cover prompts without retrieval on HotpotQA, Musique, WikiMultiHopQA, zsRE, and T-REx, and with retrieval on those datasets, FEVER, ELI5, and StrategyQA.Source: Figure 8 · p. 12, Figure 11 · p. 12, Figure 12 · p. 12