PaperlayerView exact revision on arXiv

Source-linked AI summary

When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs

Soyeong Jeong, Taehee Jung, Sung Ju Hwang, Joo-Kyung Kim, Dongyeop Kang

arXiv:2510.07499v2cs.CLcs.AIcs.LG

TL;DR

Long-context models can take in large evidence sets, but more context alone does not ensure that they connect evidence for multi-hop reasoning. The paper introduces reusable thought templates, refined through natural-language feedback, and reports consistent gains over baselines in retrieval and retrieval-free settings.

  • Problem

    Long-context models can process large evidence sets, but scaling document access alone does not address how to connect evidence for multi-hop reasoning.

    Source: §1 · p. 1, Knowledge-intensive Multi-hop Reasoning · p. 2

  • Method

    The framework constructs reusable reasoning templates from training data and iteratively refines them with natural-language feedback.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Results

    Thought templates consistently outperform standard prompting and RAG baselines across knowledge-intensive multi-hop benchmarks, in both retrieval-free and retrieval settings.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Analyses show that thought templates exhibit compositionality, transferability across models, and domain-awareness.

    Source: §6 · p. 9

    The approach assumes training queries and answers are available for template construction, a requirement that may be difficult to meet in low-resource domains.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent Long-Context Language Models (LCLMs) can process hundreds of thousands of tokens in a single prompt, enabling new opportunities for knowledge-intensive multi-hop reasoning by integrating large sets of retrieved documents or, in some cases, directly all necessary information. However, simply feeding more documents into the context window fails to capture how evidence should be connected. We address this gap with thought templates, which recast reasoning as reusable thought caches, derived from prior problem solving traces, structuring how evidence is combined and guiding multi-hop inference with factual documents. To keep these templates effective, we propose an update strategy that iteratively refines templates derived from training data through natural-language feedback. Across diverse benchmarks and LCLM families, our approach delivers consistent gains over strong baselines in both retrieval-based and retrieval-free settings. Furthermore, we show that optimized templates can be distilled into smaller open-source models, demonstrating its broad applicability and transparent reasoning reuse. We refer to our framework as Thought Template Augmented LCLMs (ToTAL).

1 Introduction

Knowledge-intensive multi-hop reasoning requires connecting evidence across documents, a capability that simply enlarging LCLM contexts does not ensure. The section introduces reusable thought templates refined through feedback and reports gains in retrieval-free and retrieval settings.

  • Multi-hop tasks require connecting evidence across documents, while larger LCLM contexts alone do not ensure that evidence is integrated.Source: §1 · p. 1
  • Thought templates cache reusable reasoning patterns while documents provide facts, and LCLMs selectively combine relevant templates for each query.Source: §1 · pp. 1–2
  • Templates are constructed compositionally from multi-hop QA data and refined iteratively with natural-language feedback derived from model errors.The feedback specifies revisions like a gradient update without changing model weights.Source: §1 · p. 2
  • Thought templates consistently boost LCLM performance in retrieval-free and retrieval settings, while feedback-driven updates add gains.Source: §1 · p. 2

2 Proposed Method

The proposed method addresses the limits of evidence aggregation in long-context multi-hop reasoning with reusable templates that organize evidence without fine-tuning model weights. It constructs templates from training data and iteratively refines low-performing ones through feedback.

Knowledge-intensive Multi-hop Reasoning

2.1 Preliminaries

Knowledge-intensive Multi-hop Reasoning

2.2 Thought Template Augmented LCLMs

  • TOTAL uses thought templates—structured thought processes built from training data and refined through textual-gradient feedback—to guide evidence organization and multi-step inference without model fine-tuning.Source: §2.2 · pp. 2–3
  • Templates decompose example-specific solutions into reusable sub-templates that can be selectively composed across queries.Initial templates are generated using training queries and gold answers, optionally with solution paths.Source: §2.2 · p. 3
  • Template performance is measured on queries where each template is selected; low-scoring templates are targeted for refinement while stronger ones remain stable.An auxiliary language model compares the query, prediction, gold answer, and applied template to produce textual feedback.Source: §2.2 · p. 3

3 Experimental Setup

The experiments evaluate TOTAL on four multi-hop QA benchmarks against LCLM baselines, using retrieval-free and retrieval-based settings. The setup also specifies model and metric choices, corpus construction, and template sampling and validation.

3.1 Datasets

  • The evaluation covers MuSiQue, CRAG, FanOutQA, and Housing QA, spanning multi-passage reasoning, dynamic queries, long-context Wikipedia documents, and statutory legal questions.Source: §3.1 · p. 4

3.2 Baseline Models

  • TOTAL is compared with NAÏVE, chain-of-thought prompting, corpus-in-context, and CIC combined with chain-of-thought; its reusable evidence-guiding patterns require no additional fine-tuning.Source: §3.2 · p. 4

3.3 Implementation Details

  • Claude-Sonnet 4, Gemini 2.5 Flash, GPT-4.1, OSS (120B), and DeepSeek-R1 are evaluated; BM25 retrieves documents, with F1 used for three datasets and Accuracy for Housing QA.Source: §3.3 · p. 4

3.4 Data Processing

  • Corpora are capped or subsampled to fit a 128k-token budget, while FanOutQA retrieval uses its 2,142-document corpus and retrieves five documents.Initial templates are built from 50 training QA pairs without test-query overlap; a separate training subset supports updates, and the threshold τ is set on validation data. Table 1 reports dataset metrics and an overall Average; Table 2 reports retrieval results.Source: §3.4 · p. 4, Table 1 · p. 5, Table 2 · p. 5

4 Results and Analyses

Across retrieval-free and retrieval-based benchmarks, TOTAL consistently outperforms baselines, with iterative updates adding gains. Analyses show that template quality, quantity, transferability, and use patterns shape the approach’s results.

4.1 Main Results

  • TOTAL consistently outperforms all baselines across datasets, while naïve, CoT, and corpus-in-context prompting yield limited gains on complex multi-hop queries.The comparison highlights reusable templates as structured reasoning guidance rather than evidence aggregation or surface-level prompting.Source: §4.1 · p. 5

4.2 When Partial Context Given

  • As retrieved documents increase, recall and QA performance improve, but retrieval remains a bottleneck; templates improve performance across retrieval sizes.TOTAL also outperforms CIC when both systems receive the same retrieved documents.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3 Effectiveness of Template Update Strategy

  • Iterative natural-language feedback improves the initial template set, with performance gains plateauing around the second update iteration.Even without updates, the initial templates outperform CIC.Source: §4.3 · p. 6, Figure 4 · p. 6

4.4 Transferability of Templates

  • Templates transfer across frontier LCLMs and improve open-source models, including in shorter-context and retrieval-augmented settings.Templates generated by an open-source model can surpass CIC, though frontier-derived templates perform better.Source: §4.4 · p. 6, Table 3 · p. 6, Table 4 · p. 6
  • FanOutQA templates substantially improve results on a closely related dataset, whereas Housing QA templates transfer less effectively; broader cross-domain reuse remains future work.The contrast indicates that templates capture domain-specific reasoning structures.Source: §4.4 · p. 6, Table 5 · p. 6

4.5 Impact of Template Quality; 4.6 Impact of Template Quantity

  • Compositional templates generalize better than holistic ones, oracle templates show a higher performance ceiling, and the top-scoring 25% remains competitive while the full pool performs best.Removing compositionality causes a measurable performance drop.Source: §4.5 · p. 7, Table 6 · p. 7, §4.6 · p. 7

4.7 Template Analyses Beyond Performance

  • Queries cluster with their templates and usage is long-tailed; co-occurrence is more flexible across several datasets than in Housing QA, where a few rigid bundles dominate.Seven of the ten most frequently co-occurring MuSiQue templates come from different training queries.Source: §4.7 · pp. 7–8, Figure 7 · p. 7, Figure 8 · p. 8

4.8 Qualitative Study

  • In a case study, TOTAL connects the same retrieved facts into an interpretable multi-step explanation and answers a query that CIC leaves unanswerable.The template-guided steps link the artwork to its creator, locate the creator’s biographical and geographical context, and connect those facts to the historical event.Source: §4.8 · p. 8

5 Related Work

Related work covers long-context models, structured reasoning templates, and natural-language feedback as a surrogate for model optimization. The paper distinguishes its reusable, compositional templates from approaches that retrieve a single trace or refine prompts.

Long-Context Language Models

  • Long-context models extend input capacity to hundreds of thousands or millions of tokens, enabling retrieved evidence and many in-context examples to fit in one prompt.Recent benchmarks also probe these capabilities (Li et al., 2023; Zhang et al., 2024; Yang et al., 2025c; Bai et al., 2025; Lee et al., 2025).Source: §5 · p. 8

Reasoning with Thought Templates

  • Prior thought-template work stores or searches reasoning traces, but often targets narrow domains or simple steps without factual knowledge; TOTAL composes multiple templates within one generation.Earlier approaches include Yang et al. (2024b, 2025a), Wu et al. (2024), and Yang et al. (2025b).Source: §5 · p. 8

Text Gradient

6 Conclusion

TOTAL combines factual documents with reusable thought templates and refines those templates through textual feedback without modifying model parameters. It consistently outperforms standard prompting and RAG baselines across retrieval-free and retrieval-based multi-hop benchmarks.

6 Conclusion

  • TOTAL consistently outperforms standard prompting and RAG baselines on diverse knowledge-intensive multi-hop benchmarks, with and without retrieval.Source: §6 · p. 9
  • Thought templates combine factual documents with reusable reasoning patterns, refined through textual gradients without modifying model parameters.Source: §6 · p. 9
  • Analyses find that thought templates exhibit compositionality, transferability across models, and domain-awareness, alongside improvements in factual accuracy.Source: §6 · p. 9

Limitations

The method has data, feedback, and template-format boundaries: construction assumes training queries and answers, feedback may be noisy, and the current design uses textual templates.

Limitations

  • Template construction assumes training queries and answers, a requirement that may be difficult to satisfy in low-resource domains.Source: Limitations · p. 9
  • An auxiliary model produces update feedback, which may be biased or noisy and lead to suboptimal refinement.Source: Limitations · p. 9
  • The current design focuses on textual templates; more structured templates or multimodal contexts remain possible extensions.Source: Limitations · p. 9

Ethics Statement

Because the method supplies LCLMs with many evidence documents or sometimes an entire corpus, those materials may contain harmful, sensitive, or personally identifiable information.

Ethics Statement

  • Supplying many evidence documents, or sometimes an entire corpus, may expose harmful, sensitive, or personally identifiable information.Source: Ethics Statement · p. 9
  • The stated risk concerns evidence documents provided to LCLMs, including cases where the entire corpus is used.Source: Ethics Statement · p. 9
  • Practitioners are advised to remain mindful of these risks and consider bias detection and mitigation strategies when deploying the method.Source: Ethics Statement · p. 9

A.1 Template Statistics

Templates are generated from question–answer pairs and then iteratively updated. The initial pool sizes are reported separately for four datasets.

A.1 Template Statistics

  • The initial templates are generated from 50 question–answer pairs using a detailed prompt shown in Figure 15.Source: §A.1 · p. 13
  • The initial template pools contain 172 templates for MuSiQue, 162 for CRAG, 133 for FanOutQA, and 149 for HousingQA.Source: §A.1 · p. 13
  • After construction, the templates are iteratively updated as described in the following subsection.Source: §A.1 · p. 13

A.2 Template Update Strategy

The update strategy iteratively refines initial templates using textual-gradient feedback and KEEP, ADD, FIX, or DISCARD actions. On MuSiQue and CRAG, performance gains peak around the second iteration, with further refinement sometimes reducing performance.

A.2 Template Update Strategy

  • Textual-gradient feedback guides iterative template refinement through four actions: KEEP, ADD, FIX, and DISCARD.Source: §A.2 · p. 13
  • Early template updates are mostly FIX, while KEEP and ADD become more frequent as high-quality templates stabilize and occasionally gain variants.Source: §A.2 · p. 13
  • On MuSiQue, F1 improves through the second iteration and then slightly declines; on CRAG, gains continue through the second iteration before a third-iteration drop from over-refinement.Source: §A.2 · p. 13

A.3 Efficiency and Cost Analysis

  • Construction and update are performed offline, with Table 8 reporting average token usage and API calls per template.Source: Table 8 · p. 13, §A.3 · p. 13
  • At inference, TOTAL uses no additional model calls and has per-query cost comparable to CIC, aside from moderately more input and output tokens.Source: Table 9 · p. 13, §A.3 · p. 13

A.4 Qualitative Study on Template Update

Related papers

Loading 2510.07499v2…