Source-linked AI summary
Retrieval Grounding Latent Reasoning for Dense Retrieval
Gang Zhou, Xiongxi Yu, Hu Tian, Yang Wei, Lu Pan, Ke Zeng, Shibiao Xu, Xiaolong Zheng
TL;DR
Reasoning-intensive retrieval needs representations that capture multi-stage relevance reasoning, but existing latent-reasoning supervision often focuses on final retrieval outcomes. RGLT grounds non-autoregressive latent transitions in stage-wise retrieval effects and consistently outperforms strong baselines while preserving efficient inference.
Problem
Existing latent-reasoning retrievers lack direct supervision connecting intermediate state transitions with incremental retrieval gains.
Method
RGLT uses instruction-conditioned silent-token trajectories with process distillation, intermediate retrieval supervision, and stage-level retrieval-effect credit assignment.
Results
Across reasoning-intensive benchmarks, RGLT consistently outperforms strong baselines; on BRIGHT it achieves 34.20 nDCG@10 and 39.97 Recall@10.
Takeaways & Limitations
RGLT enables the terminal latent state to serve directly as a single-vector query embedding without autoregressive CoT generation or additional pooling.
Abstract
from arXiv · showhide
Reasoning-intensive retrieval requires text representations to capture not only semantic similarity, but also the reasoning needed to determine relevance under a given retrieval instruction. Existing reasoning-enhanced embedding models improve retrieval by incorporating reasoning information into dense representations, yet their supervision is typically dominated by the final retrieval objective. As a result, latent reasoning trajectories may learn shortcut reasoning patterns that preserve retrieval performance without producing meaningful incremental retrieval gains. We propose Retrieval Grounding Latent Reasoning (RGLT), a latent reasoning framework for dense retrieval that explicitly connects intermediate latent transitions with retrieval improvements. RGLT performs non-autoregressive reasoning in hidden space through an instruction-conditioned latent reasoning trajectory constructed from silent tokens. It combines process-supervised explicit-to-implicit distillation with retrieval-grounded supervision, using stage-wise CoT reconstruction to shape intermediate latent states and retrieval-effect credit to optimize incremental retrieval gains across the latent reasoning trajectories. Experiments on reasoning-intensive retrieval benchmarks show that RGLT consistently outperforms strong baselines while preserving efficient embedding inference.
1 Introduction
Reasoning-intensive retrieval requires representations that capture multi-stage reasoning beyond surface similarity, but existing latent-reasoning retrievers do not explicitly supervise whether intermediate transitions improve retrieval. RGLT addresses this gap with non-autoregressive, instruction-conditioned latent trajectories trained through stage-wise CoT distillation and retrieval-effect supervision.
- Motivation: Reasoning-intensive retrieval must resolve implicit constraints and intermediate concepts because relevant documents may share little surface-level similarity with specialized queries.
- Prior Work: Explicit-CoT retrieval methods add autoregressive latency and depend on rationale verbosity and lexical form, motivating continuous latent reasoning.
- Problem: Existing latent-reasoning methods optimize final retrieval, representation similarity, or state alignment without explicitly modeling incremental retrieval effects during intermediate transitions.
- Method: RGLT constructs an ordered latent trajectory from fixed silent tokens and injects retrieval-oriented guidance through a query- and instruction-conditioned retrieval-intent anchor.
- Method: RGLT assigns each latent transition credit for recovering the ranking gain of its corresponding explicit CoT stage, grounding intermediate reasoning in retrieval improvement.
- Results: RGLT surpasses autoregressive counterparts on three reasoning-intensive benchmarks while substantially reducing inference latency.
2 Related Work
Prior retrieval work improves reasoning-intensive retrieval through explicit reasoning, reasoning-oriented data, instruction-tuned dense representations, query expansion, and latent reasoning. RGLT most closely differs from LaSER by optimizing intermediate latent transitions for retrieval gains rather than only aligning states or final outputs.
- 2 Related Work: Reasoning-oriented methods construct training instances, while Search-R3 (Gui & Cheng, 2025) and GRACE (Sun et al., 2025) generate reasoning before forming retrieval representations.Examples include ReasonIR (Shao et al., 2025), RaDeR (Das et al., 2025), and ReasonEmbed (Chen et al., 2026).
- 2 Related Work: LLM-based retrievers use instruction tuning, synthetic supervision, and joint generative–representational learning while encoding queries in a single forward pass.These approaches include Wang et al. (2024), Muennighoff et al. (2025), and Zhang et al. (2025a).
- 2 Related Work: Query expansion externalizes reasoning into text through hypothetical or pseudo-documents, as illustrated by HyDE (Gao et al., 2023) and Query2doc (Wang et al., 2023).DIVER (Sun et al., 2026) is also identified as an iterative query expansion method.
- 2 Related Work: Latent-reasoning approaches replace textual reasoning with hidden-space computation, including Coconut (Hao et al., 2025), CODI (Shen et al., 2025), GIRCSE (Tsai et al., 2025), and AdaQR (Zhang et al., 2025b).These methods use continuous thoughts, distilled CoT representations, soft embedding tokens, or multi-query relevance mechanisms.
- 2 Related Work: RGLT differs from LaSER by transferring stage-specific retrieval effects from explicit CoT to latent transitions and directly optimizing intermediate retrieval improvements.LaSER aligns explicit and latent trajectories but does not assess whether transitions produce actual retrieval gains.
3 Methodology
RGLT performs instruction-conditioned, non-autoregressive latent reasoning by evolving fixed silent-token states within a single encoder pass. It trains these stages with explicit-to-implicit reconstruction and retrieval-grounded supervision so intermediate transitions progressively improve retrieval discrimination.
- Latent Reasoning Architecture: RGLT appends K fixed silent tokens to an instruction-conditioned query, forming a latent reasoning trajectory whose successive states refine the retrieval representation in one forward pass.This avoids autoregressive decoding while preserving efficient embedding inference.
- Instruction Conditioning: A unified query-instruction anchor conditions silent-token attention through gated residual modulation, while query-only application preserves reusable document embeddings in dual-encoder retrieval.The terminal silent token summarizes the trajectory and serves directly as the final query embedding.
- Process-Supervised Distillation: Process-supervised explicit-to-implicit distillation divides K silent tokens into S stages and requires each stage to reconstruct the cumulative explicit-CoT prefix available up to that point.Only tokens through stage k × B are exposed for reconstructing r≤k, preventing information leakage and encouraging progressively richer reasoning.
- Retrieval-Grounded Supervision: Retrieval supervision is applied to intermediate and final latent states so the trajectory maintains retrieval discrimination instead of receiving signal only at its endpoint.Competitive normalization over relevant documents and hard negatives makes improvements relative within the local candidate set.
- Retrieval-Grounded Supervision: The retrieval-effect loss transfers stage-specific gains from explicit CoT to latent transitions, matching retrieval improvements rather than forcing latent states to reproduce explicit representations.This targets retrieval-effective transitions that progressively improve discrimination across reasoning stages.
4 Experiments
Across four research questions, RGLT improves reasoning-intensive retrieval, progressively strengthens representations through latent stages, and retains efficient single-vector inference without textual chain-of-thought generation. Experiments also identify retrieval-effect supervision and a latent budget of K = 16 as important design choices.
- RQ1: Effectiveness: 34.20 nDCG@10 and 39.97 Recall@10 make RGLT the best completed BRIGHT result, improving over LaSER by 14.4% and 16.6%, respectively.RGLT also exceeds the Fair Baseline by 8.50 nDCG points and outperforms GIRCSE and Rewrite-then-Retrieve by 8.20 and 6.10 nDCG points.
- RQ1: Effectiveness: RGLT improves nDCG@10 in 9 of 12 domains and Recall@10 in 11 of 12, indicating broader and more balanced retrieval rather than domain-specific gains.LaSER remains better on biology, earth science, and StackOverflow, although the gaps on biology and earth science are small.
- RQ2: Components: Retrieval-effect grounding raises nDCG@10 from 30.87 for naive terminal-ST to 34.20, outperforming gated fusion at 32.14 and state alignment.The proposed design also increases Recall@10 from 37.94 to 39.97 versus gated fusion, showing that stage terminals can directly serve as retrieval representations.
- RQ3: Representation evolution: nDCG@10 rises from 26.39 at the base state to 32.25, 32.48, 33.56, and 34.20 across ST4, ST8, ST12, and ST16.Recall@10 follows the same progressive trend, increasing from 30.24 to 39.97, so retrieval utility accumulates throughout the latent trajectory.
- RQ4: Efficiency and sensitivity: RGLT avoids textual CoT generation, preserves single-vector nearest-neighbor search and unchanged document encoding, while textual CoT incurs 186× latency and LaSER incurs 1.51×.Performance peaks at K = 16 with four tokens per stage and drops at K = 24, suggesting that larger latent budgets complicate optimization.
5 Conclusion
RGLT is a latent reasoning framework for reasoning-intensive dense retrieval that grounds each latent transition in the retrieval effect of its corresponding CoT stage. This supervision learns an ordered evolution of retrieval representations, with the terminal latent state serving directly as a single-vector query embedding.
- 5 Conclusion: RGLT grounds each latent transition in the retrieval effect of its corresponding CoT stage, rather than supervising only the final representation or aligning intermediate states.This explicitly connects latent reasoning progression to retrieval improvement.
- 5 Conclusion: The resulting ordered latent evolution produces a terminal state that can be used directly as a single-vector query embedding.
Supplementary Material
The supplementary material provides implementation details for reproducing RGLT, covering CoT structuring, memory-efficient training, evaluation protocols, and inference efficiency. At inference, queries use the final silent state while documents remain reusable single vectors.
- The supplement details how CoT sequences are structured and segmented for RGLT implementation.
- A deterministic gradient replay strategy makes training tractable under ultra-long CoT branches and a cross-device document pool.
- The supplement specifies evaluation protocols and efficiency optimizations, including KV-…
- At inference, each query is represented by the final silent state, while each document remains a reusable single vector.
A Structured Segmentation of CoT
The method segments each chain-of-thought sequence into four reasoning stages, then supervises cumulative prefixes so later stages retain earlier deductions and logical coherence.
- Stage identification: CoT sequences are segmented into four stages: problem identification, reasoning and relevant information, detailed solution, and verification and answer convergence.A robust parser prioritizes high-level semantic headings and structural step markers to handle raw texts without standardized paragraph boundaries.
- Cumulative supervision: The four segments are trained as cumulative targets rather than isolated slices, preserving prior deduction history for stage-wise supervision.For stage k, the target is r≤k = r1∥r2∥· · · ∥rk, where ∥ denotes concatenation with a blank line.
B Training under Memory Constraints
Training uses a frozen Qwen3-8B backbone with low-rank LoRA and RGLT parameters, optimizing a joint objective that combines retrieval, reasoning reconstruction, and retrieval-effect supervision.
- Training configuration: Training augments a frozen Qwen3-8B base with LoRA (rank 32) and RGLT (rank 8) parameters.The same frozen backbone and parameter sets are reused throughout training.
- Training objective: The joint objective combines final retrieval, per-stage retrieval credit, CoT reconstruction, and retrieval-effect matching between silent-stage and explicit CoT-prefix transitions.These four terms connect retrieval performance with intermediate reasoning transitions.
C Evaluation Details and Inference Optimization · C.1 Benchmark Protocol
The benchmark protocol evaluates instruction-conditioned retrieval across BRIGHT, FollowIR, and BrowseComp-Plus using task-specific metrics and controlled document/query encoding procedures. Results are macro-averaged where specified, with additional evaluation of instruction-driven ranking changes on FollowIR.
- C.1 Benchmark Protocol: BRIGHT evaluates all 1,384 queries across complete 12-domain corpora using query instructions, 512-token maximum length, and macro-averaged domain nDCG@10 and Recall@10.LeetCode documents are split into four shards and merged before scoring; evaluation recomputes document vectors and records model and checkpoint fingerprints in cache metadata.
- C.1 Benchmark Protocol: FollowIR uses official MTEB 1.38.32 instruction-retrieval tasks for Robust04, News21, and Core17, with MAP@5, nDCG@5, and MAP@5 as their respective standard metrics.Each metric is macro-averaged across the three tasks.
- C.1 Benchmark Protocol: FollowIR passes queries and instructions through separate contextual-anchor channels while keeping documents instruction independent.This separates instruction conditioning from document encoding in the benchmark protocol.
- C.1 Benchmark Protocol: FollowIR additionally reports official p-MRR to measure whether changed instructions move newly relevant documents upward and newly irrelevant documents downward.The p-MRR score is macro-averaged over the three tasks alongside the standard metrics.
- C.1 Benchmark Protocol: The protocol preserves document-side consistency by using instruction-independent document representations for FollowIR and a fixed document-only index for BrowseComp-Plus.These procedures prevent query-time instruction changes from modifying the evaluated document representations.
- C.1 Benchmark Protocol: BrowseComp-Plus reports Recall@5, Recall@100, and Recall@1000 on its fixed corpus, queries, and relevance labels.The document index is built once from the document-only path and remains unchanged at query time.
C.2 KV-Cache Inference and Efficiency Measurement
RGLT inference reuses the shared query-prefix KV cache so the anchor and latent branches avoid duplicate encoding while preserving full-recomputation equivalence. This design limits extra computation to two short suffix blocks and yields approximately 1.2× latency rather than 2×.
- Query KV-Cache Reuse: The anchor and latent branches reuse separate cache objects pointing to the same read-only query-prefix tensors, while temporary instruction and anchor caches are discarded before latent processing.The latent branch starts directly from the original Cq and bypasses the instruction text.
- Preserving Positional Integrity: Strict positional tracking preserves equivalence to full recomputation when sharing caches across variable-length batches, with left-padding and per-row suffix position IDs.All K silent tokens are processed concurrently in a single c…
- Strict Validation and Profiling: Cached inference is validated against uncached recomputation by requiring identical top-k rankings and near-floating-point-epsilon maximum embedding differences.Latency profiling isolates prefill, anchor-suffix, and latent-suffix regions with torch.cuda.synchronize().