LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval

Jiajie Jin; Yanzhao Zhang; Mingxin Li; Dingkun Long; Pengjun Xie; Yutao Zhu; Zhicheng Dou

Source-linked AI summary

LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval

Jiajie Jin, Yanzhao Zhang, Mingxin Li, Dingkun Long, Pengjun Xie, Yutao Zhu, Zhicheng Dou

arXiv:2603.01425v1Mar 2, 2026cs.CLcs.IR

TL;DR

Dense retrievers largely use reasoning-capable LLMs as static encoders, while explicit rewrite-then-retrieve pipelines add prohibitive autoregressive latency. LaSER uses shared-backbone explicit and latent views with output and trajectory alignment to internalize reasoning, outperform baselines while approaching rewrite-pipeline performance at dense-retriever efficiency.

Problem
LLM-based retrievers primarily optimize contrastive representations, leaving their reasoning capabilities insufficiently supervised for complex queries, while explicit reasoning pipelines incur prohibitive latency.
Method
LaSER uses a dual-view self-distillation framework in which an Explicit view teaches reasoning semantics and a Latent view learns implicit thinking through output and trajectory alignment.
Results
LaSER consistently outperforms explicit and implicit reasoning baselines and achieves performance comparable to rewrite-then-retrieve pipelines while maintaining low inference latency.
Takeaways & Limitations
Explicit reasoning-path semantics can be compressed into latent states, enabling efficient complex-query understanding without text generation during inference.
Takeaways & Limitations
LaSER is described as a preliminary step toward fully autonomous latent reasoning, with future work targeting reinforcement learning for its intermediate trajectory.

Abstract

from arXiv · show

LLMs have fundamentally transformed dense retrieval, upgrading backbones from discriminative encoders to generative architectures. However, a critical disconnect remains: while LLMs possess strong reasoning capabilities, current retrievers predominantly utilize them as static encoders, leaving their potential for complex reasoning unexplored. To address this, existing approaches typically adopt rewrite-then-retrieve pipelines to generate explicit CoT rationales before retrieval. However, this incurs prohibitive latency. In this paper, we propose LaSER, a novel self-distillation framework that internalizes explicit reasoning into the latent space of dense retrievers. Operating on a shared LLM backbone, LaSER introduces a dual-view training mechanism: an Explicit view that explicitly encodes ground-truth reasoning paths, and a Latent view that performs implicit latent thinking. To bridge the gap between these views, we design a multi-grained alignment strategy. Beyond standard output alignment, we introduce a trajectory alignment mechanism that synchronizes the intermediate latent states of the latent path with the semantic progression of the explicit reasoning segments. This allows the retriever to think silently and effectively without autoregressive text generation. Extensive experiments on both in-domain and out-of-domain reasoning-intensive benchmarks demonstrate that LaSER significantly outperforms state-of-the-art baselines. Furthermore, analyses across diverse backbones and model scales validate the robustness of our approach, confirming that our unified learning framework is essential for eliciting effective latent thinking. Our method successfully combines the reasoning depth of explicit CoT pipelines with the inference efficiency of standard dense retrievers.

1 Introduction

LLM-based retrievers offer stronger semantic understanding but generally leave their reasoning capabilities underused. LaSER distills explicit reasoning into latent retrieval while avoiding autoregressive generation.

LLM-based retrievers are mainly optimized for representational distinctiveness, leaving their inherited reasoning capabilities underused.
Complex queries involving implicit intent, multi-hop logic, or ambiguity often require more than superficial semantic matching.
Rewrite-then-retrieve pipelines improve performance through explicit query reasoning but incur prohibitive latency from verbose autoregressive generation.
LaSER uses Explicit and Latent views to transfer explicit reasoning capabilities into latent retriever states.
Trajectory alignment supplements output alignment by guiding latent tokens to capture the semantic progression of explicit Chain-of-Thought reasoning.
LaSER improves reasoning-intensive retrieval with only a few latent inference tokens and avoids autoregressive decoding.
LaSER consistently outperforms baselines across backbone architectures and model scales while matching the efficiency of standard dense retrieval.

2 Related Work

Related work spans data-centric and mechanism-centric approaches to reasoning-enhanced retrieval. LaSER targets the efficiency gap by internalizing rewriting and reasoning within a single-stage retriever.

Rewrite-then-retrieve systems use external LLMs for query expansion or reasoning rationales but add latency through autoregressive decoding.
LaSER internalizes rewriting and reasoning in retriever parameters to retain generative expansion benefits within a single-stage dual-encoder.
Reasoning-enhanced retrieval methods are broadly categorized as data-centric or mechanism-centric approaches.
Implicit methods generate latent thinking tokens or multi-vector representations without visible text generation, improving efficiency but limiting semantic supervision.
Knowledge distillation in information retrieval commonly transfers capabilities through output-level or feature-level alignment.

3 Methodology

LaSER is presented as a unified framework for internalizing explicit reasoning in dense retrievers. Its methodology centers on multi-grained self-distillation.

LaSER internalizes explicit reasoning capabilities into dense-retriever latent space through a unified framework.
The framework formalizes retrieval, then details training objectives and a multi-grained self-distillation mechanism.

3.1 Problem Formulation

Dense retrieval maps queries and documents into a shared embedding space and scores them by cosine similarity. LaSER replaces shallow single-pass query encoding with latent reasoning intended to preserve efficiency while capturing deeper semantics.

Dense retrieval learns an encoder that maps queries and documents into an m-dimensional embedding space and scores relevance with cosine similarity.
Standard LLM-based retrievers prepend task instructions, append an [EOS] token, and use its final hidden state as the query embedding.
Single-pass query encoding can fail to capture complex implicit intents in reasoning-intensive queries.
Rewrite-then-retrieve methods use an external LLM to generate explicit reasoning paths before the retriever encodes enriched context, but textual generation adds latency and deployment costs.
LaSER generates K continuous latent thinking tokens whose states produce the final query representation while maintaining a single-model architecture.

3.2 Overview of Framework

LaSER uses a shared generative LLM backbone with dual explicit and latent views. Training aligns these views, while inference uses only latent reasoning to avoid text generation.

LaSER uses a unified generative LLM backbone with causal attention and shared parameters for its dual-view training paradigm.
Explicit-View: The Explicit-View receives queries augmented with high-quality CoT rationales and produces representations synthesizing the query with its reasoning path.
Latent-View: The Latent-View takes only the original query, generates latent thinking tokens, and produces the query representation used during inference.
Inference: During inference, LaSER exclusively uses latent thinking tokens, targeting standard dense-retriever latency without text generation.

3.3 Latent View

The Latent-View performs continuous autoregressive reasoning by appending soft latent tokens to the query representation. It aggregates the resulting reasoning states while keeping inference latency comparable to standard dense retrievers.

The Latent-View replaces discrete token generation with autoregressive continuous thought vectors that refine query-intent understanding differentiably.
Given query embeddings, the backbone processes the sequence to produce initial hidden states before latent reasoning begins.
Figure 2 depicts distinct explicit and latent pathways during training, with only the latent pathway used for inference.
At each thinking step, the model projects the previous hidden state into vocabulary probabilities and computes a soft latent token as an expected embedding.
Each soft token is appended autoregressively, allowing subsequent steps to attend to the query and preceding latent tokens.
After K steps, mean-pooling over reasoning states produces a unified query representation, while typically K < 10 keeps latency comparable to standard retrievers.

3.4 Explicit-View

The Explicit-View acts as a semantic teacher by encoding queries together with externally generated CoT rationales. Its explicit reasoning states also provide intermediate signals for training alignment.

For each query, an external reasoner generates a comprehensive CoT rationale containing intermediate logic and potential query intent.
The augmented explicit input concatenates the original query, generated rationale, and [EOS] token.
The explicit path encodes this augmented input in one forward pass and uses its last hidden state as the final representation.
Intermediate hidden states from selected tokens in the explicit sequence provide signals representing the intermediate reasoning process.

3.5 Optimization via Self-Distillation

LaSER jointly trains both views with contrastive learning and multi-grained self-distillation. Output alignment transfers relevance preferences, while trajectory alignment supervises latent states to follow explicit reasoning semantics.

The joint objective combines contrastive learning with multi-grained distillation to transfer reasoning capability from the Explicit-View to the Latent-View.
Both views use standard InfoNCE loss with ground-truth positive documents and negatives to learn discriminative retrieval signals.
Output-Level Distillation: Score-based output distillation aligns relevance-score probability distributions rather than forcing identical query embeddings across views.
Output-Level Distillation: The latent view thereby learns the Explicit-View’s fine-grained document-relevance judgments without requiring identical embedding coordinates.
Process-Level Trajectory Alignment: Trajectory Alignment supervises intermediate latent thinking steps because output-only alignment can leave latent tokens without meaningful intermediate semantics.
Process-Level Trajectory Alignment: The method maps fixed-budget latent tokens to semantic keyframes in variable-length explicit reasoning paths and aligns their intermediate states.
Process-Level Trajectory Alignment: KL alignment makes latent tokens act as semantic checkpoints that approximate the explicit reasoning logic instead of drifting.

4 Experiments Design

The experiments train LaSER on synthetic reasoning-path data and evaluate it across in-domain and out-of-domain deep-reasoning benchmarks. They compare against dense, rewrite-then-retrieve, explicit-reasoning, and implicit-reasoning retrievers across multiple backbone scales and controlled training settings.

Datasets: 81k training examples across 12 domains provide GPT-4o-mini-generated reasoning paths for LaSER’s Explicit view.Each training query is paired with a reasoning path used as explicit input.
Evaluation: Evaluation covers BRIGHT, FollowIR, and BrowseComp-Plus, using nDCG@10, Recall@5/100/1000, p-MRR, MAP@5, and nDCG@5.FollowIR includes Robust’04, Core’17, and News’21 subsets with subset-specific metrics.
Baselines: Trainable baselines generally use the same backbone and training data as LaSER to support rigorous comparisons.This condition applies unless otherwise noted.
Baselines: LaSER is compared with standard dense retrievers, rewrite-then-retrieve pipelines, explicit reasoning methods, and the latent-token method GIRCSE.The standard-retriever group also includes a Fair Baseline trained with standard contrastive learning.
Implementation: Experiments use Qwen3 models from 0.6B to 8B and LLaMA 3.2 models from 1B to 3B, plus LLaMA 3.1-8B for the 8B scale.All models are fine-tuned for one epoch with LoRA on four A100 GPUs.

5 Results

LaSER improves reasoning-intensive retrieval over standard, explicit-pipeline, and implicit-reasoning baselines while maintaining a more efficient latent-thinking inference mode. Ablations and robustness analyses show that latent reasoning, explicit supervision, multi-grained alignment, and privileged distillation signals are central to its gains.

Main Results: LaSER (Qwen3-8B) achieves an average nDCG@10 of 29.3 on Bright, exceeding the original Qwen3-Embedding-8B (14.0) and Fair Baseline (25.7).The reported gain over the Fair Baseline is approximately 15%.
Main Results: LaSER surpasses explicit query-rewriting approaches by 0.7 points on Qwen3-0.6B and 1.2 points on Qwen3-8B.It also outperforms explicit methods that generate multiple thinking tokens, including Search-R3 and InBedder.
Main Results: Across 9 settings spanning model scales and datasets, LaSER outperforms the implicit-reasoning method GIRCSE in 8 cases.Both methods use identical input contexts and computational budgets; LaSER additionally uses process-level and output-level distillation.
Ablation Study: Removing the latent view reduces Bright performance from 23.10 to 19.93, indicating that latent thinking tokens expand the representational capacity beyond a single-vector retriever.Removing the explicit view also impairs performance because intermediate latent tokens lose explicit semantic guidance.
Ablation Study: Removing either output-level or process-level distillation significantly harms reasoning and generalization, while excluding output distillation causes the larger decline.Dynamic online teacher learning also improves results on both in-domain and out-of-domain datasets.
Robust Analysis: LaSER remains superior across all 6 Bright settings and reaches 23.10 on lightweight backbones versus 22.35 for the rewrite pipeline.Its latency is approximately 1.7× that of the standard retriever, with relative overhead decreasing at larger model scales; increasing inference K improves performance, whereas increasing training K from 3 to 6 does not.

6 Conclusion & Future Work

LaSER internalizes explicit Chain-of-Thought reasoning into dense retrievers through multi-grained alignment, combining reasoning depth with efficient retrieval. Experiments show strong performance against baselines, while future work targets further optimization of latent reasoning trajectories.

Conclusion: LaSER internalizes explicit CoT reasoning into the latent space of dense retrievers through self-distillation.Its multi-grained alignment synchronizes latent thinking tokens with explicit reasoning trajectories.
Conclusion: LaSER significantly outperforms state-of-the-art baselines while achieving performance comparable to computationally expensive rewrite-then-retrieve pipelines.The method maintains low inference latency, combining reasoning capability with retrieval efficiency.
Future Work: The authors describe LaSER as a preliminary step toward fully autonomous latent reasoning.Future work will explore reinforcement learning to optimize intermediate reasoning trajectories based on retrieval utility.