PaperlayerView exact revision on arXiv

Source-linked AI summary

CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning

Jie He, Richard He Bai, Sinead Williamson, Jeff Z. Pan, Navdeep Jaitly, Yizhe Zhang

arXiv:2511.18659v3cs.CL

TL;DR

RAG systems face separately optimized retrieval and generation, with discrete document selection limiting retriever feedback from generation. CLaRa compresses documents into shared continuous representations and jointly trains retrieval and generation; it reports strong QA benchmark performance, including at a text compression ratio of 16.

  • Problem

    RAG retrieval and generation are often optimized separately, and discrete document selection blocks generator gradients from informing retriever learning.

    Source: §1 · p. 1

  • Method

    CLaRa combines salient-information compressor pretraining with shared compressed document representations and joint query-reasoner and generator training using a language-modeling objective.

    Source: §1 · pp. 1–2, §3 · p. 5

  • Results

    CLaRa reports state-of-the-art retrieval and generation performance across QA benchmarks, surpassing text-only DRO methods at text compression ratio 16.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Shared continuous representations can reduce generator input length and redundant encoding while allowing retrieval and generation to be optimized in a common semantic space.

    Source: §1 · p. 2

    The compressors are pretrained primarily on Wikipedia data, which may reflect coverage biases in large-scale public corpora.

    Source: Impact Statement · p. 11

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge but still suffers from long contexts and disjoint retrieval-generation optimization. In this work, we propose CLaRa (Continuous Latent Reasoning), a unified framework that performs embedding-based compression and joint optimization in a shared continuous space. To obtain semantically rich and retrievable compressed vectors, thereby reducing the document length fed into the generator, we introduce SCP, a key-preserving data synthesis framework based on question answering and paraphrase supervision. CLaRa then trains the reranker and generator end-to-end via a single language modeling loss, with gradients flowing through both modules using a differentiable top-k estimator. Theoretically, this unified optimization aligns retrieval relevance with answer quality. Experiments across multiple QA benchmarks show that CLaRa achieves state-of-the-art compression and reranking performance, even at a text compression rate of 16, outperforming text-based fine-tuned baselines.

1 Introduction

CLaRa addresses the separation between retrieval and generation by using shared compressed document representations and joint optimization. Across QA benchmarks, it reports state-of-the-art retrieval and generation performance, including at a text compression ratio of 16.

1 Introduction

  • RAG systems separately optimize retrieval and generation, leaving retrievers to rely on surface similarity without generator feedback about needed information.Discrete document selection also prevents gradients from flowing from the generator back to the retriever.Source: §1 · p. 1
  • CLaRa encodes each document once into compact continuous representations shared by retrieval and generation, preserving salient information while reducing generator input length.The shared representation is intended to make retrieval differentiable and support joint optimization in a common semantic space.Source: §1 · pp. 1–2
  • SCP uses salient-information pretraining, while Stage II jointly trains the query reasoner and generator with next-token loss and differentiable top-k selection.The unified objective provides gradients for retriever learning without explicit labels.Source: §1 · p. 2
  • At text compression ratio 16, CLaRa surpasses text-only DRO methods and achieves state-of-the-art retrieval and generation performance, outperforming supervised and unsupervised baselines.Evaluation covers four single-hop and multi-hop QA benchmarks with Mistral-7B and Phi-4B.Source: §1 · p. 2

2 SCP: Salient Compressor Pretraining

SCP trains a compressor to retain salient document semantics rather than spend limited capacity on token-by-token reconstruction. It combines QA and paraphrase supervision with compressor training and semantic alignment.

2 SCP: Salient Compressor Pretraining

  • SCP replaces token-level reconstruction with pretraining data that emphasizes salient information, so the compressor learns to retain essential semantics.The intended compressed representations support answering from shorter inputs.Source: §2 · pp. 2–3

2.1 Guided Data Synthesis for Semantic Preservation

  • Synthetic data construction elicits QA and paraphrases, verifies factual consistency and coverage, and regenerates missing content before training.Samples that fail final coverage criteria are excluded after up to ten verification-and-regeneration rounds.Source: §2.1 · p. 3
  • Simple QA targets distinct atomic facts, complex QA links multiple facts, and paraphrases preserve meaning while varying surface expression.Together, QA and paraphrases provide complementary factual-grounding and linguistic-compactness signals.Source: §2.1 · p. 3

2.2 Compressor Pretraining

  • The compressor appends learnable memory tokens to a document and uses their final-layer hidden states as its compressed representation.A generator receives the compressed representation with an instruction and is trained on QA or paraphrase targets.Source: §2.2 · p. 4
  • Compression alignment minimizes MSE between averaged document-token and memory-token hidden states, and the total loss balances alignment with generation quality.For downstream QA, optional instruction tuning jointly fine-tunes compressor and answer-generator adapters using retrieved documents, instructions, and teacher responses.Source: §2.2 · p. 4

3 CLaRa: Retrieval and generation joint training

CLaRa unifies retrieval and generation over shared compressed document representations, training its query reasoner and generator through a shared language-modeling objective. Differentiable top-k selection lets generation provide retrieval feedback without explicit relevance labels.

3 CLaRa: Retrieval and generation joint training

  • The query reasoner scores compressed document embeddings by cosine similarity, and the top-k embeddings are concatenated with the query for answer generation.The query reasoner is initialized from the compressor and learns to anticipate relevant document content.Source: §3 · p. 5
  • CLaRa trains the query reasoner and generator via a shared language-modeling loss, using generation as weak retrieval supervision without explicit relevance labels.The compressor remains frozen to support offline document encoding while the query reasoner and generator are optimized.Source: §3 · p. 5
  • Discrete top-k retrieval blocks generator gradients to the retriever, so CLaRa uses straight-through estimation to preserve discrete forward selection while enabling soft-selection gradients.The method is presented as an alternative to repeated retrieval sampling, which prior RL-based training is described as making unstable and inefficient.Source: §3 · pp. 5–6
  • The authors describe two complementary retriever signals: ranking documents that support correct generation and representation-level feedback that makes embeddings more useful for downstream reasoning.They argue that this dual feedback stabilizes joint training by progressively aligning retrieval and generation in a shared semantic space.Source: §3 · p. 6
  • In the query-reasoner case study, decoded tokens “NFL” and “Oklahoma” do not appear in the question but occur in the positive document as clues for answering it.The figure marks newly reasoned tokens in red and key evidence in blue.Source: §3 · p. 6, Figure 4 · p. 6

4 Experiments

Experiments evaluate CLaRa’s compression, end-to-end QA, and retrieval across QA benchmarks and compare it with text-based and compression baselines. Results show gains over compression baselines, competitive QA with shorter context, and strong retrieval under weak generation supervision.

4.1 Experimental setup

  • Experiments use NQ, HotpotQA, MuSiQue, and 2WikiMultihopQA, and compare compression, reranking, and end-to-end QA against established methods, including jointly optimized systems.The evaluation describes CLaRa as jointly optimizing reranking and generation over compressed representations, unlike the raw-text baselines.Source: §4.1 · pp. 6–7

4.2 Evaluation of Compression Effectiveness

  • CLaRa’s compressed representations outperform compression baselines, with average gains of 1.13% (Normal) and 5.35% (Oracle) over PISCO and 5.37% and 17.31% over LLMLingual-2.Normal retrieves the top five documents; Oracle includes an annotated positive among them.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Joint Training Results

  • At 16× compression, CLaRa-Mistral-7B improves F1 over text-based DRO-Mistral-7B from 51.01 to 51.41 on NQ and 43.65 to 47.18 on 2Wiki; Oracle F1 exceeds 75% on NQ and HotpotQA.Normal-setting performance is stable across compression ratios and peaks at 16–32×.Source: §4.3 · p. 8, Table 2 · p. 8
  • Instruction-tuned initialization performs better in Normal conditions, especially on NQ and HotpotQA, while the initialization gap narrows with Oracle retrieval.The authors report robust performance across retrieval qualities and compression ratios.Source: §4.3 · p. 8

4.4 Retrieval performance

  • For retrieval, pretraining-initialized CLaRa consistently outperforms its instruction-tuned version; instruction tuning is reported to improve answer generation but bias retrieval toward localized evidence.The authors identify global semantics as important for retrieval.Source: §4.4 · p. 9
  • On HotpotQA at compression ratio 4, pretraining-initialized CLaRa reaches Recall@5 of 96.21%, versus 85.93% for BGE-Reranker, a +10.28% difference.The comparison is reported despite CLaRa using weak generation supervision rather than annotated relevance labels.Source: Figure 5 · p. 8, §4.4 · p. 9

5 Ablation Study

The ablations examine how pretraining data composition and an MSE alignment loss affect compressed representations. Diverse QA and paraphrase objectives perform best, while MSE yields modest but consistent gains and improves embedding alignment.

Pretraining Data Mix

  • Pretraining pairs documents with QA-style question–answer examples and paraphrased documents.The study varies the pretraining objectives and reports results under Oracle evaluation.Source: §5 · p. 9
  • Using SimpleQA or Paraphrase alone already outperforms the no-pretraining baseline for both Mistral-7B and Phi4-mini.Source: §5 · p. 9
  • Combining SimpleQA with ComplexQA, or adding paraphrases, achieves the best performance, with benefits especially evident under Oracle retrieval.The authors associate diverse objectives with greater semantic coverage and generalization.Source: §5 · p. 9

Effect of MSE Loss

  • Adding MSE alignment improves performance by an average of 0.3–0.6 points across datasets and produces stronger overlap between compressed and original document embeddings.Without MSE, the t-SNE distributions are described as clearly separated.Source: §5 · pp. 9–10

6 Related Work

Prior compression methods often lack retrieval–generation co-optimization or preserve insufficiently fine-grained semantics. CLaRa combines compressed shared representations with differentiable end-to-end optimization.

6.1 Embedding-based/Soft Compression

  • Single-vector projection methods can lose fine-grained semantics, while query-aware compression requires recompression per query and increases latency (Cheng et al., 2025; Louis et al., 2025b).PISCO jointly trains encoder and decoder, but the authors argue targeted pretraining can still improve representations (Louis et al., 2025a).Source: §6.1 · p. 10

6.2 End-to-End Optimization for RAG

  • Reinforcement-learning approaches allow joint optimization but are unstable and computationally heavy, while differentiable reranking still processes full documents (Shi et al., 2025; Huang et al., 2025).Both approaches leave representation mismatch and context-length issues unresolved.Source: §6.2 · p. 10
  • CLaRa uses length-flexible compressed vectors in a shared latent space for differentiable selection and shorter contexts, training with generator language-modeling loss alone.The authors describe this as aligning training and inference without explicit retrieval supervision.Source: §6.2 · p. 10

7 Conclusion

CLaRa combines compression pretraining with end-to-end retrieval and generation training to improve embedding-based RAG. Experiments across QA benchmarks show improvements in retrieval accuracy and answer quality alongside reduced input length and computation cost.

7 Conclusion

  • CLaRa addresses high-quality document compression for RAG systems that use document embeddings for question answering.Its pretraining constructs simple and compositional QA pairs plus paraphrased documents to retain essential semantics.Source: §7 · p. 10
  • An end-to-end framework unifies document representations across reranking and generation, with the authors reporting improved retrieval accuracy and answer quality.The framework uses multiple prompting-based pretraining objectives to create diverse supervision signals.Source: §7 · p. 10
  • Experiments on multiple QA benchmarks report that embedding-based compression reduces input length and computation cost while supporting a more unified RAG paradigm.The conclusion characterizes the resulting paradigm as more semantically coherent.Source: §7 · p. 10

Impact Statement

The impact statement describes potential efficiency benefits and acknowledges corpus-coverage and ethical considerations. The accompanying material also reports pretraining-data quality and outlines gradient, selection, and evaluation procedures.

Impact Statement

  • Compact, semantically dense representations may improve efficiency and scalability in resource-constrained information-access settings.The impact statement names scientific search, education, and knowledge-intensive assistants as potential beneficiaries through reduced latency and resource use while maintaining answer quality.Source: Impact Statement · p. 11
  • Compressors pretrained primarily on Wikipedia may reflect public-corpus coverage biases; more diverse, domain-specific data may help improve robustness across domains and modalities.The authors present this as a limitation of the current study and a direction for future work.Source: Impact Statement · p. 11
  • The authors identify no immediate negative ethical risks unique to CLaRa beyond common retrieval-model risks, while flagging transparency, controllability, and failure modes for future examination.The concerns are especially noted for reasoning-oriented or agentic RAG in complex decision-making settings.Source: Impact Statement · p. 11

A Gradients for Non-shared vs. Shared Representations in RAG

Optional: Cosine-similarity score backpropagation

B.1 Datasets

  • SCP pretraining uses 2M documents with 2M SimpleQA sets, 2M ComplexQA sets, and 2M paraphrased documents; compression instruction tuning uses 453k COCOM questions.For instruction tuning, Mistral-7B generates answer targets from queries and retrieved Wikipedia-2021 documents.Source: §B.1 · p. 17

B.3 Evaluation Metrics

  • Evaluation measures compressor answer coverage with ACC, reranker retrieval with Recall@k, and generated answers with Exact Match and F1.The experiments use Mistral-7B-Instruct as the default backbone and Phi-4-mini-instruct to assess generalization across model families.Source: §B.2 · pp. 18, 21, §B.3 · p. 21

C Pretraining Data Quality

  • Only 21 of 600 manually reviewed pretraining samples showed mild hallucinations, and the authors report that almost all covered source-document key information.The 600 samples comprised 200 examples of each output type and were assessed by one author.Source: C Pretraining Data Quality · p. 24

D Pretraining Data Scaling

Increasing pretraining data generally improves compressor performance across datasets and retrieval settings, though gains vary by model and diminish at larger corpus sizes.

D Pretraining Data Scaling

  • 37.94 to 39.02 average score under the Normal setting for Mistral-7B as pretraining data increases from 0.5M to 1M; performance remains stable at 2M.The passage reports this trend across four QA datasets after instruction tuning.Source: D Pretraining Data Scaling · p. 24
  • Over 6 points average gain in the Oracle setting when pretraining data increases from 0.5M to 1M.The authors interpret the result as evidence that more pretraining data helps preserve task-relevant information.Source: D Pretraining Data Scaling · p. 24
  • Phi4-mini shows relatively modest improvements, and performance gains diminish with extremely large pretraining sets.The authors suggest model capacity may constrain the benefits of scaling data.Source: D Pretraining Data Scaling · p. 24

E Training Curves

Across training, stronger compression is associated with higher loss and lower recall, while query-reasoner tuning and compressor-based initialization improve end-to-end QA results. Testing also shows some tolerance to changing retrieval size after training with top-5 documents.

E Training Curves

  • Higher compression ratios raise pretraining validation loss—sharply at CR=128 and 256 for Phi4-mini—and lower end-to-end recall while raising evaluation loss.During end-to-end training, recall increases and evaluation loss decreases; the authors hypothesize Phi4-mini’s sharper rise reflects limited capacity and semantic degradation at extreme compression.Source: E Training Curves · pp. 24–25, Figure 6 · p. 25, Figure 7 · p. 25

F.1 Effect of Freezing the Compressor and Query Reasoner

  • Under Oracle CR=32, Mistral-7B F1 rises from 52.54% to 70.91% when the query reasoner and generator are jointly fine-tuned.During compression learning and instruction tuning, full tuning yields less than 2.0% average gain over generator-only tuning across most Normal-setting datasets.Source: §F.1 · p. 25, Table 13 · p. 26, Table 12 · p. 26

F.2 Retrieval number generalization

  • Trained with top-5 retrieval, the model’s F1 generally rises rapidly then declines gradually as test-time k increases from 1 to 10, with a relatively small performance drop.The authors report good generalization to retrieval sizes different from the training setting.Source: §F.2 · p. 26, Figure 8 · p. 27

F.3 Effect of Query Reasoner Initialization

  • 66.84%→70.91% F1 on HotpotQA and 62.68%→66.32% F1 on 2Wiki: compressor-initialized query reasoners outperform random initialization across settings.The authors attribute this advantage to the pretrained compressor’s encoded query-document semantic relationships.Source: §F.3 · p. 27, Table 14 · p. 27

F.4 Efficiency Analysis

Compressed representations reduce answer-decoding time relative to full-text documents. Reconstruction results show semantic similarity remains high as compression rises, even as lexical overlap and entity preservation decline.

F.4 Efficiency Analysis

  • Decoding with compressed representations takes about 40% of full-text decoding time; compressing 20 retrieved documents into five representations can be done offline, outside real-time answering latency.Timing was measured on one NVIDIA H100 GPU with 80GB memory.Source: §F.4 · p. 27, Table 15 · p. 28

G.1 Information Preservation

  • BERTScore stays nearly 90% across compression ratios, while ROUGE-1, ROUGE-L, and entity preservation average over 50% but decline as compression increases.Reconstruction was evaluated on both unseen downstream-QA documents and seen pretraining-corpus documents.Source: §G.1 · p. 28, Table 16 · p. 29

G.2 Grounding Analysis

Grounding evaluation reports different patterns for compression-only and end-to-end settings. Paraphrase examples indicate that compressed representations retain semantic content while changing surface wording; the appendix also documents the prompts used in synthesis and answering.

G.2 Grounding Analysis

  • End-to-end Mistral-7B at CR=4 with top-20 positive documents scores 49.95 faithfulness and 60.59 factual correctness; compression-evaluation faithfulness declines as compression rises and model size falls.In compression evaluation, factual correctness is comparatively lower than faithfulness.Source: §G.2 · p. 30

H Paraphrase Case Study

  • Paraphrases generated from compressed representations preserve essential document meaning while substantially changing surface expressions, including sentence structure and word order.The authors connect this pattern to a pretraining objective that favors core semantic meaning over lexical-sequence memorization.Source: H Paraphrase Case Study · p. 30, Table 18 · p. 31

I Prompts

  • The appendix prompts cover QA generation, paraphrase production, information-completeness validation, missing-information completion, and answering from compressed-document representations.Source: I Prompts · p. 31
Loading 2511.18659v3…