PaperlayerView exact revision on arXiv

Source-linked AI summary

MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval

Amirhossein Abaskohi, Raymond Li, Gaetano Cimino, Peter West, Giuseppe Carenini, Issam H. Laradji

arXiv:2606.18508v1cs.CLcs.IR

TL;DR

RAG systems must balance precise evidence retrieval against the search cost and semantic noise created by different chunk granularities. MCompassRAG uses topic metadata and a distilled lightweight retriever to make coarse-grained chunks more searchable, improving information efficiency by 8.24% on average with over 5× lower latency than strong baselines.

  • Problem

    RAG systems face a granularity trade-off: fine-grained chunks improve precision but expand search, whereas larger chunks reduce candidates while mixing topics and producing noisy similarity scores.

  • Method

    MCompassRAG enriches coarse chunk representations with topic metadata and distills an LLM teacher into a lightweight retriever for topic-aware retrieval without inference-time LLM calls.

  • Results

    8.24% average information-efficiency improvement over the strongest non-LLM baseline was achieved across six retrieval benchmarks, with 5× lower latency than strong LLM-based baselines.

  • Takeaways & Limitations

    Topic metadata can make coarse-grained chunks more precisely searchable without increasing the retrieval search space.

  • Takeaways & Limitations

    MCOMPASSRAG depends on reliable topic modeling, which can be difficult in low-resource or specialized domains.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) systems depend critically on how documents are chunked and searched. Fine-grained chunks can improve retrieval precision but expand the search space, increasing latency and cost; larger chunks reduce the number of candidates but make dense similarity less reliable, as the representation for each chunk mixes multiple topics and introduces more semantic noise. This trade-off becomes especially limiting in deep research tasks, where retrieval must be both fast and precise across large, heterogeneous corpora. We introduce MCompassRAG, a metadata-guided retrieval framework that uses topic-level signals as a semantic compass for selecting relevant evidence. Instead of relying only on cosine similarity between queries and noisy chunk embeddings, MCompassRAG enriches chunk representations with topic metadata in the same embedding space and trains a lightweight retriever through LLM-teacher distillation. At inference time, MCompassRAG performs topic-aware retrieval without additional LLM calls, improving both efficiency and evidence quality. Across six complex retrieval benchmarks, MCompassRAG improves information efficiency (IE) by 8.24% on average with over 5 times lower latency than the strongest efficient RAG baselines. Code is available on https://github.com/AmirAbaskohi/MCompassRAG.

1 Introduction

MCompassRAG addresses the chunk-granularity dilemma by making coarse chunks more searchable with topic metadata, enabling efficient topic-aware retrieval without inference-time LLM calls. It uses metadata selection and LLM-teacher distillation to improve evidence selection without expanding the retrieval search space.

  • Deep research magnifies RAG’s chunking problem because systems search large corpora and issue many retrieval calls before answering.
  • Fine-grained chunks provide precise evidence but expand the candidate search space, whereas larger chunks improve efficiency while mixing topics and discourse roles.
  • Prior granularity and reranking methods can increase preprocessing, indexing, scoring, or inference latency, especially for repeated deep-research retrieval.
  • MCompassRAG enriches coarse chunks with topic metadata and embeds documents, chunks, and topics in a shared topic-aware semantic space.
  • At query time, a compact metadata-derived topic representation guides scoring of enriched chunks, allowing retrieval beyond a single noisy chunk embedding.
  • MCompassRAG selects query-relevant metadata and distills an LLM teacher into a lightweight extreme multi-label retriever, avoiding inference-time LLM calls.

2 Related Work

Related work frames RAG retrieval as a trade-off between fine-grained precision and coarse-grained context, while semantic-guidance methods improve query–evidence alignment through augmentation, decomposition, and iterative refinement.

  • Retrieval Granularity and Structured Retrieval in RAG: Fine-grained retrieval improves evidence precision but expands the search space and can lose context, whereas coarse-grained retrieval preserves context but produces noisier dense similarities.Coarse chunks may mix topics and irrelevant content, reducing representation reliability.
  • Semantic Guidance and Efficient Retrieval: Query-centered methods improve retrieval by generating hypothetical answers, adding related terms, or decomposing complex questions into simpler retrieval steps.Examples include HyDE, query expansion, and decomposition-based retrieval.
  • Semantic Guidance and Efficient Retrieval: Adaptive and iterative retrieval methods refine evidence through repeated retrieval, reranking, or sufficiency checking.These methods focus on improving the evidence set after or across retrieval iterations.

3 MCOMPASSRAG

MCOMPASSRAG makes coarse-grained chunks more searchable by augmenting query and chunk representations with topic metadata. It uses LLM-teacher distillation during training, while inference relies on cached metadata, abstraction, and lightweight student scoring without LLM calls.

  • Framework overview: MCOMPASSRAG augments query and chunk embeddings with topic-level metadata to identify relevant semantic directions within large chunks.This reduces reliance on cosine similarity between queries and noisy chunk embeddings.
  • Training: Training uses expanded-query relevance supervision from an LLM teacher, while the student receives only base queries and learns with binary cross-entropy and distillation losses.The information asymmetry encourages the student to recover missing context through metadata selection and abstraction.
  • Metadata bank: Chunk-topic distributions are cached in a corpus-level metadata bank, whose topic centroids represent semantic regions in the retriever embedding space.The bank provides query-side guidance without directly trusting unreliable topic distributions from short queries.
  • Inference pipeline: At inference, a lightweight selector ranks metadata entries, an abstraction module combines topic signals, and enriched query–chunk representations are scored by an MLP retriever.The selector uses query and metadata-entry embeddings, while abstraction suppresses redundant or noisy metadata and constructs topic summaries.
  • Inference efficiency: Inference requires no LLM calls because chunk embeddings, topic distributions, and topic-enriched representations are precomputed offline before metadata selection and student scoring.Online processing consists of lightweight metadata selection, abstraction, scoring, and top-k retrieval.

4 Experiments and Results

MCompassRAG is evaluated against diverse retrieval and RAG baselines, with topic-aware retrieval improving performance across benchmarks while avoiding inference-time LLM calls. It also delivers a strong efficiency–quality trade-off, using fewer tokens and lower latency than competing efficient methods.

  • Retrieval performance: MCOMPASSRAG with 10 topic signals consistently outperforms all baselines across every benchmark and retrieval metric.The evaluation covers six benchmarks, with the strongest gains on harder multi-hop tasks.
  • Retrieval performance: 47.97 IE on DRBench exceeds SAKI-RAG’s 37.47, while MCOMPASSRAG also leads all three metrics on LegalBench-RAG.SCI-DOCS and SQuAD are comparatively easier, but MCOMPASSRAG still remains competitive there.
  • Retrieval performance: 94.13 vs. 94.67 IE on SCI-DOCS and 93.80 vs. 94.10 on SQuAD show gaps under one point against full LLM retrieval without inference-time LLM calls.The topic-free versus LLM + 10 Topics comparison indicates that topic metadata provides guidance beyond raw chunk embeddings.
  • Downstream quality and efficiency: 4,126 tokens per query and 174 ms end-to-end latency make MCOMPASSRAG substantially cheaper than SAKI-RAG at 5,584 tok and 925 ms and REFRAG at 7,800 tok and 720 ms.Its favorable performance–latency trade-off is attributed to topic-guided retrieval without post-retrieval LLM filtering.
  • Downstream quality and efficiency: MCOMPASSRAG recovers much of the quality of expensive long-context methods through topic-guided coarse retrieval under a fixed retrieval budget.Long-context methods can exploit all available document evidence, whereas MCOMPASSRAG avoids that cost by retrieving a bounded amount of evidence.

5 Ablations

Ablations show that both metadata selection and abstraction are important for retrieval quality, with topic counts performing best in an intermediate range. Embedding-backbone experiments further show that stronger encoders generally improve quality, while projection helps smaller encoders align with topic metadata.

  • Abstraction and Selection Policy: Removing either the abstraction module or selection policy lowers IE, with the largest drop occurring when both are removed.The selection policy identifies query-relevant metadata, while abstraction denoises and compresses selected topic distributions.
  • Number of Metadata Topics: IE improves as selected topics increase to typically 12–15 topics, then declines as additional topics introduce noise.This trend appears across DRBench and Dragonball and across four ablation variants.
  • Embedding Backbone: QWEN3-EMBEDDING-8B performs best, while QWEN3-EMBEDDING-4B remains close with lower computational cost.The comparison covers Dragonball, LegalBench-RAG, and SCI-DOCS against multiple encoder alternatives.
  • Embedding Backbone: The projected QWEN3-EMBEDDING-0.6B consistently outperforms its unprojected counterpart, indicating benefits from embedding-space alignment.A lightweight linear layer maps smaller-encoder outputs into the topic-metadata embedding space.
  • Embedding Backbone: MCOMPASSRAG remains competitive with several baselines even when using the much smaller ALL-MINILM-L6-V2 encoder.This supports the value of topic metadata beyond encoder scale alone.

6 Conclusion and Future Works

MCOMPASSRAG combines topic-level metadata with LLM-teacher distillation to enable topic-aware retrieval without inference-time LLM calls. Across six benchmarks, it improves information efficiency while substantially reducing latency, with ablations supporting its design choices and future work targeting end-to-end optimization and approximate selection.

  • Contributions: MCOMPASSRAG enriches coarse chunk representations with topic-level signals and trains a lightweight student retriever through LLM-teacher distillation.This enables topic-aware retrieval without inference-time LLM calls.
  • Results: 8.24% average information efficiency improvement over the strongest non-LLM baseline was achieved across six retrieval benchmarks.The reported improvement is averaged across six retrieval benchmarks.
  • Results: Over 5× lower latency compared to strong LLM-based baselines was achieved across the benchmark evaluation.This efficiency result accompanies the information-efficiency improvement.
  • Analysis: Ablations found that both the metadata selection policy and abstraction module are necessary, while the distillation pipeline generalizes without in-domain training data.These findings support the framework’s component choices and training approach.
  • Future Works: Future work includes jointly optimizing the topic model and retriever end-to-end and developing approximate selection strategies.End-to-end optimization could better align topic representations, while approximate selection could improve efficiency.

Limitations · A Prompts Used for Training · B Benchmark and Baseline Details

MCompassRAG depends on reliable topic modeling and careful hyperparameter choices, while its training appendix specifies query generation and teacher-based relevance labeling prompts. The supplied passages do not describe benchmark or baseline details.

  • Limitations: MCompassRAG’s topic-guided retrieval depends directly on the quality of its underlying topic model.Poorly trained or misaligned topic representations can produce uninformative metadata signals, especially in low-resource or specialized domains.
  • Limitations: The framework introduces several hyperparameters, including the topic count K and selected metadata entries.The supplied limitation passage identifies these hyperparameters but is truncated before describing their full implications.
  • A Prompts Used for Training: Prompt A.1 generates base and expanded queries from training chunks.It uses three consecutive chunks: the previous, target, and next chunks.
  • A Prompts Used for Training: Prompt A.2 assigns relevance labels to query–chunk pairs during LLM-teacher distillation.The teacher receives a question and candidate knowledge chunk and outputs a binary label.
  • A Prompts Used for Training: The base query must require information from the target chunk without copying or revealing its answer.The prompt asks for a natural user question that does not directly copy the answer or reveal it.
  • A Prompts Used for Training: The expanded query adds useful background context from neighboring chunks without answer hints.The expansion should clarify the information need using context from the previous and next chunks.
  • A Prompts Used for Training: A candidate chunk is relevant only when it provides direct or supporting evidence needed to answer the question.Vague topical similarity alone is insufficient; the required output is 1 for relevant or 0 for not relevant.

B.1 Benchmark Dataset Details · B.2 Baseline Method Details

The evaluation spans seven benchmarks covering scientific, legal, open-domain multi-hop, reading-comprehension, enterprise deep-research, and long-context tasks. It compares MCOMPASSRAG with eleven baselines targeting retrieval or generation, including proposition-based, sentence-linking, inline, and agentic retrieval approaches.

  • B.1 Benchmark Dataset Details: Seven benchmarks cover scientific, legal, open-domain multi-hop, reading-comprehension, enterprise deep-research, and long-context tasks.Table 6 reports benchmark statistics, including average document length, evaluation-query counts, and whether cross-document reasoning is required.
  • B.1 Benchmark Dataset Details: SCI-DOCS evaluates scientific document embeddings across seven document-level tasks, including citation prediction, document classification, and recommendation.The suite contains tens of thousands of anonymized user signals of document relatedness and was introduced with SPECTER to address limitations of smaller, narrower evaluations.
  • B.1 Benchmark Dataset Details: HotpotQA contains 113k Wikipedia-based question-answer pairs requiring multi-document reasoning with sentence-level supporting-fact annotations.Its factoid comparison questions test extracting and comparing relevant facts across entities, making it suitable for chunk-level retrieval evaluation.
  • B.1 Benchmark Dataset Details: SQuAD contains 107,785 question-answer pairs across 536 Wikipedia articles, with every answer given as a text span from the corresponding passage.Its largely single-passage questions provide a complementary single-hop retrieval axis to HotpotQA’s multi-hop setting.
  • B.2 Baseline Method Details: The study compares MCOMPASSRAG against eleven baselines and identifies whether each primarily targets retrieval or generation.The baseline descriptions cover each method’s core methodology and pipeline focus.
  • B.2 Baseline Method Details: DenseXRetrieval uses atomic, self-contained propositions as dense-retrieval units, generated by a fine-tuned Propositionizer through two-step distillation.The propositions are intended to encapsulate distinct factoids while reducing passage-level granularity.
  • B.2 Baseline Method Details: SAKI-RAG combines SentenceAttnLinker’s inter-sentence attention modeling with a Dual-Axis Retriever that filters candidates by semantic similarity and contextual relevance.The framework addresses text fragmentation in long-document RAG through a semantically enriched knowledge repository and two-dimensional candidate expansion and filtering.
  • B.2 Baseline Method Details: ReflectiveRAG retrieves inline during reasoning, while A-RAG exposes hierarchical retrieval interfaces that let the language model adapt its retrieval strategy.ReflectiveRAG can begin streaming without a blocking retrieval gate before the first token; A-RAG differs from single-shot concatenation and predefined step-by-step workflows.

C Training and Implementation Details

MCompassRAG is trained with benchmark-specific or transferred data and fixed default optimization settings. Evaluation combines retrieval metrics at multiple depths with a fixed 1K-token evidence budget to ensure fair downstream comparison across chunk granularities.

  • Training: Default training uses AdamW with learning rate 2 × 10−5, batch size 16, weight decay 0.01, dropout 0.1, and 3 epochs.Distillation uses τ = 1.0 and α = 0.5; generation uses τ = 0.7 and top-p = 0.9.
  • Evaluation: Recall, Precision, and Information Efficiency are reported at k ∈ {1, 3, 5}, with IE@k = Precision@k×Recall@k, averaged over three runs.The protocol addresses unfairness caused by different chunk granularities when comparing a fixed number of retrieved chunks.
  • Evaluation: A fixed 1K-token budget adds retrieved chunks in ranked order, giving every method the same maximum evidence while preserving native chunking strategies.Experiments use L = 50 and M = 10.

D Retrieval Performance at Different Cutoffs · E Effect of Topic Granularity of Topic Model

MCOMPASSRAG maintains its retrieval advantage across cutoffs while remaining close to the LLM + 10 Topics oracle. Topic-model performance peaks at K = 100, indicating that moderate granularity best balances topic coherence and coverage.

  • D Retrieval Performance at Different Cutoffs: Retrieval performance is evaluated at k=1, k=3, and k=5 across all six benchmarks using IE, precision, and recall.Table 8 specifically reports IE @3, Precision @3, and Recall @3, with MCOMPASSRAG rows shaded.
  • D Retrieval Performance at Different Cutoffs: Precision and recall increase monotonically as k grows because retrieving more documents covers more relevant passages.This trend holds for all evaluated methods.
  • D Retrieval Performance at Different Cutoffs: MCOMPASSRAG outperforms every nonoracle baseline at every cutoff while staying within a narrow margin of the LLM + 10 Topics oracle.The relative ordering of methods remains consistent across k=1, k=3, and k=5.
  • E Effect of Topic Granularity of Topic Model: Table 10 examines retrieval performance across different numbers of topics K in the underlying topic model.The analysis covers all three benchmarks.
  • E Effect of Topic Granularity of Topic Model: Performance peaks at K = 100 and then degrades monotonically as K increases beyond this point.This pattern is consistent across all three benchmarks.
  • E Effect of Topic Granularity of Topic Model: At very high granularities, K = 500–2000, overly fine-grained topic models degrade retrieval quality.The reported results support using K = 100 across the experiments.
  • E Effect of Topic Granularity of Topic Model: Topic-signal quality, determined by K, is equally important to the number of topic signals passed at inference time.Even an optimal number of passed topics cannot prevent retrieval degradation from overly fine-grained or coarse topic models.

F Topic Model Domain Adaptation: Training on Target Corpus · G Qualitative Analysis · G.1 LegalBench-RAG: definitional ambiguity in M&A agreements.

MCompassRAG benefits from optional target-corpus topic modeling while remaining effective with a general WikiWeb2M-trained model. Qualitative analysis shows that topic metadata resolves LegalBench-RAG’s definitional ambiguity by promoting the correct M&A definition without an inference-time LLM call.

  • F Topic Model Domain Adaptation: Training on Target Corpus: The main experiments use WikiWeb2M-trained topic centroids and document-topic vectors as a general-purpose metadata source.This configuration tests whether MCompassRAG can rely on a broadly trained topic model.
  • F Topic Model Domain Adaptation: Training on Target Corpus: An in-domain variant trains the topic model directly on each benchmark’s target corpus while leaving the rest of the MCompassRAG pipeline unchanged.The variant addresses domain-specific terminology and evidence structures that a general corpus may not fully capture.
  • F Topic Model Domain Adaptation: Training on Target Corpus: Target-corpus topic modeling improves performance on Dragonball, LegalBench-RAG, and SCI-DOCS, with larger gains on LegalBench-RAG and Dragonball.The larger gains occur where domain-specific terminology, entities, and narrative structure are especially important.
  • F Topic Model Domain Adaptation: Training on Target Corpus: The moderate rather than dramatic gains indicate that MCompassRAG does not require topic-model retraining for every new corpus.A general-purpose topic model can provide useful metadata guidance, while in-domain modeling remains an optional enhancement when sufficient target-corpus data is available.
  • G Qualitative Analysis: The qualitative analysis presents two retrieval-failure examples: definitional ambiguity in LegalBench-RAG and embedding-space analysis in Dragonball Finance.Both examples illustrate failures that dense similarity alone cannot handle.
  • G.1 LegalBench-RAG: definitional ambiguity in M&A agreements.: Dense retrieval ranks the wrong Acquisition Proposal definition first, assigning C2 cosine similarity 0.81 versus the gold chunk C3 at 0.78.C2 and C3 share substantial surface vocabulary, creating the ambiguity in the M&A agreement’s §6.03 region.
  • G.1 LegalBench-RAG: definitional ambiguity in M&A agreements.: MCompassRAG activates topic signals for fiduciary-out, board-determination, and majority-threshold frames while suppressing signals orthogonal to the query’s information need.The selector’s topic-aware behavior distinguishes the gold definition from adjacent no-shop and acquisition-related clauses.
  • G.1 LegalBench-RAG: definitional ambiguity in M&A agreements.: The abstraction module aligns the query-side topic vector with C3, and the MLP assigns C3 relevance 0.89 versus C2 relevance 0.57, promoting C3 to rank 1 without an inference-time LLM call.The disambiguation was learned through the teacher–student asymmetry described in Section 3.3.

G.2 Dragonball Finance: topic-guided separation in embedding space.

Figure 5 shows that topic enrichment separates the CSR query and gold chunk from semantically overlapping governance-report candidates. MCompassRAG achieves this by aligning query and chunk topic signals, enabling latent-topic rather than lexical-overlap disambiguation.

  • Candidate separation: The eight candidates span corporate governance topics including board composition, executive remuneration, risk management, financial highlights, shareholder structure, and internal audit.C7 and C8 are surface-overlap distractors sharing “corporate citizenship” and “long-term value creation” with the gold chunk.
  • Candidate separation: In raw embedding space, the query and gold CSR chunk are proximate, but hard negatives remain nearby because coarse governance language creates broad semantic overlap.The example query asks about Sparkling Clean Housekeeping Services’ sustainability and social responsibility efforts in 2019.
  • Topic enrichment: After topic enrichment, the query–gold alignment tightens substantially as both activate the CSR topic signal.The metadata selector activates the CSR topic centroid for the query, while the gold chunk’s topic distribution loads on the same signal.
  • Topic enrichment: Topic-misaligned candidates receive lower MLP-classifier relevance scores, confirming that MCompassRAG disambiguates through latent topic structure rather than lexical overlap.This mechanism suppresses candidates that do not align with the activated topic centroid.
Loading 2606.18508v1…