MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval

1 Introduction

• Fine-grained chunks offer precise evidence but expand the candidate search space, whereas larger chunks improve efficiency but mix topics and make dense-similarity scores noisy.Prior approaches can add preprocessing costs, indices, scoring stages, or inference latency, motivating a way to improve coarse-chunk searchability instead.Source: §1 · p. 1
• MCompassRAG enriches coarse chunks with topic vectors in the retriever’s semantic space, making large chunks more precisely searchable without increasing the retrieval search space.The topic vectors expose the main semantic directions covered by each chunk, helping retrieval look beyond a single noisy chunk embedding.Source: §1 · pp. 1–2
• Across six complex retrieval benchmarks, MCompassRAG improves information efficiency by 8.24% on average over the strongest non-LLM baseline and runs at over 5× lower latency than strong LLM-based RAG baselines.Figure 1b illustrates the performance–latency trade-off; the quantitative comparisons are stated in the text.Source: §1 · p. 2, Figure 1 · p. 2
• A metadata selection and abstraction mechanism selects query-relevant topics from a corpus-level bank and summarizes them into a compact query-topic vector for chunk scoring.The framework is agnostic to the topic model so long as topics are embedded in the retriever’s semantic space.Source: §1 · p. 2
• MCompassRAG distills an LLM teacher into a lightweight retriever trained with an extreme multi-label objective, enabling topic-aware evidence selection without inference-time LLM calls.The student learns to identify multiple relevant chunks from metadata-enriched representations while preserving most teacher-guided retrieval quality.Source: §1 · p. 2

Retrieval Granularity and Structured Retrieval in RAG

• Fine-grained retrieval sharpens evidence but expands search and risks lost context, whereas coarse chunks preserve context yet mix topics; MCOMPASSRAG makes coarse chunks more searchable with topic metadata.Prior approaches include proposition-level retrieval (Chen et al., 2024b), LLM-guided and adaptive chunking (Zhao et al., 2025b,a), query-adaptive granularity (Zhang et al., 2026), and hierarchical retrieval (Sarthi et al., 2024). Other work enriches evidence or promotes diversity and coverage (Khan et al., 2026); these approaches may require finer indexing, adaptive selection, hierarchical structures, extra scoring, or LLM filtering.Source: §2 · p. 2

Semantic Guidance and Efficient Retrieval

• Query augmentation, decomposition, and iterative retrieval guide evidence selection, while MCOMPASSRAG uses corpus-derived topic metadata before retrieval.HyDE (Gao et al., 2023), query expansion (Wang et al., 2023; Zhou et al., 2024), and decomposition-based retrieval (Zheng et al., 2024) align queries with evidence; adaptive retrieval can add computation through repeated retrieval, reranking, or sufficiency checking. Unlike these or generation-side context compression (Lin et al., 2025), MCOMPASSRAG guides retrieval without inference-time LLM calls and remains compatible with query expansion, iterative retrieval, reranking, and context compression.Source: §2 · p. 3

3.4 Inference

• Inference makes no LLM calls: cached chunk indices are scored after query embedding, metadata selection, and abstraction, with topic extraction and chunk encoding performed offline.The framework can use any topic model whose topics are represented in the retriever embedding space and that provides chunk-level topic distributions.Source: §3 · p. 4, §3.4 · pp. 5–6

3.1 Topic Metadata and Metadata Bank

• Chunk topic distributions are cached in a metadata bank, with centroids represented in retriever embedding space; longer chunks support reliable offline topic estimates.The bank supplies corpus-level guidance because a short query’s own topic distribution may be unreliable.Source: §3.1 · p. 4

3.2 Metadata Selection and Representation

• MCompassRAG selects top-L bank entries, abstracts them to suppress noisy or redundant signals, and combines top-M topic summaries with query and chunk embeddings for MLP relevance scoring.Retrieval is formulated as extreme multi-label classification, with each chunk treated as a candidate label.Source: §3.2 · pp. 4–5

3.3 Training with LLM-Teacher Distillation

• GPT-4o generates base and expanded queries and labels positive and negative candidates; the teacher sees expanded queries while the student sees base queries and learns with BCE and distillation.The student encoder, topic centroids, and cached chunk distributions remain fixed; only the selector, abstraction module, and MLP are trained.Source: §3.3 · p. 5

4.1 Experimental Setup

• Experiments cover seven benchmarks, with six used for retrieval evaluation and LongBenchV2 reserved for downstream evaluation; retrieval quality uses Recall, Precision, and IE over three runs.MCOMPASSRAG is trained separately per benchmark, with synthetic data when labels are insufficient; encoders and cached topic representations remain fixed while the selector, abstraction module, and classifier are trained.Source: §4.1 · p. 6

4.2 Comparison with Retrieval Baselines

• 47.97 vs. 37.47 IE on DRBench shows MCOMPASSRAG outperforming SAKI-RAG, the strongest non-LLM baseline, while leading across all six benchmarks and retrieval metrics.It approaches the LLM + 10 Topics oracle without inference-time LLM calls: the IE gaps are under 1 point on SCI-DOCS and SQuAD and within 2–3 points on the remaining benchmarks.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 Downstream Performance and Efficiency

• 4,126 tokens per query and 174 ms end-to-end latency make MCOMPASSRAG substantially cheaper than SAKI-RAG and REFRAG, while maintaining competitive generation quality.The cited baselines use 5,584 tokens and 925 ms for SAKI-RAG, and 7,800 tokens and 720 ms for REFRAG; MCOMPASSRAG uses topic-guided retrieval without post-retrieval LLM filtering.Source: §4.3 · p. 7, Table 2 · p. 8
• Training MCOMPASSRAG is a one-time cost rather than inference overhead, and a retriever trained on MS Marco can generalize across datasets when switching corpora.Its generation-quality gap against the strongest efficient baselines is largely attributed to their inference-time LLM reranking or context selection; MCOMPASSRAG avoids that filtering through topic-guided retrieval.Source: §4.3 · p. 7
• MCOMPASSRAG’s fixed retrieval budget limits access to evidence compared with long-context methods, but topic-guided coarse retrieval recovers most of their evidence quality at a fraction of the cost.It uses over 10× fewer tokens than PageIndex and the LLM baseline while delivering generation scores within a reasonable margin.Source: §4.3 · p. 7

Effect of Abstraction and Selection Policy

• Removing either selection or abstraction lowers IE, and removing both yields the largest drop; their complementary roles make the full MCOMPASSRAG pipeline best across benchmarks.Selection identifies query-relevant metadata, while abstraction denoises and compresses selected topic distributions into a query-side signal.Source: §5 · p. 8, Table 3 · p. 9

Training Data Generalizability

• Trained on MSMarco or CLaRa (He et al., 2025) without target-benchmark data, MCOMPASSRAG outperforms all non-LLM Table 1 baselines on every benchmark.The performance gap relative to in-domain training is modest in most settings, enabling deployment in new domains without additional annotation.Source: §5 · p. 8, Table 3 · p. 9

Sensitivity to the Embedding Model

• QWEN3-EMBEDDING-8B performs best, while 4B remains close at lower computational cost; projecting 0.6B consistently outperforms its unprojected counterpart.The projected variant adds a linear layer to align the smaller encoder's outputs with the topic-metadata embedding space; ALL-MINILM-L6-V2 remains competitive with several Table 1 baselines.Source: Table 4 · p. 9, §5 · p. 9

Effect of the Number of Metadata Topic

• IE improves as selected topics increase to an intermediate range, typically around 12–15, then decreases as excess topics add noise; the student trails the teacher modestly near the optimum.The pattern holds across variants with and without selection and abstraction, indicating that high-topic-count degradation stems from added topic noise.Source: §5 · p. 8, Figure 3 · p. 10

Topic Model

• For the topic-model ablation, CEMTM learns topic distributions from contextualized vision-language embeddings; when centroids are unavailable, they are approximated by averaging top-topic-word embeddings.Table 5 reports IE, Precision, and Recall averaged over retrieval cutoffs k=1, 3, and 5; the supplied passages do not state comparative results.Source: Table 5 · p. 10, §5 · p. 10

• MCOMPASSRAG enriches coarse chunk representations with topic-level signals and distills an LLM teacher into a lightweight retriever for retrieval without inference-time LLM calls.Source: §6 · p. 10
• Information efficiency improves by 8.24% on average over the strongest non-LLM baseline, while latency is over 5× lower than strong LLM-based baselines.Source: §6 · pp. 10–11
• Ablations show that both the metadata selection policy and abstraction module are necessary, and that distillation generalizes without in-domain training data.Source: §6 · p. 11
• Future work could jointly optimize the topic model and retriever, develop approximate selection for very large corpora, and integrate MCOMPASSRAG into iterative deep research agents.Joint optimization could better align topic representations and further close the student–teacher gap; efficiency gains could compound across an agent’s multiple retrieval rounds.Source: §6 · p. 11

Limitations

• Poorly trained or misaligned topic models can make retrieval metadata uninformative, a particular concern in low-resource or specialized domains.Source: Limitations · p. 11
• MCompassRAG requires tuning K, L, M, and k, whose interactions are non-trivial; performance sensitivity to topic count requires validation.Source: Limitations · p. 11
• Aggregating topic centroids into weighted-sum vectors loses individual topic structure and grows noisier as more topics are included.The paper proposes exploring sparse or cross-attention topic integration to better preserve per-topic structure.Source: Limitations · p. 11

A Prompts Used for Training

• Prompt A.1 generates queries using the previous, target, and next chunks as input context.Source: A Prompts Used for Training · p. 15
• Prompt A.2 has the LLM teacher label a query–chunk pair relevant only when the chunk provides direct or supporting answer evidence, not vague topical similarity.Source: A Prompts Used for Training · p. 15
• Prompt A.1 first asks for a natural base question requiring target-chunk information without copying or revealing the answer.Source: A Prompts Used for Training · p. 15
• Prompt A.1 expands the base query with helpful neighboring-chunk context to clarify the information need without answer hints.Source: A Prompts Used for Training · p. 15

B.1 Benchmark Dataset Details

• Evaluation covers seven benchmarks across scientific, legal, open-domain multi-hop, reading-comprehension, enterprise deep-research, and long-context tasks; Table 6 summarizes dataset statistics.The table defines average document length in characters, evaluation-query counts, and whether each benchmark requires cross-document reasoning.Source: §B.1 · p. 15, Table 6 · p. 16
• LegalBench-RAG evaluates legal RAG retrieval through precise, minimal snippet retrieval, with 6,858 expert-annotated query-answer pairs over a corpus exceeding 79 million characters; Dragonball covers finance, legal, and medical scenarios in Chinese and English.Dragonball contains 6,711 questions and evaluates generated responses using the keypoint-based metrics Completeness, Hallucination, and Irrelevance.Source: §B.1 · p. 15
• HotpotQA requires reasoning over multiple supporting documents and provides sentence-level supporting-fact annotations for chunk-level retrieval evaluation; SQuAD (Rajpurkar et al., 2016) offers a complementary single-passage retrieval axis and contains 107,785 question-answer pairs on 536 Wikipedia articles.Source: §B.1 · pp. 15–16
• DRBench (Abaskohi et al., 2026) evaluates enterprise deep research across 100 tasks and 1,093 sub-questions, while LongBenchV2 (Bai et al., 2025) tests long-context generation without chunk-level retrieval labels.DRBench searches private company knowledge bases and the public web; LongBenchV2 contexts range from 8k to 2M words and cover six task categories.Source: §B.1 · p. 16

B.2 Baseline Method Details

• The eleven baselines include DenseXRetrieval (Chen et al., 2024b), which indexes distilled atomic propositions; Meta-Chunking (PPL and MSP) (Zhao et al., 2025b), which uses LLM-guided boundaries; and RAPTOR (Sarthi et al., 2024), which retrieves across a summary tree.The section describes these methods as alternatives targeting parts of the retrieval or generation pipeline.Source: §B.2 · p. 16
• SAKI-RAG (Tao et al., 2025) links sentence attention and expands or filters candidates; ReflectiveRAG assesses evidence sufficiency and redundancy; DF-RAG (Khan et al., 2026) dynamically selects diverse chunks.DF-RAG builds on Maximal Marginal Relevance and adapts diversity per query without additional fine-tuning.Source: §B.2 · p. 17
• REFRAG (Lin et al., 2025) compresses retrieved chunks and selectively expands them for generation, while PageIndex uses LLM-guided active search through a hierarchical document tree.PageIndex retrieves inline during model reasoning, enabling streaming without a blocking retrieval gate before the first token.Source: §B.2 · p. 17
• A-RAG (Du et al., 2026) lets the model adapt retrieval strategy and decide when evidence is sufficient; Chroma Context-1 iteratively searches and edits context as an agentic search model.Chroma Context-1 is described as a 20B-parameter model used alongside a frontier reasoning model.Source: §B.2 · p. 17

Training details

• Each dataset uses 2,000 training chunks and 10 synthetic queries per chunk, yielding 20,000 query–chunk pairs before negative sampling.DRBench and LongBenchV2 use EDR-200 and LongBenchV1, respectively, because they lack suitable retrieval training labels.Source: C Training and Implementation Details · p. 17
• MCOMPASSRAG is trained separately per benchmark, using 10% of available data for synthetic training when a benchmark lacks a sufficiently large training set.The metadata selector, abstraction module, and MLP relevance classifier are trained while the student encoder, topic centroids, and cached chunk-topic distributions remain fixed.Source: C Training and Implementation Details · pp. 17–18
• Default training uses AdamW at 2 × 10^-5 learning rate for 3 epochs, with batch size 16, weight decay 0.01, dropout 0.1, τ = 1.0, and α = 0.5.Generation uses temperature τ = 0.7 and top-p = 0.9; teacher relevance scoring uses τ = 0.0 for deterministic judgments.Source: C Training and Implementation Details · p. 18

Evaluation

• Retrieval Recall, Precision, and IE (IE@k = Precision@k×Recall@k) are measured at k ∈ {1, 3, 5} and averaged over three runs.Downstream generation uses benchmark-dependent Accuracy, F1, ROUGE-L, METEOR, or BERTScore.Source: C Training and Implementation Details · p. 18
• For downstream comparisons, ranked chunks are added until each method reaches a 1K-token budget, preserving native chunking; experiments use L = 50 and M = 10.This gives each method the same maximum amount of evidence despite different chunk sizes.Source: C Training and Implementation Details · p. 18

D Retrieval Performance at Different Cutoffs

• Across all methods, precision and recall increase monotonically as k rises from 1 to 5.Retrieving more documents provides greater coverage of relevant passages.Source: D Retrieval Performance at Different… · p. 18
• At k=1, 3, and 5, MCOMPASSRAG outperforms all nonoracle baselines and stays within a narrow margin of the LLM + 10 Topics oracle.The ordering is consistent across all six benchmarks and cutoffs.Source: D Retrieval Performance at Different… · p. 18, Table 8 · p. 19

E Effect of Topic Granularity of Topic Model

• Across all three benchmarks, retrieval performance peaks at K = 100 and degrades monotonically as topic-model granularity increases beyond that point.The study uses K = 100 across its experiments.Source: E Effect of Topic Granularity of Topic… · p. 19
• At K = 500–2000, increasingly fine-grained, sparse topics yield less coherent centroids and noisier chunk and query representations that are harder to match reliably.Weighted aggregation of the topic centroids produces the noisy representations.Source: E Effect of Topic Granularity of Topic… · p. 19
• The student–teacher performance gap is largest at K = 100, where the LLM teacher can use richer, more coherent topic structure than the student's compressed topic summary.At K ≥500, degraded topic quality harms both models and their performance converges.Source: E Effect of Topic Granularity of Topic… · p. 19
• Topic-signal quality, determined by K, matters alongside how many topic signals are passed at inference: overly fine-grained or coarse topic models degrade retrieval even with an optimal number.This finding complements the analysis in Section 5.Source: E Effect of Topic Granularity of Topic… · p. 19

F Topic Model Domain Adaptation: Training on Target Corpus

• The main experiments use WikiWeb2M-trained topic centroids and document-topic vectors; the in-domain variant trains the topic model on each target corpus while keeping the rest of the pipeline unchanged.This tests whether domain-specific terminology and evidence structures benefit from target-corpus topic modeling.Source: F Topic Model Domain Adaptation:… · p. 19

G Qualitative Analysis

• The qualitative analysis presents two retrieval-failure examples: definitional ambiguity in LegalBench-RAG and embedding-space analysis in Dragonball Finance.Both illustrate retrieval failures that dense similarity cannot handle.Source: G Qualitative Analysis · p. 20

G.1 LegalBench-RAG: definitional ambiguity in M&A agreements.

• For a query defining Superior Proposal, dense retrieval ranks the lexically similar Acquisition Proposal clause C2 (0.81) above the gold definition C3 (0.78).The clauses share vocabulary, but cosine similarity cannot distinguish a proposal definition from a clause evaluating whether a proposal is superior.Source: §G.1 · p. 20
• MCOMPASSRAG selects fiduciary-out and board-determination topic T-A and majority-threshold topic T-B, while suppressing solicitation-restriction and merger-consideration signals.The abstraction module pools T-A and T-B into a compact query-side topic vector aligned with C3’s topic representation.Source: §G.1 · pp. 20–21

F Topic Model Domain Adaptation: Training on Target Corpus

• Target-corpus topic models improve retrieval on Dragonball, LegalBench-RAG, and SCI-DOCS, with larger but moderate gains on LegalBench-RAG and Dragonball.The general-purpose topic model remains useful, so in-domain topic modeling is an optional enhancement when target-corpus data and training budget are available.Source: F Topic Model Domain Adaptation:… · pp. 19–20, Table 11 · p. 21

G.1 LegalBench-RAG: definitional ambiguity in M&A agreements.

• The student scores gold clause C3 at 0.89 versus 0.57 for C2, ranking C3 first without an inference-time LLM call.Teacher–student distillation trains the retriever on an LLM teacher’s judgment that C3 is relevant and C2 is not, using expanded query framing about board determination and financial-advisor consultation.Source: §G.1 · p. 21

G.2 Dragonball Finance: topic-guided separation in embedding space.

• The eight candidates span governance, remuneration, risk, finance, shareholder structure, internal audit, and two surface-overlap distractors.C7 shares “corporate citizenship” with the gold chunk, while C8 shares “long-term value creation.”Source: §G.2 · p. 21, Figure 5 · p. 23
• MCompassRAG disambiguates candidates through latent topic structure rather than lexical overlap.The query selector activates the CSR topic centroid, which aligns with the gold chunk’s topic distribution; hard negatives dominated by governance, finance, and risk drift away.Source: §G.2 · pp. 21–22
• Topic enrichment tightens query–gold alignment and pushes hard negatives, including surface-overlap distractors C7 and C8, away from the query.The query and gold chunk are already relatively proximate in raw embedding space, but several hard negatives remain nearby; after enrichment, C7 and C8 receive lower relevance scores because their topic distributions do not load on the CSR centroid.Source: §G.2 · pp. 21–22, Figure 5 · p. 23