Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding

1 Introduction

• MiA-RAG-14B achieves the best average rank across five long-context benchmarks, surpassing vanilla 72B RAG.The evaluation spans English and Chinese, varied domains and task formats, and vanilla RAG and GraphRAG configurations.Source: Figure 1 · p. 1, §1 · p. 2
• Unlike current RAG systems that rely mainly on local evidence-level signals, MiA-RAG supplies a global semantic frame for long-context understanding.The introduction identifies personalized collections, including long-context QA, code generation, and project assistants, as promising settings.Source: §1 · pp. 1–2
• MiA-RAG approximates a document’s global impression with hierarchical summaries and conditions both retrieval and generation on this mindscape.The shared global memory helps the retriever form context-enriched query representations and the generator interpret retrieved evidence in its broader context.Source: §1 · p. 2
• Analysis finds that the mindscape aligns query representations with global semantics and guides attention during integrative reasoning.The authors interpret this as evidence of global-context internalization rather than surface-level pattern matching.Source: §1 · p. 2
• The authors formulate mindscape-aware thinking as a computational capability for LLMs and report experiments supporting its integration into LLM-based systems.The proposed framing draws on human context-dependent global semantic representations.Source: §1 · pp. 1–2

Context-Aware Embeddings

• Context-aware embedding methods encode chunks using extended-context LLMs, while discourse-graph approaches augment them with neighboring information (Xu et al., 2024).The cited work also notes that these methods do not directly teach models how to fuse the added information.Source: §2 · pp. 2–3
• Wu et al. (2025) enrich chunk representations with local context, whereas MiA-RAG injects global semantics into queries without modifying chunk embeddings.The paper characterizes Wu et al. as achieving state-of-the-art retrieval with modest training data requirements.Source: §2 · p. 3

Long Context Compression

• Unlike compression approaches such as MemoRAG (Qian et al., 2025) and LeanRAG (Zhang et al., 2026), MiA-RAG uses the mindscape to condition both retrieval and generation.The authors describe this design as avoiding intermediate search steps or external graph construction.Source: §2 · p. 3

3.1 Preliminaries

• MiA-RAG conditions retrieval and generation on an explicit global semantic scaffold, situating local evidence within global context.The framework is presented as emulating how humans leverage global context for understanding, retrieval, and reasoning.Source: §3.1 · p. 3, §3 · p. 3

3.2 Hierarchical Mindscape Construction

• Hierarchical summarization produces a document-level mindscape that can be updated by regenerating affected paths from modified chunks to the root.Chunk summaries are concatenated in order and summarized into one global representation.Source: §3.2 · p. 3

3.3 MiA-Emb: Mindscape-Aware Retriever

• MiA-Emb is trained with a multi-task objective using a supervision dataset designed for mindscape-aware retrieval.The method develops the Mindscape-Aware Retriever from constructed supervision.Source: §3.3 · p. 4

3.3.1 Supervision Construction

• Existing NarrativeQA questions lack fine-grained evidence links, so MiA-RAG constructs silver-standard chunk- and node-level supervision automatically.The resulting dataset contains 27,117 questions, averaging 2.3 silver chunks and 2.9 nodes per question.Source: §3.3.1 · p. 4, Table 1 · p. 4
• Using only silver chunks—about 1/30 of the tokens—already exceeds full-context performance in the oracle experiment validating the annotations.Table 1 presents the silver-chunk annotation validation.Source: §3.3.1 · p. 4, Table 1 · p. 4

3.3.2 Model Optimization

• MiA-Emb incorporates the global summary and task-specific control tokens into each query representation, balancing query intent and global guidance with residual integration and an InfoNCE objective.The control tokens activate node- or chunk-retrieval modes.Source: §3.3.2 · p. 4

3.4.1 Training Data Construction

• Generator training combines NarrativeQA long-context QA examples with CLIPPER synthetic narrative-claim verification examples, using varied retrieved contexts to simulate inference noise.NarrativeQA contexts mix silver evidence and irrelevant text; CLIPPER uses MiA-Emb top-k retrieval, with k ∈ {3, 5, 10}.Source: §3.4.1 · p. 4

3.4.2 Model Optimization

• MiA-Gen is optimized on its constructed training dataset using autoregressive cross-entropy loss.Source: §3.4.2 · p. 4

Evaluated Models

• The evaluation fine-tunes Qwen3-Embedding-8B as MiA-Emb and Qwen2.5-14B-Instruct as MiA-Gen, comparing them with Qwen baselines and leading RAG methods.Comparators include HippoRAG v2, MemoRAG, and RAPTOR.Source: §4 · p. 4

Public Long Narrative Understanding Tasks

• Long-narrative evaluation includes held-out NarrativeQA books in-domain and EN.MC from ∞Bench, public Nocha, and DetectiveQA-En/Zh out-of-domain.Many benchmarks exceed common LLM input limits, such as 128K tokens.Source: §4 · p. 4

Experimental Setting

• Table 2 reports overall results on Long-story QA and Reasoning benchmarks at top-3, top-5, and top-10 retrieval depths.Its rows indicate summary incorporation and mindscape involvement; the final column averages metrics per row.Source: Table 2 · p. 5

5.1 Study I: Retrieval Results

• MiA-Emb consistently outperforms retrieval baselines, including the story-understanding model Sit-Emb, and transfers from NarrativeQA to out-of-domain DetectiveQA.Removing the summary substantially degrades retrieval performance.Source: §5.1 · p. 5, Table 3 · p. 5

5.2.1 End-Task Results

• +16.18% over vanilla-RAG-14B and +8.63% over vanilla-RAG-72B; MiA-RAG achieves the best results across benchmarks, languages, domains, and task formats.The complete pipeline evaluation covers five long-context benchmarks.Source: §5.2.1 · p. 5
• +6.95% for the 72B generator and +7.55% for the 14B generator when MiA-Emb replaces the vanilla retriever.These are improvements in average scores in the full pipeline.Source: §5.2.1 · p. 5

5.2.2 Ablation Study

• Removing summaries degrades generation, while removing either claim-verification or question-answering supervision also consistently reduces performance.The ablation scores are averaged over the 3/5/10-chunk settings.Source: §5.2.2 · p. 6, Table 4 · p. 6

5.3 Study III: MiA-GraphRAG for Global QA

• MiA-RAG achieves clear gains over baselines in GraphRAG question answering for global sense-making by retrieving relevant graph nodes.The evaluation focuses on holistic document understanding.Source: §5.3 · p. 6

5.4 Study IV: Impact of Model Scales

• MiA-Emb-0.6B surpasses the Vanilla 8B model in retrieval recall and downstream QA and reasoning, while MiA-Gen-14B surpasses the vanilla 72B model.MiA-Emb is evaluated across 0.6B–8B backbones, and MiA-Gen gains persist across 1.5B–72B models.Source: §5.4 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

5.5 Study V: Impact of Summary Quality

• MiA-RAG maintains stable performance across open-source and GPT-4o summarizers; NarrativeQA summary lengths of 240–1,243 tokens have only a weak correlation with Recall@10 (Pearson r = 0.23).The 14B summarizer achieves results comparable to GPT-4o; summary information functions as a global scaffold rather than a lossless store of fine-grained evidence.Source: §5.5 · p. 6, Table 5 · p. 6

6.1 The Role of Global Summaries

• Summary-only generation underperforms vanilla RAG and MiA-RAG, indicating summaries act as semantic scaffolds for retrieval and reasoning, not standalone evidence.Same-summary retrieval controls support this interpretation.Source: §6.1 · pp. 6–7

6.2 Geometric Properties of the MiA Embedding Space

• MiA-Emb-8B has a 37.1° average projection angle versus 43.5° for Qwen-Emb-8B, with smaller angles indicating closer query alignment to document semantic subspaces.The analysis embeds queries and chunks into a shared t-SNE space.Source: §6.2 · p. 7, Figure 5 · p. 7

6.3 Residual Stream and Attention Analysis of the MiA Embedding Model

• Middle layers show rising silver-chunk recall alongside increased global-summary attention, suggesting summary-derived cues enter the query representation as retrieval improves.The layer-wise analysis compares 100 queries for which both models achieve Recall@10 = 100%.Source: §6.3 · p. 7, Figure 6 · p. 7
• At layers corresponding to retrieval gains, MiA-Emb’s final embedding token attends to summary phrases containing answer-relevant entities, events, or locations.The analysis describes this as selective extraction and integration of semantically aligned global cues.Source: §6.3 · p. 8, Figure 7 · p. 8

6.4 Attention Pattern Analysis in the Generation Model

• MiA-Gen has high MCEA in middle and late layers, while replacing the valid summary with unrelated text sharply reduces its MCEA; the vanilla model is less sensitive.MCEA is an attention-level diagnostic of mindscape-conditioned evidence integration, not a raw score for direct cross-model comparison.Source: §6.4 · p. 8, Figure 8 · p. 8

• MiA-RAG approximates a document’s global impression with hierarchical summaries and conditions both retrieval and generation on this persistent global memory.Source: §7 · p. 8
• The framework achieves strong performance on evidence-based long-context understanding and global sense-making tasks.Source: §7 · p. 8
• Empirical analysis finds support for global semantic query understanding, selective retrieval, and integrative reasoning over dispersed evidence.Source: §7 · p. 8

Limitations

• A precomputed summary adds preprocessing overhead, experiments mainly cover narrative long-context tasks, and commercial-LLM-derived supervision may contain latent bias or hallucinated content.The cost is incurred once per document and can be amortized; incremental updates regenerate only affected paths, partly mitigating the overhead for evolving documents.Source: Limitations · p. 9

A Supports of Mindscape-Aware Capabilities in Broader Research Fields

• Mindscape-aware processing is presented as a computational approximation of gist-based cognition, with related support from schema, encoding-specificity, and neuroscience accounts.These accounts describe globally integrated semantic knowledge guiding interpretation and retrieval, including context reinstatement during recall.Source: A Supports of Mindscape-Aware… · pp. 11–12

B.1 Positive Evidence Construction

• Because long-context benchmarks lack query–evidence alignments, the paper constructs silver evidence for chunk- and node-level retrieval.For node retrieval, it extracts entities and descriptions into a document knowledge graph, then labels relevant nodes for each query.Source: §B.1 · p. 12

B.2 Negative Evidence Construction

• Retriever training contrasts supporting evidence with hard negatives from retrieved candidates and randomly sampled simple negatives.Hard negatives are semantically similar but outside the silver set; simple negatives are sampled from the document and exclude positives and hard negatives.Source: §B.2 · p. 12

B.3 Model Training

• MiA-Emb encodes the query with the mindscape and retrieval-task markers, then mixes query and task hidden states through a residual connection and jointly optimizes chunk- and node-retrieval losses.The InfoNCE objectives use positive silver evidence and negative candidates; δ balances local query focus against global context, while β balances the two retrieval losses.Source: §B.3 · p. 13

C.1 Dataset Statistics

• The evaluation datasets cover public long-narrative understanding, with a separate global sense-making setting built from LongBench summary-generation datasets.Table 6 summarizes query counts, average token lengths, and evaluation metrics.Source: §C.1 · p. 13, Table 6 · p. 13

C.2 Baselines

• Baselines include dense vanilla RAG, HippoRAG-v2 graph retrieval, and MemoRAG memory-enhanced retrieval.The vanilla pipeline uses the same prompt template and retrieval budget across methods; MemoRAG uses its released Mistral memory model and BGE-M3 retriever.Source: §C.2 · pp. 13–14

C.3 Implementation Details

• MiA-Emb applies LoRA to Qwen3-Embedding-8B, while MiA-Gen fully fine-tunes Qwen2.5-14B-Instruct; chunk sizes vary by dataset.Chunks are 1,200 tokens with 100-token overlap for NarrativeQA, DetectiveQA, and ∞Bench, and 200-token overlap for NoCha.Source: §C.3 · p. 14

C.4 Computational Cost

• 1.43× faster end-to-end, MiA-RAG with a summary and 3 chunks records 437.67 ms/query and accuracy 80.79, versus 624.93 ms/query and accuracy 77.29 for the 10-chunk vanilla baseline.Building a mindscape takes approximately 24.79 seconds per book and is a one-time preprocessing cost amortized across queries; summary concatenation increases ∞Bench retrieval latency from 10.48 to 100.23 ms/query.Source: §C.4 · p. 14, Table 8 · p. 14

D.1 Performance Across Various Embedding Models

• On DetectiveQA-ZH, MiA-Emb achieves the strongest retrieval results across configurations, outperforming general-purpose, locally contextualized, and supervised embedding baselines.The comparison includes GTE, NV-Embed-v2, Voyage, SitEmb, and SFT-Emb; the table reports Answer Recall@K.Source: §D.1 · pp. 14–15, Table 9 · p. 15

D.2 Retrieval Results on LoCoMo

• Without retraining, MiA-Emb improves over Qwen3-Emb-8B across all retrieval depths on the out-of-domain LoCoMo conversational-memory benchmark.Table 10 presents the out-of-domain retrieval results.Source: §D.2 · p. 15, Table 10 · p. 15

D.3 Results on Helmet

• On Helmet NarrativeQA, MiA-Emb improves EM and F1 across generator pairings, and the full MiA-RAG achieves the strongest results with shorter context lengths.Adding the summary benefits all RAG configurations, especially when retrieval quality is already high.Source: §D.3 · p. 15, Table 11 · p. 15

D.4 Study III: MiA-GraphRAG for Global QA

• On global sense-making questions from documents exceeding 100K tokens, MiA-Emb performs best across comprehensiveness, diversity, and empowerment comparisons.The 300 questions span English QM-Sum and GOV and Chinese VCSum; Table 12 reports pairwise comparisons under a shared graph-construction pipeline.Source: §D.4 · pp. 15–16, Table 12 · p. 16

D.5 Model Scale Analysis

• MiA-Emb outperforms vanilla and SFT-Emb across retriever scales and continues gaining as size increases, while vanilla embeddings show diminishing returns at larger scales.MiA-Gen also beats the vanilla generator under identical retrieved contexts, with a widening gap at larger sizes, especially on reasoning-heavy NoCha.Source: §D.5 · p. 16

D.6 On the Role of Residual Integration

• For Qwen3-Embedding-8B, appending summaries harms retrieval, so the residual connection preserves query semantics and prevents the summary from overwhelming them.MiA-Emb remains stable with or without the residual, having learned a finer-grained fusion of query semantics and summary information.Source: §D.6 · p. 16, Table 14 · p. 16

E Definition of MCEA Metric

• MCEA investigates whether the mindscape guides attention toward local evidence during generation.Under a valid mindscape, it provides an attention-level diagnostic of mindscape-conditioned evidence integration for Integrative Reasoning.Source: E Definition of MCEA Metric · pp. 16–17
• For each chunk, the alignment score is the product of z-score-normalized mindscape-to-chunk and query-to-chunk attention values.The normalization uses each quantity’s mean and standard deviation across chunks at that layer.Source: E Definition of MCEA Metric · pp. 16–17
• MCEA is the difference between mean alignment scores for relevant silver chunks and noise chunks at each layer.Higher MCEA indicates that chunks receiving stronger mindscape attention are also preferentially attended by the query, especially relevant evidence over noise.Source: E Definition of MCEA Metric · p. 17

F Prompt Templates for MiA-RAG

• The prompt templates cover hierarchical summarization, retriever supervision, sense-making, unified mindscape-and-query retrieval input, and generator instructions for multiple settings.The generator settings include mindscape-augmented QA, standard QA baselines without summaries, and global sense-making QA.Source: F Prompt Templates for MiA-RAG · p. 17
• The chunk-summary prompt asks for a coherent, under-500-character summary that excludes unrelated material while retaining important plot information.It frames the input as a local part of a larger work and requests only the summary.Source: F Prompt Templates for MiA-RAG · p. 18