Role-aware Heuristic Episodic Attention for Conversational LLMs

1 Introduction

• Prior work reports that LLM performance often degrades in multi-turn conversations, limiting their reliability as consistent conversational agents (Shuster et al., 2022; Kwan et al., 2024; Laban et al., 2025).The introduction contrasts this with strong single-turn performance (Brown et al., 2020; Ouyang et al., 2022; Achiam et al., 2023).Source: §1 · p. 1
• Cumulative contextual decay comprises attention pollution from distracting or erroneous content, dilution by less relevant history, and drift as topics change and persistent constraints lose influence.A model may answer the immediate question while forgetting an earlier instruction that still applies.Source: §1 · pp. 1–2
• A full-history prompt does not distinguish constraints governing future responses from episodic information useful for a particular query; larger windows, retrieval, and compression leave this selection problem open.A global instruction can remain applicable after a topic change even when its wording has little semantic overlap with the new query.Source: §1 · p. 2
• REA retains identified global instructions in a dedicated Instructional Memory prefix, stores episodic exchanges in Episodic Memory, and uses Heuristic Context Retrieval to select raw, compressed, or omitted turns.Model replies are compressed while user inputs are preserved; the generator receives the complementary sources together.Source: §1 · p. 2
• 2.91× latency reduction on Long-MT-Bench+ accompanies improved long-horizon quality and instruction adherence; evaluations also span three backbones and Chinese and English role-playing tasks.The introduction also reports that removing IM reduces instruction adherence from 8.18 to 1.95 in the EM+HCR configuration.Source: §1 · p. 2

Long-Context Modeling and Compression

• Long-context modeling expands sequence capacity, while compression, summarization, retrieval, and reranking offer ways to reduce or organize context (Ding et al., 2024; Munkhdalai et al., 2024; Jiang et al., 2023; Louis et al., 2025; Mullick et al., 2024; Lee et al., 2024; Xu et al., 2025).REA focuses on the policy governing which content persists: identified global instructions bypass episodic retrieval, while historical replies can be presented at different resolutions.Source: §2 · p. 2

Structured Memory for Conversation

• Conversational memory systems such as MEMGPT (Packer et al., 2023), MEMOCHAT (Lu et al., 2023), and MEMORYOS (Kang et al., 2025) manage persistent state, while REA combines a persistent instruction prefix with reply compression and query-dependent retrieval.A same-backbone comparison with MemoChat evaluates REA’s complete policy against an established conversational memory system.Source: §2 · p. 3

Instruction Following and Consistency

• Instruction tuning and work on context-position effects address complementary aspects of instruction use; REA explicitly retains recognized global constraints in the generator input (Ouyang et al., 2022; Liu et al., 2023; Xiao et al., 2023).The paper presents this as complementing instruction tuning by preserving instructions expected to remain applicable throughout a conversation.Source: §2 · p. 3

3.1 Architecture Overview

• REA separates instruction and episodic memories, scores stored exchanges against each new query, and reconstructs a hybrid input with the current query.Before answering, it updates Instructional Memory; after generating a reply, it compresses and stores the completed exchange in Episodic Memory.Source: §3.1 · p. 3

3.2 Episodic Memory via Latent Compression

• Episodic Memory preserves user wording and stores model replies in raw and latent forms for resolution-adaptive recall.Separate compression and generation LoRA adapters share one backbone; generation consumes hybrid text and latent sequences.Source: §3.2 · pp. 3–4

3.3 Instructional Memory and Recognizer

• Instructional Memory retains global constraints, not task-specific requests, as text for later responses independently of episodic similarity.Recognition combines rule-based filtering with a lightweight Qwen3-0.6B classifier.Source: §3.3 · p. 4

3.4 Heuristic Context Retrieval

• HCR uses maximum pairwise cosine similarity and two thresholds to select raw exchange text, user text plus latent reply, or omission.The default thresholds are 0.5 and 0.7, and an omitted exchange can be retrieved on a later query.Source: §3.4 · pp. 4–5

3.5 Hybrid Context Reconstruction

• Hybrid reconstruction concatenates the instruction prefix, selected episodic content, and current query, combining persistent constraints with differently resolved history.Selected text is converted to embeddings while latent vectors are preserved.Source: §3.5 · p. 5

4.1 Experimental Setup

• The evaluation covers dialogue quality, instruction adherence, and state tracking across benchmarks ranging from 2 to 60+ turns, with multiple context-management baselines and a Mistral-7B primary backbone.MT-Bench, MT-Eval, and Long-MT-Bench+ use judge scores; the instruction task reports IAR and state tracking reports JGA.Source: §4.1 · p. 5

4.2 Main Results

• REA scores 7.36 versus Vanilla’s 6.32 on Long-MT-Bench+ (16.5% relative gain) and reduces average latency from 27.29 s to 9.39 s, a 2.91× speedup.On MT-Bench, scores are close: 8.49 for REA and 8.54 for Vanilla; REA adds 11.9% latency in that short-interaction setting.Source: §4.2 · pp. 5–6, Table 1 · p. 6

4.3 Backbone and Bilingual Evaluations

• REA improves MT-Eval scores across all three tested 1.7B–7B backbones and raises CharacterBench averages from 3.64 to 3.80 in Chinese and 3.72 to 4.05 in English.Memory and Persona improve in both languages.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 Diagnostic Analysis

• REA sustains instruction adherence and response quality as conversations grow, and improves distributed-state JGA, including Actions at 0.51 versus Vanilla’s 0.05.The diagnostic tracks IAR over turns and response quality on Long-MT-Bench+; distributed Code and Math JGA also rise from 0.09 to 0.16 and from 0.00 to 0.20.Source: §4.4 · p. 7, Figure 4 · p. 8

4.5 Ablation and Component Analysis

• On the instruction task, EM alone raises IAR from 5.76 to 7.37, while adding HCR without IM yields 1.95 versus 8.18 for full EM+HCR+IM.This ablation supports separately retaining persistent instructions alongside episodic retrieval.Source: §4.5 · p. 7, Table 4 · p. 7
• IR achieves 0.8947 recall and 0.6182 precision; missing instructions yields approximately 2.0 IAR, while dynamic reply selection scores 8.18 versus 6.63 for retaining all replies and 7.89 for discarding them.The tested extra-turn false positive yields approximately 7.7 IAR and remains closer to actual REA and oracle recognition.Source: §4.5 · pp. 7–8

5 Conclusion

• On Long-MT-Bench+, REA achieves a 16.5% relative judge-score gain and a 2.91× latency reduction over Vanilla.Same-backbone, cross-backbone, and bilingual evaluations add evidence within the tested settings.Source: §5 · p. 8
• REA keeps recognized global instructions in Instructional Memory while recalling episodic exchanges at variable resolution.The conclusion presents explicit persistence policies and selective episodic context as practical tools for multi-turn LLM interaction.Source: §5 · p. 8

Limitations

• Evaluation covers 1.7B–7B backbones and Chinese and English conversations; broader model and language coverage remains future work.Turn-level similarity can miss cross-turn references, and response-quality evaluation relies on automated judges that would benefit from human assessment.Source: Limitations · pp. 8–9

A.1 Dual-LoRA Implementation and Initialization

• REA's Episodic Memory uses a dual-LoRA architecture; model replies are encoded into latent vectors, while user inputs remain raw text.Eight soft compression tokens are used per turn, and the current query is also encoded for retrieval scoring.Source: §A.1 · p. 11

A.2 Training Protocol and Optimization

• REA is trained end-to-end to synchronize compression and generation objectives using 40,000 multi-turn sessions and a shared 1e−5 learning rate.Training uses AdamW, an effective batch size of 128, and two epochs on one NVIDIA RTX 3090 GPU.Source: §A.2 · p. 11

A.3 Training Data Curation

• Training data filtering removes large code blocks and simple interactions under 10 tokens, then retains dialogues spanning 3–15 turns.Instances pair conversation history and a current query with its target reply; response-generation loss trains both adapters jointly.Source: §A.3 · p. 11

A.4 Instruction Recognizer Configuration

• The Instruction Recognizer uses Qwen3-0.6B to classify global directives, such as style and format constraints, separately from ordinary queries.Its few-shot prompt is shown in Table 6.Source: §A.4 · p. 11, Table 6 · p. 11

B Theoretical Complexity and Efficiency Analysis

• The hybrid context length combines the instruction prefix, current query, and selected exchanges; retrieval scanning scales with stored exchanges, while generation cost scales with selected context length.For comparison, full-history dense prefill has a term O((TL)^2); persistent EM storage retains original text plus O(Tnd) latent values.Source: B Theoretical Complexity and… · p. 12

B.1 Empirical Performance and Memory Utilization

• On Long-MT-Bench+, the measured 27.29/9.39 latency ratio is approximately 2.91; benchmark averages are not a controlled scaling curve in T.Table 7 reports latency and peak VRAM for the evaluated benchmarks.Source: §B.1 · p. 12, Table 7 · p. 12

C.1 Benchmark Dataset Details

• The evaluation uses MT-Bench for 80 two-turn questions, MT-Eval for medium-length dialogue, Long-MT-Bench+ for 60+ turns on average, and a state-tracking diagnostic.MT-Eval has 168 samples averaging 6.96 turns; its Recollection subset diagnoses attention drift, while state tracking distributes key instructions across turns to test attention pollution.Source: §C.1 · p. 12

C.2 Baseline and Model Implementation Details

• Primary experiments use Mistral-7B-Instruct-v0.2 with greedy deterministic decoding, a 64K-token input cap, and 512-token outputs.The Vicuna comparison has a 4K context limit; Table 8 lists baseline implementation details and hyperparameters.Source: §C.2 · p. 13, Table 8 · p. 13

C.3 Evaluation Details

• Dialogue quality is scored by GPT-4 as a 0–10 Acc metric, while IAR checks global-instruction adherence and JGA exact-matches the final state-tracking answer.CharacterBench uses a five-point protocol and reports six aspect-level scores, separately from the primary dialogue evaluation.Source: §C.3 · p. 13

D.1 Validation Dataset Construction

• The recognizer validation set contains 418 IFEval-derived inputs, relabeled as 38 global constraints and 380 ordinary or task-specific inputs.This imbalance is intended to reflect the relative infrequency of persistent global constraints in conversation.Source: §D.1 · pp. 13–14

D.2 Recognizer Performance Metrics

• The recognizer recovers most labeled global constraints, but lower precision means some task-specific or ordinary inputs also enter Instructional Memory.Because only 38 inputs are positive, accuracy should be interpreted alongside class-specific precision and recall; Table 9 reports the metrics.Source: §D.2 · p. 14, Table 9 · p. 14

D.3 Impact Analysis of Error Types (FP vs. FN)

• Withholding true global instructions yields approximately 2.0 IAR, while adding one standard conversational turn to Instructional Memory yields approximately 7.7.The comparison includes oracle recognition, REA's Qwen3-0.6B pipeline, simulated false negatives, and simulated false positives; it supports retaining genuine constraints and tolerance to this particular extra entry.Source: §D.3 · p. 14

• The diagnostic task tests whether models track critical constraints or state variables distributed across a noisy, multi-turn dialogue.Source: E Detailed Diagnostic Analysis of… · p. 14
• Full Instruction supplies all constraints together in one turn, while Shared Instruction distributes the same rules across turns, increasing exposure to intervening replies.The Full condition provides a single-turn reference; Shared requires integration across a growing context window.Source: E Detailed Diagnostic Analysis of… · p. 14
• REA improves over distributed Vanilla in all three state-tracking subsets and reaches the single-turn Code score.Table 10 reports Joint Goal Accuracy (JGA), comparing Full, Shared, and REA conditions.Source: E Detailed Diagnostic Analysis of… · p. 14, Table 10 · p. 14

• Acc sensitivity to the two HCR thresholds is assessed on the Follow-up and Refinement subsets.Source: F In-depth Hyperparameter Sensitivity… · p. 14, Figure 6 · p. 15
• Results show greater sensitivity to τlow than to τhigh in the tested range, motivating attention to the exclusion threshold.Source: F In-depth Hyperparameter Sensitivity… · p. 14, Figure 6 · p. 15
• The default HCR thresholds are τlow = 0.5 and τhigh = 0.7.Source: F In-depth Hyperparameter Sensitivity… · p. 14

G Complete Bilingual Results

• Aggregate scores improve in the Chinese and English evaluations, alongside a small decrease in English Believability.Table 11 reports six aspects on a five-point scale, unlike the ten-point Acc scale used for primary dialogue benchmarks.Source: G Complete Bilingual Results · p. 15

H Qualitative Case Study and Discussion

• The qualitative study examines a persistent global constraint and visualizes prompt structures to analyze REA’s resilience.Source: H Qualitative Case Study and Discussion · p. 15

H.1 Behavioral Adherence Comparison

• Both systems meet the initial-letter constraint in Turn 1 and miss it in Turn 4; REA meets it again in Turn 10, unlike Vanilla.The comparison uses the instruction to begin each response with “r” and stay under 200 words.Source: §H.1 · p. 15, Table 12 · p. 16

H.2 Internal Prompt Visualization

• The illustrated REA prompts explicitly retain global instructions, while Vanilla’s instruction is buried under accumulated model replies.Table 13 compares the internal prompts sent to the generator.Source: §H.2 · p. 15, Table 13 · p. 16

H.3 Retrieval Decisions: Success and Failure

• A retrieval failure occurs when resolving “them” requires an earlier exchange that receives a low similarity score and is omitted at answer time.The success case uses low-resolution history for a later rewrite while Instructional Memory preserves global constraints.Source: §H.3 · p. 15, Table 14 · p. 16