PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context

Zhexi Feng, Ruiyi Zhang, Yongbo Yang, Pengtao Xie

arXiv:2608.25655v1cs.CLcs.AI

TL;DR

Long-running assistants must recover the operative episode from flat, mixed-topic conversations rather than merely retrieve relevant snippets. This paper introduces SCALE-QA and TSIM for episode-centered conversational memory, with TSIM achieving the highest accuracy across every backend setting and improving over the strongest baseline by 5.6–17.6 accuracy points.

  • Problem

    Flat, mixed-topic conversations can leave decisive constraints dormant for thousands of turns, creating an episode-integrity challenge beyond conventional long-context question answering.

  • Method

    The paper introduces SCALE-QA and TSIM, which reconstruct coherent episodes and rank their evidence before assembling answers from unsegmented conversational threads.

  • Results

    TSIM achieves the highest accuracy in every backend setting, improving over the strongest corresponding baseline by 5.6–17.6 accuracy points.

  • Takeaways & Limitations

    SCALE-QA frames conversational memory evaluation as dynamic reconstruction of the still-operative episode rather than retrieval from a static context.

  • Takeaways & Limitations

    SCALE-QA is counterfactually constructed rather than sampled from naturally occurring assistant logs, so real-log validation remains important.

Abstract

from arXiv · show

Conversations with chat assistants increasingly span many topics in a single long-running thread, challenging memory systems. Existing long-context and memory benchmarks often expose session or topic boundaries, or probe direct personal-memory questions. These settings understate a harder assistant-memory regime: a flat mixed-topic thread where the system must infer which earlier episode makes a later task decision valid. We introduce SCALE-QA, a constraint-grounded task QA benchmark for flat unsegmented threads targeting episode integrity failure. The dataset contains 3,000 audited questions across 10 domains, uses deterministic four-way multiple-choice grading, and includes a deterministic runtime builder; experiments use all 3,000 questions through 128k and a stratified 400-question diagnostic at 1M. SCALE-QA questions are ordinary task-oriented requests whose correct answer depends on causally related evidence introduced earlier in the conversation. We also propose Temporal-Semantic Interleaved Memory Reconstruction (TSIM), which segments the turn stream into coherent episodes and indexes them through a hierarchical multi-view memory stack with deterministic episode-level summary and cluster-routing views. Experiments show that SCALE-QA challenges strong RAG baselines and long-context LLMs alike; across three open-source and proprietary LLM backends, TSIM achieves the highest accuracy in every backend setting, gaining 5.6-17.6 accuracy points over the strongest corresponding baseline.

1 Introduction

SCALE-QA targets episode integrity failure in flat, mixed-topic conversational threads, where recovering the complete operative episode—not merely relevant snippets—is necessary for correct task decisions. TSIM reconstructs and indexes episodes, achieving 73.8% accuracy at 128k versus 29.8% for GPT-4o-mini Full Context and improving over the strongest corresponding baseline by 5.6–17.6 points.

  • Motivation: The benchmark addresses a regime distinct from ordinary long-context document QA because mixed-topic threads can leave constraints dormant for thousands of turns before they determine the only correct answer.Existing evaluations often relax flat mixed-topic structure, counterfactual construction, exact evidence auditability, or controlled context-length construction.
  • Problem formulation: Episode integrity failure occurs when decisive evidence is present but the system retrieves a plausible snippet, stale default, or compressed summary instead of the complete operative episode.The operative episode is the contiguous set of turns that jointly makes a local constraint or state binding for a later decision.
  • SCALE-QA: SCALE-QA measures dormant, private, cross-domain constraints in flat unsegmented threads through 3,000 audited questions across 10 task-oriented domains.It includes exact evidence traces and a deterministic length-controlled runtime builder, addressing gaps in clean isolation, attribution, and comparison across memory systems.
  • TSIM: TSIM reconstructs semantic episodes from the turn stream and indexes raw, episode-summary, and cluster-routing views instead of fixed token chunks.Its design tests whether memory should first recover the right episode and then present compact evidence to the answer model.
  • Empirical result: 73.8% Accuracy: TSIM versus 29.8% for GPT-4o-mini Full Context at 128k, with 5.6–17.6-point gains over the strongest corresponding baseline across three answer backends.These results support episode reconstruction as a stronger memory unit than chunk retrieval in the evaluated regime.

2 Related Work

SCALE-QA extends prior long-context, episodic-memory, and retrieval benchmarks by testing whether systems reconstruct a complete operative episode from a flat, mixed-topic conversation. It builds on LongMemEval while targeting complementary task-oriented decision QA with dormant local constraints rather than isolated retrieval.

  • Long-context benchmarks: SCALE-QA tests recovery of dormant local constraints from one flat mixed-topic conversation, complementing long-context benchmarks focused on understanding, positional robustness, and noisy or agentic contexts.Long-context studies also show that evidence may remain unused even when it fits within the context.
  • Episodic memory in language agents: Unlike episodic-memory systems that retrieve or store isolated memories, SCALE-QA evaluates reconstruction of the complete operative episode behind a later task decision.The benchmark draws on episodic-memory ideas from cognitive psychology and language-agent memory streams, recall buffers, episodic buffers, cognitive architectures, and pre-storage reasoning.
  • Closest predecessor: LongMemEval: Building on LongMemEval’s scalable-history and chat-distractor design, SCALE-QA targets complementary flat, unsegmented, task-oriented decision QA without boundary metadata.LongMemEval covers information extraction, multi-session, temporal, update reasoning, and abstention over length-configurable timestamped chat histories.
  • Representative benchmark example: In a representative SCALE-QA example, only reconstructing an earlier resource-policy episode yields the constraint-consistent FastText choice, whereas incomplete retrieval selects invalid BERT-Large or DistilBERT models.The figure contrasts TSIM with standard RAG and partial-recovery systems and includes LongMemEval’s direct-evidence example.
  • Retrieval and memory systems: TSIM tests whether inferred episodes are better retrieval units than fixed chunks, graph neighborhoods, or individual memory items used by prior retrieval and memory systems.The comparison includes RAG, dense retrieval, in-context RALM, RAPTOR, MemGPT, HippoRAG, and GraphRAG.

3 SCALE-QA: A Benchmark for Interleaved Long-Context Conversational QA

SCALE-QA evaluates whether systems can recover latent decision-relevant episodes from flat, mixed-topic conversations when ordinary task requests depend on dormant earlier constraints. It combines deterministic four-way grading, counterfactual local evidence, broad domain coverage, and audited runtime packaging without exposed session boundaries.

  • Benchmark formulation: Each instance presents a flat, unsegmented mixed-topic turn stream and a four-way task question whose answer requires recovering exact, hidden evidence turns.The latent decision-relevant span is an operative episode, distinct from mechanically defined chunks and explicit sessions.
  • Benchmark formulation: SCALE-QA uses deterministic four-way multiple-choice grading, with plausible distractors invalidated by local evidence rather than generic priors.Questions are ordinary task requests whose correct options depend on dormant earlier constraints.
  • Dataset construction and validation: Construction combines few-shot seed generation, counterfactual scenarios, deterministic normalization, filtering, adversarial distractor refinement, and human review to enforce answerability and evidence grounding.Counterfactual local constraints reduce reliance on public priors while preserving realistic decision structure; validation achieved 3,000/3,000 exact evidence matches across 4,346 audited snippets.
  • Dataset scope: The benchmark covers 10 task-oriented domains with 300 examples each, balanced correct options, and evidence ranging from operational notes and rules to code-like fragments and policy clauses.Diagnostic patterns include state overwrite, long-range bridge, and constraint trap; these are analysis views rather than separate failure modes.
  • Runtime protocol: Runtime packages mix accepted records with heterogeneous distractors, stale constraints, and unrelated dialogue at 16k–128k full-dataset lengths and diagnostic lengths through 1M, while removing session metadata.All systems receive identical packages, seeds, noise, and batch mappings; the gold answer and evidence trace remain unchanged.

4 TSIM: Temporal-Semantic Interleaved Memory Reconstruction

TSIM reconstructs operative episodes from mixed conversation streams, indexes each episode through multiple views, and ranks episode-level evidence before answer generation. This episode-centered design substantially improves retrieval on SCALE-QA over standard chunk-based RAG.

  • 4 TSIM: Temporal-Semantic Interleaved Memory Reconstruction: 7.7% CL Hit with Gemma2:9b at 128k versus 70.7% for TSIM shows episode retrieval, rather than answer selection, dominates the failure.Standard RAG often retrieves incomplete chunks that omit neighboring turns needed to make constraints operative.
  • 4 TSIM: Temporal-Semantic Interleaved Memory Reconstruction: TSIM uses semantic-shift segmentation, multi-view episode indexing, and evidence-first episode ranking to reconstruct episodes without externally supplied gold blocks.Its interface preserves fine-grained evidence while maintaining episode-level coherence before assembling compact answer evidence.
  • 4 TSIM: Temporal-Semantic Interleaved Memory Reconstruction: TSIM streams the turn sequence, infers contiguous operational episodes online, and applies minimum/maximum length guards without future turns, gold blocks, or offline clustering.The segmenter uses a similarity threshold θs = 0.70 and a model-to-user transition bonus b = 0.03, serving as a lightweight proxy rather than gold discourse boundaries.
  • 4 TSIM: Temporal-Semantic Interleaved Memory Reconstruction: Raw, summary, and cluster views anchor retrieval to episodes, while cluster hits route and boost episodes and the final prompt contains top-ranked episodes rather than isolated turns.Raw and summary evidence receive the largest scoring mass; clusters route candidate episodes instead of becoming prompt content.

5 Experimental Setup

The experiments evaluate the full 3,000-question SCALE-QA benchmark across constructed contexts from 16k to 128k, with a 1M diagnostic subset and three primary answer backends. TSIM is compared with multiple retrieval and memory baselines under matched runtime conditions using accuracy, evidence-hit rate, token, and latency metrics.

  • Dataset and context lengths: Experiments use all 3,000 SCALE-QA questions at 16k–128k context lengths and extend evaluation to a 1M diagnostic subset.The runtime builder deterministically constructs the evaluated contexts.
  • Answer backends: The main 128k comparison evaluates Gemma2:9b, Gemini 2.5 Flash, and GPT-4o-mini as local, long-context commercial, and high-throughput answer backends.DeepSeek R1 and Gemini 2.5 Flash additionally probe context scaling under strong-reasoning and long-window settings.
  • Compared systems: TSIM is compared with Standard RAG, Hybrid-RRF Chunk RAG, RAPTOR strict no-block (Sarthi et al., 2024), MEMGPT (Packer et al., 2023), and HIPPORAG (Gutiérrez et al., 2024).Full Context is reported only as a native-context diagnostic, while the GPT-4o-mini 128k block also includes Tuned Hybrid-Rerank Chunk RAG.
  • Controlled evaluation: All systems receive identical length-batched runtime packages, noise seeds, and writeback regimes, isolating differences to the memory or retrieval strategy.Frozen TSIM configuration, token accounting, variance audits, latency caveats, and runtime details are provided in Appendices B–F.
  • Metrics: The evaluation reports four-way MCQ Accuracy, closed-loop CL Hit, context tokens, and latency.CL Hit measures expected-evidence presence in retrieved context along the closed-loop trajectory.

6 Results

Across backends and context budgets, TSIM consistently outperforms full-context and retrieval baselines by reconstructing the operative episode rather than merely retrieving evidence. Its gains persist with stronger reasoning models, longer contexts, and component-level ablations.

  • Cross-backend main comparison: 69.6%, 80.2%, and 73.8% accuracy make TSIM the highest-accuracy method in Gemma2:9b, Gemini 2.5 Flash, and GPT-4o-mini at 128k.Under Gemini 2.5 Flash, TSIM improves over MEMGPT by 5.58 points while using approximately half the prompt context: 1.3K versus 2.3K tokens.
  • Component ablation: 74.2% accuracy with the full multi-view episode stack exceeds 55.5% with semantic-drift episodes, 43.4% with fixed-token retrieval, and 26.2% with Standard RAG.Accuracy rises monotonically as segmentation and multi-view memory components are added.
  • Context scaling: 93.8% versus 81.2% at 128k and 96.5% versus 87.2% at 1M show TSIM remains ahead of stronger long-context full-context models.At 1M, TSIM uses about 1.3k retrieved tokens and 2.16s latency, versus 1.05M prompt tokens and 23.87s for Full Context.
  • Failure analysis: 70.7% CL Hit and 69.6% accuracy for TSIM on Gemma2:9b contrast with 7.7% evidence retrieval and 24.4% accuracy for Standard RAG.Baseline failures are dominated by missing the decisive episode, while residual TSIM errors involve verbose or conflicting memory and local constraint misuse.
  • Reconstruction diagnostics: 0.810 all-evidence recall@5 for TSIM exceeds 0.719, 0.647, and 0.577 for fixed windows and 0.456 for Standard RAG.On LongMemEval-S, TSIM also reaches 71.0% judged accuracy versus 61.2% for fixed chunks and 56.6% for turn-level retrieval.

7 Conclusion

SCALE-QA targets episode integrity failure in long mixed-topic conversations, where recovering the operative episode matters more than merely fitting evidence within the context window. TSIM outperforms established baselines and answers with far fewer retrieved tokens than a long-window model while achieving higher diagnostic accuracy.

  • Conclusion: SCALE-QA exposes episode integrity failure by requiring recovery of dormant local constraints from long mixed-topic conversations.The central bottleneck is reconstructing the relevant episode, not whether the evidence fits inside the context window.
  • Conclusion: TSIM outperforms Standard RAG, Hybrid-RRF Chunk RAG, RAPTOR, MEMGPT, and HIP-PORAG on the 3,000-question SCALE-QA dataset.SCALE-QA remains oracle-answerable and zero-shot hard.
  • Conclusion: At 1M tokens, TSIM reaches 96.5% accuracy from about 1.3k retrieved tokens, versus 87.2% for a long-window model using 1.05M tokens and 23.87s.The diagnostic shows that seeing the evidence does not ensure reconstructing the episode that makes it operative.
  • Conclusion: Semantic-drift episodes outperform fixed-token chunks, while TSIM’s multi-granularity memory stack improves episode usability without simply inflating the prompt.These ablations explain why TSIM performs well on the benchmark.

8 Limitations

SCALE-QA’s counterfactual construction and four-way format improve auditability but limit representativeness and coverage of deployed, open-ended assistant-memory behavior. Real-log validation and complementary open-ended evaluation remain important future work.

  • Dataset construction: SCALE-QA is counterfactually constructed rather than sampled from assistant logs, limiting coverage of deployed systems’ distributions, styles, and privacy constraints despite reliability audits.The authors include exact evidence audits, oracle/zero-shot calibration, and a 300-example blind human audit with three anonymous annotators.
  • Evaluation format: Four-way multiple-choice grading makes episode-integrity failures reproducible and evidence-auditable but omits partial, hedged, tool-follow-up, and long-form explanatory responses.SCALE-QA is therefore positioned as a targeted diagnostic for constraint-grounded task QA, with open-ended assistant-memory evaluation left as complementary future work.

9 Ethics Statement

SCALE-QA uses synthetic, counterfactually privatized scenarios to evaluate context-grounded memory without providing professional advice, reducing privacy and leakage risks. LLM-assisted generation may introduce bias, so deterministic gates and human audits mitigate but do not eliminate risks; the benchmark is not for deployed clinical, legal, or financial decisions.

  • Ethics Statement: Counterfactually privatized synthetic scenarios model realistic medicine, law, finance, and operations decisions while reducing privacy risks and benchmark leakage.These scenarios evaluate context-grounded memory and evidence use rather than provide professional advice.
  • Ethics Statement: Three reviewers performed construction quality control and three anonymous auditors assessed realism; all six were unpaid research-group members familiar with the task.Reported runtime packages combine an author-curated synthetic seed with WildChat under its ODC-BY terms (Zhao et al., 2024).
  • Ethics Statement: LLM-assisted scenario seeds and constraints may inherit generation or domain-selection biases, although deterministic gates and human audits reduce but do not eliminate them.The authors position SCALE-QA as an evaluation resource, not a substitute for deployment-oriented decision systems in clinical, legal, or financial settings.

A SCALE-QA Dataset Construction and Audits … C Method and Baseline Implementation Details

SCALE-QA is a deterministically constructed, auditable benchmark for episode-integrity failure in flat mixed-topic threads, using four-way multiple-choice grading and exact evidence traces. Its runtime protocol controls context length and compares non-episodic retrieval with TSIM’s episode-segmented, multi-level memory implementation under identical evaluation conditions.

  • A SCALE-QA Dataset Construction and Audits: Every accepted record passes deterministic generation, normalization, adversarial refinement, evidence alignment, oracle-answerability, and hardness checks, followed by review from three human reviewers.Machine filtering accepted 28.8% of candidates, while human review accepted 84.3% of machine-filtered records.
  • A SCALE-QA Dataset Construction and Audits: SCALE-QA targets episode-integrity failure in flat mixed-topic threads, distinguishing itself from session-structured long-memory evaluation through task-oriented questions and deterministic four-way MCQ grading.The benchmark isolates memory recovery from generation-style variation while retaining open-ended evaluation as complementary.
  • A SCALE-QA Dataset Construction and Audits: The dataset contains 3,000 questions distributed evenly across ten domains, with globally balanced answer labels of A/B/C/D = 750/750/750/750.Stress cues are representative and may overlap within examples, including stale state, long-range bridges, and local exception traps.
  • A.1 Blind Human Realism Audit: The blind realism audit covered 300 stratified examples with three annotators, who independently saw complete source dialogues while gold answers, evidence, reasoning, and system outputs remained hidden.All three annotators completed every example, yielding 900 valid annotation rows; retained disagreements involved overlapping stress cues rather than removed invalid items.
  • B Length-Controlled Runtime Evaluation Protocol: The runtime evaluates systems at a user-specified benchmark-side target context length, deterministically packing truth records and noise while preserving identical selected records, seeds, batches, and executable files across methods.The 3,000-question truth corpus is approximately 393,245 benchmark-side tokens; 128k and 256k settings use length-batched evaluation, while 512k and larger targets can use the full corpus.
  • C Method and Baseline Implementation Details: All methods share persistent writeback and receive no gold blocks, future turns, or method-specific noise; the Tuned Hybrid-Rerank control combines sparse, dense, HyDE, fusion, expansion, and MMR components but remains non-episodic.The control does not use semantic episode segmentation or TSIM’s multi-granularity memory.
  • C Method and Baseline Implementation Details: TSIM stores raw-turn, episode-summary, and cluster-summary embeddings separately, using deterministic episode summaries and cluster routing; its scoring combines raw and summary hits with retrieved clusters and local semantic expansion.Episode summaries prefix relative recency and episode identifiers to truncated episode text, while cluster expansion supplies attached episodes from retrieved L2 clusters.

D TSIM Configuration Selection and Ablation · E Additional Experimental Results

TSIM uses one frozen, evidence-reconstruction-oriented configuration across answer backends, with sensitivity analyses showing stable retrieval near the selected segmentation threshold. Additional results audit performance across all ten domains without introducing another leaderboard.

  • D TSIM Configuration Selection and Ablation: TSIM uses one frozen configuration across answer backends, with calibrated constants selected for evidence-reconstruction stability rather than backend-specific answer accuracy.The protocol balances evidence coverage, answer transfer, and compact context across disjoint search, confirmation, and answer-validation splits.
  • D TSIM Configuration Selection and Ablation: The implementation comparison distinguishes the main retrieval and memory systems at a high level, while Official TSIM uses a local-window cosine rule for semantic-drift boundaries.The final system avoids a heavier offline clustering procedure for boundary decisions.
  • D TSIM Configuration Selection and Ablation: Table 16 summarizes the frozen-configuration selection protocol, and Table 18 lists the exact constants used in the reported TSIM configuration.The supplied passage establishes their roles but does not provide the constants themselves.
  • D TSIM Configuration Selection and Ablation: The reported configuration is not claimed uniquely optimal; nearby semantic-drift, retrieval-breadth, and routing-weight settings remain strong across disjoint validation splits.Search-only peaks are not selected.
  • D TSIM Configuration Selection and Ablation: Figure 6 evaluates single-variable M1 threshold sensitivity on 600 confirmation questions at 32k, 64k, and 128k with all other settings fixed.It reports all-evidence Recall@5 with Wilson 95% confidence intervals.
  • D TSIM Configuration Selection and Ablation: The paired-bootstrap 95% intervals for Recall@5 differences between θs = .66 and .70 include zero at 32k, 64k, and 128k, indicating stable retrieval near the selected threshold.The intervals are [−0.002, 0.042], [−0.013, 0.035], and [−0.038, 0.023], respectively.
  • E Additional Experimental Results: Table 19 expands the aggregate cross-backend comparison into all ten domains as an audit table rather than an additional leaderboard.Its purpose is domain-level auditing of the aggregate comparison.

E.1 Episode Reconstruction Diagnostics … F.1 System Cost and Qualitative Diagnostics

TSIM reconstructs evidence-bearing episodes more reliably than flat retrieval controls and transfers effectively to LongMemEval-S, while trading higher retrieval-side cost for shorter answer contexts. Diagnostics quantify retrieval coverage, prompt robustness, and implementation limitations across the evaluated protocols.

  • E.1 Episode Reconstruction Diagnostics: The official calibrated configuration achieves 86.33 Search Hit, 79.33 Confirm Hit, 87.50 Answer Hit, and 78.50 Answer Acc with 1011.6 Answer Ctx.These values are compared in the representative-neighborhood development sweep with compact and lean alternatives.
  • E.1 Episode Reconstruction Diagnostics: Exact-evidence diagnostics distinguish all-evidence recall@5 from co-containment because multi-snippet questions use different denominators and reconstructed episodes differ from control chunks or fixed windows.The metrics are computed over all 3,000 questions for recall and the multi-snippet subset for co-containment.
  • E.1 Episode Reconstruction Diagnostics: On all 3,000 questions, TSIM’s summary, L2, and raw views achieve evidence-episode hit rates of 0.861, 0.708, and 0.638, respectively.A hit requires returned content to map to a reconstructed episode containing matched gold evidence; frozen retrieval depths are 28 raw, 20 summary, and 2 L2.
  • E.2 LongMemEval-S Transfer Diagnostic: LongMemEval-S transfer is transductive rather than held-out generalization: configuration selection uses full-set retrieval criteria while gold answers, evidence, and question types remain hidden at inference.The adaptation uses one uniform configuration across all 500 cleaned V1 questions and Gemini 2.5 Flash at temperature zero.
  • E.2 LongMemEval-S Transfer Diagnostic: TSIM improves LongMemEval-S accuracy by 9.8 points over fixed-chunk retrieval and 14.4 points over BGE turn retrieval, while its 3.4-point gap to session boundaries is nonsignificant.The evaluation uses 500 cleaned V1 questions, with all methods receiving the same flat turn stream except the boundary-assisted session diagnostic.
  • F.1 System Cost and Qualitative Diagnostics: Under the evaluated protocol, TSIM uses higher embedding, indexing, and retrieval cost but reduces GPT-4o-mini answer context to about 1k tokens versus 3.0k for the tuned non-episodic control.These are protocol-specific accounting results; provider-side KV-cache reuse could change Full Context economics in deployment.
  • F.1 System Cost and Qualitative Diagnostics: The audit reports measured system accounting, paired-bootstrap validation over the same 3,000 questions, prompt-robust Full Context testing, and qualitative cases covering fragmentation, noisy windows, evidence misuse, and routing errors.Full Context prompt variants are selected on a 20-example development split and evaluated on a disjoint 100-example subset; latency and storage are implementation-specific.
Loading 2608.25655v1…