PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization

Hyangsuk Min, Hwanjun Song

arXiv:2608.09043v1cs.CLcs.AI

TL;DR

Streaming dialogue summaries often lack the historical context needed to interpret the current window. ReMEMBER retrieves and refines evidence for unresolved dependencies, improving memory recall and gap-resolution completeness under a fixed budget across histories up to 160K tokens.

  • Problem

    Streaming dialogue summarization must interpret a non-self-contained current window whose unresolved references, attributes, and rationales require selective historical evidence.

  • Method

    ReMEMBER conditions retrieval on contextual gaps and refines retrieved chunks into turn-level, evidence-dense memory under a fixed budget.

  • Results

    0.6984 memory recall, outperforming Hybrid by 0.157; ReMEMBER also improves gap-resolution completeness by up to 0.17 across dialogue-length bins.

  • Takeaways & Limitations

    Under the same budget, targeting missing evidence improves memory recall and gap-resolution completeness across dialogue histories up to 160K tokens.

Abstract

from arXiv · show

Users of modern platforms repeatedly need summaries of recent dialogue, but the window rarely contains enough context to be interpreted on its own. We formalize this setting as streaming dialogue summarization, where a system must summarize a current window using selective memory from an unbounded history under a fixed budget. We show that the central challenge is not how much history is accessed, but whether memory recovers the evidence that the current window presupposes. We construct a benchmark and evaluation protocol that separately assesses whether memory contains gap-resolving evidence and whether the generated summary reflects it. We propose ReMEMBER, a missing-evidence memory framework that conditions retrieval on unresolved window dependencies and refines retrieved chunks into evidence-dense memory under a fixed budget. Experiments on dialogues with histories up to 160K tokens show that ReMEMBER improves memory recall and gap-resolution completeness over memory construction baselines under the same budget.

1 Introduction

Streaming dialogue summarization targets repeated summaries of recent dialogue segments despite histories whose missing context may lie far outside the current window. The paper frames selective memory as recovering that context under a strict budget and proposes ReMEMBER to retrieve and refine missing evidence.

  • Problem setting: Streaming dialogue summarization serves users who repeatedly need summaries of recent segments from continuous multi-turn dialogue streams.The setting covers message apps and collaborative workspaces, where users catch up on recent discussions or decisions rather than seek a global summary.
  • Challenge: Recovering the current window’s dependencies requires selective memory because full histories can span tens to hundreds of thousands of tokens and make full-context processing prohibitive and unreliable.The introduction cites lost-in-the-middle effects, needle-in-a-haystack failures, and inference overhead as long-context pathologies.
  • Motivation: Retrieval is identified as the strongest existing memory paradigm because it preserves explicit historical evidence and selects past turns relevant to the current window, but it has two streaming bottlenecks.The bottlenecks are similarity-driven construction and noisy memory utilization.
  • Method: ReMEMBER addresses missing evidence through gap-conditioned retrieval and gap-conditioned chunk refinement for budgeted streaming dialogue summarization.The first stage elicits unresolved dependencies and issues targeted queries; the second reduces memory noise by identifying the most useful retrieved content.
  • Contributions: The paper introduces a benchmark that separately evaluates whether memory contains evidence and whether summaries reflect that evidence.Its contributions also analyze representative memory paradigms and identify retrieval-based memory’s two streaming bottlenecks.

2 Related Work

Related work spans long-dialogue tasks, dialogue summarization, long-context memory construction, and query decomposition for retrieval. Streaming dialogue summarization differs by requiring models to identify unresolved information in the current window rather than answer a specified retrieval target.

  • Long Dialogue Tasks: Long-dialogue research covers multi-session response generation, long-dialogue understanding, and conversational search over accumulated conversation history.These settings include next-utterance prediction, question answering or entity tracking, and rewriting prior turns into explicit retrieval queries.
  • Long Dialogue Tasks: Streaming dialogue summarization has no specified retrieval target and must identify what the current window leaves unresolved.Its output is a coherent completion rather than a response to a question.
  • Dialogue Summarization: Dialogue summarization work models multi-turn dependencies, speaker roles, topic shifts, and discourse relations, while longer-dialogue methods use running-summary updates or hierarchical compression.The passage distinguishes dialogue-specific structural compression from approaches that scale summarization across longer dialogues.
  • Long-Context Memory Construction: Summarization-based memory preserves compact histories but may remove evidence for local contextual gaps, whereas retrieval-based memory can add irrelevant or redundant context.These limitations motivate distinguishing evidence recovery from broad context access.
  • Query Decomposition for Retrieval: Query decomposition improves evidence access by rewriting underspecified or complex inputs into explicit retrieval queries, while ReMEMBER pursues targeted retrieval through a different trigger and target.Related methods also guide retrieval with generated queries or hypothetical documents.

3 Streaming Dialogue Summarization: Formulation and Evaluation

This section defines streaming dialogue summarization as generating self-contained summaries of recent utterance windows using selective memory from unbounded history. It introduces an evidence-linked benchmark and evaluates memory recovery separately from summary faithfulness, coverage, and use of recovered context.

  • 3.1 Problem Formulation: The task observes a recent utterance window W_t and accumulated history H_t, then generates a self-contained summary using both.History-dependent content may require missing evidence from earlier dialogue.
  • 3.1 Problem Formulation: Because H_t grows unboundedly, the model maintains memory M_t that selectively preserves critical evidence for resolving contextual gaps.Task success depends on how effectively memory provides missing context for summary construction.
  • 3.2 Benchmark Construction: The benchmark links history-dependent window utterances to resolving evidence, enabling direct assessment of memory adequacy and summary self-containment.Existing dialogue summarization benchmarks generally lack this utterance-level grounding over long-term histories.
  • 3.2 Benchmark Construction: The dataset contains 35 long dialogues spanning chit-chat and workplace collaborative domains across three source datasets.It includes 20 chit-chat dialogues and 15 workplace dialogues; the domains differ in whether evidence is dispersed or contiguous in history.
  • 3.2 Benchmark Construction: Contextual gaps are annotated as referential, attribute, or relational cases when missing prior context makes salient window utterances incomplete.Salient utterances and contextual gaps jointly provide references for memory adequacy and summary quality.
  • 3.2 Benchmark Construction: A five-stage pipeline uses three independent LLMs per stage and retains outputs only by majority agreement to produce reproducible annotations.The retained gaps record the target salient utterance and resolving evidence, as indicated by the passage.
  • 3.3 Evaluation Metrics: Evaluation separates memory quality, measured by memory recall, from summary quality, measured by faithfulness, conciseness, and completeness.Completeness is further decomposed into window completeness and gap-resolution completeness.
  • 3.3 Evaluation Metrics: Window completeness measures salient current-window content covered, while gap-resolution completeness measures memory-resolved gaps reflected in the generated summary.Memory recall independently measures the proportion of annotated gaps whose resolution statements are satisfied by memory.

4 Methodology: ReMEMBER

ReMEMBER constructs streaming-dialogue memory around unresolved contextual gaps rather than the full current window. It detects gap-bearing utterances, retrieves targeted evidence, refines chunks into relevant turns, and allocates them across gaps within a fixed budget.

  • Gap Detection and Query Construction: ReMEMBER conditions retrieval on contextual gaps because window-derived queries tend to retrieve history that repeats visible content instead of presupposed evidence.The needed evidence may be a referent, state, rationale, or causal link not expressed through surface overlap.
  • Gap Detection and Query Construction: Gap detection identifies utterances that are both summary-worthy and gap-bearing, then constructs an evidence-seeking query for each unresolved dependency without hypothesizing its resolution.A summary-worthy utterance conveys information such as an action, decision, plan, or status update; gap-bearing means its interpretation depends on prior dialogue.
  • Candidate Chunk Construction: Candidate construction segments history into 128-token chunks with 32-token overlap and issues each gap query to sparse and dense retrievers.The design aims to isolate specific evidence while preserving enough turn context for retrieval; each retriever returns the top 30 chunks per gap.
  • Gap-Conditioned Turn Extraction: ReMEMBER extracts and scores individual utterances within retrieved chunks because chunk-level scoring can dilute clues concentrated in a single utterance.Each utterance and its gap query are embedded with Qwen3-Embedding-0.6B, then scored using cosine similarity.
  • Gap-Balanced Evidence Accumulation: Round-robin accumulation adds the highest-scoring turn from each gap in successive rounds until budget B is reached, preventing any single gap from monopolizing memory.This preserves representation for each unresolved dependency in Wt.

5 Evaluation

Section 5 evaluates whether ReMEMBER builds gap-resolving memory and improves streaming summaries under a fixed budget, isolating memory construction with a shared summarizer. Across memory recall, summary quality, gap types, ablations, and runtime, results favor evidence-targeted retrieval and refinement over indiscriminate or surface-based alternatives.

  • Evaluation setup: All methods receive the same current window and fixed summarizer, isolating the effect of memory construction.Memory quality uses memory recall; summary quality covers window completeness, gap-resolution completeness, faithfulness, conciseness, and their arithmetic-mean composite score.
  • Memory recall: 0.6984 memory recall: ReMEMBER outperforms Hybrid by 0.157, while full-history summaries underperform Recent Memory.The result indicates that evidence selection matters more than the amount of accessed history, and indiscriminate compression can discard task-relevant evidence.
  • Summary quality: ReMEMBER achieves the best composite score among memory construction baselines in every dialogue-length bin and improves Gap-Comp over Hybrid by up to 0.17.Window completeness, conciseness, and faithfulness remain strong, indicating that gains primarily come from resolving missing contextual dependencies.
  • Gap-type analysis: ReMEMBER most improves relational gaps by increasing both memory inclusion and summary coverage, whereas Full Memory buries evidence across all gap types.Even Ideal Memory does not guarantee complete evidence reflection, separating evidence availability from its use in the generated summary.
  • Component ablations: Removing gap-conditioned retrieval causes the larger ablation drop, while removing chunk refinement also degrades performance under the same budget.Window-level retrieval favors history overlapping with the current window rather than evidence resolving its implicit dependencies.
  • Efficiency: Approximately four seconds: ReMEMBER maintains this construction latency across dialogue-length bins while achieving the highest memory recall.Sparse, Dense, and Hybrid run in under one second but recover less gap-resolving evidence.

6 Conclusion

The paper introduces streaming dialogue summarization, showing that effective memory must recover what the current window leaves implicit. ReMEMBER addresses this by constructing memory around contextual gaps rather than surface similarity, improving recall and gap-resolution completeness under a fixed budget.

  • Streaming dialogue summarization is introduced as a new task with a benchmark and evaluation metrics.
  • Existing memory paradigms retrieve evidence aligned with the current window’s explicit statements rather than its implicit dependencies.
  • ReMEMBER constructs memory around detected contextual gaps instead of surface similarity.
  • Under the same budget, ReMEMBER improves memory recall and gap-resolution completeness across histories up to 160K tokens.

A Benchmark Construction Detail … A.3 Gap Annotation Pipeline Details

The benchmark comprises 900 window-level instances from 35 source dialogues across chit-chat and workplace domains, with contextual gaps categorized by missing referents, attributes, or relations. A five-stage pipeline identifies salient, history-dependent utterances, filters relevant history, annotates resolving evidence, and validates annotations through multi-model agreement and strict criteria.

  • A.1 Benchmark Statistics: The benchmark contains 900 window-level instances sampled from 35 source dialogues spanning LoCoMo, RealTalk21, and EverMemBench.Chit-chat instances come from LoCoMo and RealTalk21, while workplace instances come from EverMemBench.
  • A.2 Gap Taxonomy: Contextual gaps are organized into referential, attribute, and relational dependencies according to what prior history must supply.These correspond respectively to missing referents, properties or specifications, and causal, temporal, logical, or discourse relations.
  • A.2 Gap Taxonomy: Multiple missing dependencies in one salient utterance are annotated separately, with each utterance linked to a representative gap and resolving evidence.The evidence comes from prior history and explicitly resolves the missing referent, attribute, or relation.
  • A.3 Gap Annotation Pipeline Details: Salient utterances are concrete decisions, requests, action items, or status updates, while greetings, acknowledgments, praise, and emotional reactions are excluded.Consecutive utterances forming one coherent update or decision are merged into a single entry.
  • A.3 Gap Annotation Pipeline Details: An utterance receives a gap only when it cannot be summarized clearly and stably from the current window alone because earlier dialogue is required.The judgment examines unresolved references, underspecified attributes, and unstated reasoning that prevent independent summarization.
  • A.3 Gap Annotation Pipeline Details: Candidate history filtering retains chunks necessary or directly useful for understanding or summarizing the current window, excluding topical similarity alone.Relevant chunks may continue a decision or plan, resolve references, or provide earlier steps of an ongoing task.
  • A.3 Gap Annotation Pipeline Details: Evidence annotation audits candidate-chunk sentences independently, while validation requires sufficient, non-inferential evidence that makes the target utterance unambiguously self-contained.Across all stages, candidates are retained only when at least two of three LLMs agree, with disagreements resolved by majority label.

B Evaluation Metric Formulations

The evaluation separates memory’s preservation of missing historical evidence from the summary’s use, coverage, conciseness, and factual grounding of that evidence. It defines four complementary metrics with deterministic Qwen3.6-27B-based annotation and verification procedures.

  • Memory Recall: Memory recall measures the proportion of annotated contextual gaps for which memory explicitly contains the resolving information, verbatim or as a close semantic paraphrase.It evaluates historical-evidence preservation independently of the generated summary; labels use Qwen3.6-27B at temperature 0.0.
  • Window Completeness: Window completeness measures whether salient window utterances have primary claims recoverable from the summary, including both the core referent and essential relation.Narrative coverage without recovering the specific referent and relation does not qualify; Qwen3.6-27B performs this stage at temperature 0.0.
  • Gap-Resolution Completeness: Gap-resolution completeness measures how often memory-resolved gaps have their minimal gap-resolving fact explicitly expressed in the summary, including the specific referent and essential relation.It evaluates generation’s use of recovered evidence conditional on successful memory retrieval, using Qwen3.6-27B at temperature 0.0.
  • Conciseness: Conciseness measures the proportion of summary sentences serving a demonstrable communicative function through salient window coverage or resolved contextual gaps.It penalizes content serving neither function and reuses window-completeness and gap-resolution-completeness outputs without an additional model call.
  • Faithfulness: Faithfulness is evaluated at the atomic-fact level because sentence-level verification can miss partial hallucinations in long sentences conflating multiple claims.Memory methods verify facts against Mt ∪ Wt, while full-context methods retrieve the top-50 relevant chunks from Ht; both steps use Qwen3.6-27B at temperature 0.0.

C Additional Analysis of ReMEMBER

This appendix tests whether ReMEMBER’s benefits remain robust across summarization models, dialogue domains, and compact LLM choices. It separately examines summary quality, memory recall in Chit-Chat and Workplace settings, and gap detection variation.

  • Robustness across models and domains: The analyses evaluate ReMEMBER across summarization models and dialogue domains using window completeness, gap-resolution completeness, conciseness, faithfulness, and Composite quality.Composite is the mean of the other listed summary-quality measures.
  • Domain-wise memory recall: Memory Recall is reported separately for Chit-Chat and Workplace domains across memory construction methods.The domain-wise comparison tests whether recall improvements hold across both dialogue settings.
  • Gap detection model variation: Gap detection is varied across models, with domain-wise Memory Recall and Gap-Comp reported for summaries generated by Qwen3.5-4B.GapComp denotes Gap-Resolution Completeness.

C.1 Gap Detection Model Ablation · C.2 Memory Recall Across Domains

Gap detection performs comparably across three compact models, while domain characteristics explain more recall variation than model capacity. ReMEMBER achieves the highest Memory Recall among baselines in both chit-chat and workplace dialogues.

  • C.1 Gap Detection Model Ablation: Three compact gap detection models yield comparable performance across two domains.The result indicates that gap detection does not require a large model.
  • C.1 Gap Detection Model Ablation: Performance varies more across domains than across gap detection models.This suggests domain characteristics drive recall differences more than model capacity.
  • C.2 Memory Recall Across Domains: ReMEMBER achieves the highest Memory Recall among memory construction baselines in both dialogue domains.The comparison covers the chit-chat and workplace domains.
  • C.2 Memory Recall Across Domains: 0.6918 is ReMEMBER's Memory Recall in the chit-chat domain.This is the reported domain-specific recall value in Table 8.
  • C.2 Memory Recall Across Domains: 0.7018 is ReMEMBER's Memory Recall in the workplace domain.This is the reported domain-specific recall value in Table 8.
  • C.2 Memory Recall Across Domains: Gap-conditioned memory construction improves evidence recovery across both chit-chat and workplace dialogue settings.In chit-chat, summarization-based memory is relatively strong, with Hier. Summary reaching 0.5519.

C.3 Performance Across Summarizers and Domains … F Use of Generative AI

ReMEMBER delivers the strongest overall summarization performance across models and domains by improving gap-resolution completeness, while the paper documents implementation resources, scientific artifacts, and generative-AI assistance.

  • C.3 Performance Across Summarizers and Domains: ReMEMBER achieves the highest composite score across three summarizers and two dialogue domains, mainly through gains in gap-resolution completeness.The composite balances window completeness, gap-resolution completeness, conciseness, and faithfulness.
  • C.3 Performance Across Summarizers and Domains: ReMEMBER remains strongest with Qwen3.5-4B, Qwen3.5-9B, and Gemma-4-E2B-it across both domains while maintaining competitive window completeness and faithfulness.Sparse, Dense, and Hybrid retrieval methods often preserve faithfulness but have lower Gap-Comp because window-level retrieval misses required evidence.
  • C.3 Performance Across Summarizers and Domains: No Memory preserves faithfulness but cannot resolve history-dependent gaps, whereas Full Memory can reduce window completeness and conciseness, especially in the workplace domain.Ideal Memory provides an upper bound, yet its sub-perfect gap-resolution scores show that generation can still fail when evidence is supplied.
  • D Model and Hardware Details: Experiments use Qwen3.5-4B, Qwen3.5-9B, and Gemma-4-E2B-it for gap detection and summary generation, with additional larger models for benchmark construction and evaluation.Table 10 records model and embedding identifiers, datasets, hardware, precision, peak allocated VRAM, and average runtime cost.
  • E Scientific Artifacts: All experiments use existing research datasets and open-weight language models whose checkpoints are publicly accessible through Hugging Face.Benchmark statistics and implementation details appear in Appendices A.1 and D.
  • F Use of Generative AI: AI assistants supported coding with Codex and grammar checking with Claude during paper preparation.The paper also includes prompt templates for gap detection, memory recall, completeness, faithfulness, summarization, retrieval, and contextual gap resolution.
  • F Use of Generative AI: The contextual-gap workflow audits whether retrieved sentences explicitly supply concrete missing referents, attributes, or logic, then applies a final non-inference sufficiency check.The prompts distinguish coreference, attribute/state, and causal or logical gaps and require evidence to match the defined gap precisely.
Loading 2608.09043v1…