PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization

Hyangsuk Min, Hwanjun Song

arXiv:2608.09043v1cs.CLcs.AI

TL;DR

Recent dialogue windows often omit history needed to interpret them. ReMEMBER builds fixed-budget memory around unresolved gaps, improving memory recall and gap-resolution completeness over memory construction baselines under the same budget.

  • Problem

    Streaming dialogue summaries often concern recent segments whose windows omit antecedents, attributes, and rationales supplied by shared history.

    Source: §1 · p. 1

  • Method

    ReMEMBER detects unresolved window dependencies, retrieves historical evidence that resolves them, and refines retrieved chunks into turn-level memory under a fixed budget.

    Source: §1 · p. 2

  • Results

    Across histories up to 160K tokens, ReMEMBER improves memory recall and gap-resolution completeness under the same budget; memory recall is 0.6984, 0.157 above Hybrid.

    Source: §5.2 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    Evidence recovery and evidence use are distinct: even reference gap-resolving evidence does not guarantee full evidence reflection in the summary.

    Source: §5.4 · p. 9

    A persistent gap between memory recall and summary use indicates that summary generation independently limits gap resolution.

    Source: §6 · p. 10

Abstract

from arXiv · show

Users of modern platforms repeatedly need summaries of recent dialogue, but the window rarely contains enough context to be interpreted on its own. We formalize this setting as streaming dialogue summarization, where a system must summarize a current window using selective memory from an unbounded history under a fixed budget. We show that the central challenge is not how much history is accessed, but whether memory recovers the evidence that the current window presupposes. We construct a benchmark and evaluation protocol that separately assesses whether memory contains gap-resolving evidence and whether the generated summary reflects it. We propose ReMEMBER, a missing-evidence memory framework that conditions retrieval on unresolved window dependencies and refines retrieved chunks into evidence-dense memory under a fixed budget. Experiments on dialogues with histories up to 160K tokens show that ReMEMBER improves memory recall and gap-resolution completeness over memory construction baselines under the same budget.

1 Introduction

Streaming dialogue summarization uses selective memory to recover prior evidence that the current window presupposes. ReMEMBER retrieves unresolved dependencies and refines retrieved chunks into evidence-dense memory under a fixed budget.

  • Current windows often omit context needed to interpret pronouns, entities, and decisions; incremental and hierarchical methods lose fine-grained evidence, while full-history input is costly and unreliable.Prior approaches include incremental summarization (Wu et al., 2025) and hierarchical summarization (Kim and Kim, 2025; Li et al., 2025b; Ou and Lapata, 2025).Source: §1 · pp. 1–2
  • Retrieval is the strongest examined memory paradigm, but it selects from what the window expresses and returns noisy chunks, rather than focusing on unresolved evidence.The identified bottlenecks are similarity-driven memory construction and low evidence density from neighboring turns, repetitions, and topic drift.Source: §1 · p. 2
  • ReMEMBER improves memory recall and gap-resolution completeness over recency-, summarization-, and retrieval-based baselines under the same budget on histories up to 160K tokens.The experiments cover long-context chit-chat and workplace dialogues.Source: §1 · p. 2
  • ReMEMBER elicits unresolved window dependencies to retrieve resolving historical evidence, then retains the most relevant turns from retrieved chunks to reduce noise within a fixed budget.This shifts memory construction from similarity-based chunk retrieval toward evidence-dense memory.Source: §1 · p. 2
  • The paper introduces streaming dialogue summarization and a benchmark that separately evaluates whether memory contains needed evidence and whether the summary reflects it.Source: §1 · p. 2

2 Related Work

Streaming dialogue summarization differs from prior long-dialogue and summarization settings because it must identify what a current window leaves unresolved and recover the context needed to interpret it. Related memory and retrieval methods motivate this task but do not directly construct evidence for its gaps.

Long Dialogue Tasks

Dialogue Summarization

  • Prior dialogue summarization handles dialogue structure, long histories, or explicit information needs, but typically assumes a complete dialogue and produces a global summary.Methods address dialogue-specific structure (Tang et al., 2022; Jia et al., 2023; Yi et al., 2025), incremental or hierarchical long-dialogue summarization (Hwang et al., 2024; Ravaut et al., 2024; Zhu et al., 2020; Li et al., 2021; Zhang et al., 2021), and query-focused or instructive summarization (Zhong et al., 2021; Wang et al., 2023a).Source: §2 · p. 3

Long-Context Memory Construction

  • Existing memory methods compress, retrieve, or organize history, yet do not build evidence specifically to resolve target-window gaps.Summarization-based memory (Wang et al., 2025; 2026), retrieval-based memory (Zhong et al., 2024; Tan et al., 2025; Pan et al., 2025; Li et al., 2025a), intent- and goal-driven memory (Du et al., 2026; Yan et al., 2026), and structured memory (Ong et al., 2025; Yang et al., 2026) have limitations including lost evidence, irrelevant context, or optimization for other purposes.Source: §2 · p. 3

Query Decomposition for Retrieval

  • ReMEMBER retrieval is triggered by discourse incompleteness and targets concrete prior-dialogue evidence rather than plausible answers.Unlike query reformulation and generated-query or hypothetical-document approaches (Elgohary et al., 2019; Yoon et al., 2025; Zhu et al., 2025a; Mao et al., 2021; Gao et al., 2023a; Wang et al., 2023b), its queries specify non-hypothetical needs such as referents, prior states, and rationales.Source: §2 · p. 3

3 Streaming Dialogue Summarization: Formulation and Evaluation

The section defines streaming dialogue summarization as producing self-contained summaries of recent windows using selective memory of unbounded history. It introduces a benchmark linking history-dependent utterances to resolving evidence and evaluates both memory recovery and summary quality.

3.1 Problem Formulation

  • Streaming summarization generates a self-contained summary of each recent utterance window using selective memory of unbounded prior history to recover evidence needed to resolve contextual gaps.The summary should preserve salient window content, while memory stores critical historical evidence rather than all past interactions.Source: §3.1 · pp. 3–4

3.2 Benchmark Construction

  • Explicit links between history-dependent window utterances and resolving historical evidence enable assessment of memory adequacy and summary self-containment.The benchmark addresses the lack of utterance-level grounding in existing dialogue summarization benchmarks.Source: §3.2 · p. 4
  • The benchmark samples 900 instances from 1,079 candidates across 35 chit-chat and workplace dialogues, with histories ranging from 7K to 160K tokens.Dialogues are segmented into 1,024-token windows; candidates with histories under 7K tokens or entirely self-contained salient utterances are excluded.Source: §3.2 · p. 4
  • Ground truth pairs salient window utterances with contextual gaps, which capture missing prior context through referential, attribute, or relational dependencies.Each history-dependent salient utterance receives one representative gap resolved by one or more historical evidence utterances.Source: §3.2 · p. 4
  • A five-stage annotation pipeline retains outputs only by majority agreement among three independent LLMs from distinct model families.Each retained gap records its target utterance, type, resolving historical evidence, and resolution statement.Source: §3.2 · p. 4

3.3 Evaluation Metrics

  • Evaluation separates memory quality, measured by memory recall, from summary quality, measured by faithfulness, conciseness, and completeness; completeness is split into window and gap-resolution completeness.This separation distinguishes evidence recovery from summary generation and use of recovered evidence.Source: §3.3 · pp. 4–5
  • Memory recall measures the proportion of annotated gaps whose resolution statements memory satisfies, while gap-resolution completeness measures the proportion of memory-resolved gaps reflected in the summary.Memory recall assesses evidence preservation independently of summary generation.Source: §3.3 · p. 5
  • Window completeness is the fraction of salient utterances represented in the summary, independent of memory-related failures.It measures coverage of salient content observable in the current window.Source: §3.3 · p. 5

4 Methodology: ReMEMBER

ReMEMBER builds memory around contextual gaps that the current dialogue window presupposes, rather than relying on global compression or window-focused retrieval. It detects unresolved dependencies, retrieves and refines gap-relevant evidence into budgeted turn-level memory, then supplies that memory with the current window to a summary generator.

4.1 Gap-Conditioned Evidence Retrieval

  • ReMEMBER detects summary-worthy, gap-bearing utterances in the current window and turns each unresolved dependency into an evidence-seeking query that avoids hypothesizing its answer.Detection uses only the current window and requires no benchmark annotations or gap labels; queries specify the utterance, why it is not self-contained, and what evidence is needed.Source: §4.1 · pp. 5–6
  • ReMEMBER retrieves 128-token history chunks with 32-token overlap using BM25 and Qwen3-Embedding-0.6B, fuses ranked lists with reciprocal rank fusion, and retains top-K chunks per gap.Each retriever returns the top 30 chunks per gap; K is determined by the memory budget divided by chunk size.Source: §4.1 · p. 6, §4.3 · p. 6

4.2 Gap-Conditioned Chunk Refinement

  • ReMEMBER decomposes retrieved chunks into utterances, scores them against each gap query, reranks them, and extracts high-scoring complete turns with speaker information as evidence units.The utterances and gap query use the same embedding encoder as retrieval, and cosine similarity provides the gap-conditioned relevance signal without additional supervision.Source: §4.2 · p. 6
  • ReMEMBER accumulates turns round-robin across gaps until the memory budget is reached, preventing one gap from monopolizing memory and preserving representation for each unresolved dependency.In each round, it appends the next-highest-scoring turn from each gap unless already included.Source: §4.2 · p. 6

4.3 Integration with Summary Generation

  • The memory module can be paired with any standard summary generator; in experiments, the generator is fixed across memory strategies and receives only the memory and current window.The generator has no access to the full dialogue history.Source: §4.3 · p. 6

5 Evaluation

Evaluation tests whether ReMEMBER constructs gap-resolving memory and improves streaming summaries under a fixed memory budget, measuring evidence recovery and summary quality across dialogue lengths and gap types.

5.1 Experimental Setup

  • The evaluation holds the summarizer and current window fixed while each method builds memory, measuring memory recall and four summary dimensions plus their composite mean.Baselines include diagnostic references and memory-construction methods based on recency, history compression, or sparse, dense, and hybrid retrieval.Source: §5.1 · p. 7

5.2 Gap-Resolving Evidence Recovery

  • 0.6984 memory recall, 0.157 above Hybrid, makes ReMEMBER the strongest evidence-recovery method; Inc. Summary and Hier. Summary access full history yet trail Recent Memory.Indiscriminate compression discards task-relevant evidence, while retrieval methods remain limited by surface similarity and miss implicitly connected evidence regardless of retrieval budget.Source: Table 1 · p. 8, §5.2 · p. 8

5.3 Impact on Streaming Summaries

  • Up to 0.17 higher gap-resolution completeness than Hybrid, ReMEMBER has the best composite score among memory-construction baselines in every dialogue-length bin.Full Memory exposes more history but achieves lower window completeness and conciseness than retrieval-based memories, underscoring that evidence must be organized for reliable use.Source: Table 2 · p. 8, §5.3 · p. 8

5.4 Gap-Level Evidence Use

  • ReMEMBER improves memory inclusion and summary coverage for relational gaps by reasoning beyond lexical overlap, but even Ideal Memory does not ensure evidence is reflected.Full Memory buries resolving evidence among irrelevant context across gap types, a needle-in-a-haystack failure; the inclusion-to-use gap points to co-designing memory construction and summary generation.Source: Table 3 · p. 9, §5.4 · p. 9

5.5 Ablation Study

  • Removing gap-conditioned retrieval causes a larger performance drop than removing chunk refinement, though both components improve ReMEMBER.Window-level retrieval favors history overlapping with the current window, while unrefined chunks consume fixed memory capacity with gap-irrelevant context.Source: Table 4 · p. 9, §5.5 · p. 9

5.6 Latency–Memory Recall Trade-off

  • Approximately four seconds across dialogue-length bins, ReMEMBER’s construction latency accompanies the highest memory recall; retrieval methods take under one second but recover less evidence.Hier. Summary has the highest runtime and lowest recall, showing repeated compression is costly and lossy under the fixed budget.Source: §5.6 · p. 9, Figure 2 · p. 10

6 Conclusion

ReMEMBER builds memory around contextual gaps rather than surface similarity, improving memory recall and gap-resolution completeness under the same budget across histories up to 160K tokens. The conclusion also identifies a remaining gap between recalled evidence and its use in summaries.

6 Conclusion

  • Under the same budget, ReMEMBER improves memory recall and gap-resolution completeness across histories up to 160K tokens.Even when memory contains reference gap-resolving evidence, summary generation remains an independent limitation on gap resolution.Source: §6 · p. 10
  • The paper introduces streaming dialogue summarization as a task with a benchmark and evaluation metrics.Its analysis finds existing memory paradigms retrieve evidence aligned with what the current window states, rather than what it leaves implicit.Source: §6 · p. 10
  • ReMEMBER constructs memory around detected contextual gaps rather than surface similarity.Source: §6 · p. 10

A Benchmark Construction Detail … A.3 Gap Annotation Pipeline Details

The benchmark uses 900 window-level instances from 35 source dialogues, and its annotations identify history-dependent gaps and evidence that explicitly resolves them. The annotation pipeline filters for salient, non-self-contained utterances, screens relevant history, and validates evidence for exact, non-inferential resolution.

A.1 Benchmark Statistics

  • The benchmark contains 900 window-level instances sampled from 35 source dialogues: 10 LoCoMo, 10 RealTalk21, and 15 EverMemBench dialogues.Chit-chat instances come from LoCoMo and RealTalk21, while workplace instances come from EverMemBench; dialogue length is measured as |Ht| + |Wt| with Wt fixed at 1,024 tokens.Source: §A.1 · p. 16

A.2 Gap Taxonomy

  • Contextual gaps are history-resolvable omissions of a referent, attribute, or relation needed to interpret the current window, rather than summarizer-introduced errors.Referential gaps concern what an utterance refers to; attribute gaps concern a missing property or specification; relational gaps concern omitted causal, temporal, logical, or discourse connections.Source: §A.2 · pp. 16–17
  • When a salient utterance has multiple missing dependencies, annotators record one representative gap and one or more history evidence pieces that explicitly resolve it.The representative gap is the dependency whose absence most impairs summarizability; evaluation associates each history-dependent utterance with one such gap.Source: §A.2 · p. 17, §A.3 · p. 17

A.3 Gap Annotation Pipeline Details

  • Annotators select concrete decisions, requests, action items, and status updates, excluding acknowledgments and reactions and merging consecutive utterances that form one coherent update.An utterance qualifies as history-dependent only if at least two LLMs agree on its gap type.Source: §A.3 · p. 17
  • A gap is assigned only when missing information prevents clear, stable summarization from Wt alone and cannot be recovered from nearby window utterances.Extra context that would merely enrich a summary does not qualify; each utterance receives at most one representative gap.Source: §A.3 · p. 17
  • Candidate history chunks are retained only when necessary or directly useful for understanding Wt; topical similarity or unsupported inference is insufficient.Examples of relevant content include an explicitly continued plan, information resolving a reference, or earlier steps of an ongoing task.Source: §A.3 · p. 18
  • For each history-dependent utterance, each sentence in retained chunks is judged independently and resolves the gap only if it explicitly supplies the presupposed missing information.Topical overlap, shared entities, proximity, and broad background do not qualify; positive evidence is aggregated into a resolution statement.Source: §A.3 · p. 18
  • Validation accepts evidence only when it is sufficient, non-inferential, makes the utterance independently unambiguous, and adds information not already in Wt.Validation catches overlooked in-window evidence and vague or indirect resolutions.Source: §A.3 · p. 18

B Evaluation Metric Formulations

The section defines complementary metrics for whether memory recovers historical evidence, summaries express salient window claims and resolved contextual facts, and generated claims are useful and grounded.

Memory Recall

  • Memory recall assesses whether memory explicitly contains resolving evidence for annotated contextual gaps, separately from what the generated summary says.It measures memory preservation independently of summary generation.Source: B Evaluation Metric Formulations · p. 18

Window Completeness

  • Window completeness measures whether the summary recovers salient utterances’ primary claims, including both the core referent and essential relation.Narrative coverage of the same topic without those specifics does not qualify.Source: B Evaluation Metric Formulations · pp. 18–19

Gap-Resolution Completeness

  • Gap-resolution completeness checks whether the summary explicitly expresses the minimal fact resolving a gap that memory successfully retrieved.The fact must include the specific referent and essential relation.Source: B Evaluation Metric Formulations · p. 19

Conciseness

  • Conciseness measures the proportion of summary sentences that reflect salient window content or express resolved contextual gaps.It penalizes sentences serving neither function and requires no additional model call.Source: B Evaluation Metric Formulations · p. 19

Faithfulness

C Additional Analysis of ReMEMBER

The appendix examines whether ReMEMBER’s benefits are robust to summarization-model choice, dialogue domain, and the compact LLM used for gap detection.

C Additional Analysis of ReMEMBER

C.1 Gap Detection Model Ablation · C.2 Memory Recall Across Domains

ReMEMBER leads memory-construction baselines in recall across chit-chat and workplace dialogues, while compact gap-detection models perform comparably. Recall differences are more pronounced across domains than across gap-detection models.

C.1 Gap Detection Model Ablation

  • All three compact gap-detection models yield comparable performance, indicating that gap detection does not require a large model.Source: §C.1 · p. 21
  • Performance varies more across domains than across gap-detection models, suggesting domain characteristics drive recall differences more than model capacity.Source: §C.1 · p. 21

C.2 Memory Recall Across Domains

  • 0.6918 in chit-chat and 0.7018 in workplace are ReMEMBER’s highest Memory Recall scores among memory construction baselines.It improves over Hier. Summary by 0.1399 in chit-chat and over Hybrid by 0.1299 in workplace.Source: §C.2 · p. 21
  • In chit-chat, Hier. Summary reaches 0.5519 recall and Inc. Summary 0.4859, with summarization-based memory relatively strong.The passage attributes this to chit-chat dialogues being shorter on average, making compression less lossy than in longer workplace histories.Source: §C.2 · p. 21
  • In workplace dialogues, summarization-based memory degrades sharply, while retrieval-based Hybrid reaches 0.5719 recall.The passage suggests that in longer histories, repeated compression loses fine-grained evidence whereas retrieval better preserves localized historical evidence.Source: §C.2 · p. 21
  • Querying unresolved contextual gaps recovers evidence missed by both compression-based and window-level retrieval baselines.Source: §C.2 · p. 21

C.3 Performance Across Summarizers and Domains … F Use of Generative AI

ReMEMBER achieves the strongest overall summary quality across tested summarizers and dialogue domains, with gains driven mainly by gap-resolution completeness. The accompanying sections document implementation resources, scientific artifacts, and generative-AI use.

C.3 Performance Across Summarizers and Domains

  • Across three summarizers and two dialogue domains, ReMEMBER achieves the highest composite score in every setting, with gains mainly from Gap-Comp.It also improves Gap-Comp over recency-, summarization-, and retrieval-based memory, while maintaining competitive window completeness and faithfulness.Source: §C.3 · pp. 21–22
  • Retrieval-based memories often preserve faithfulness but have lower Gap-Comp, while summarization-based memories sometimes improve window completeness but lose fine-grained historical evidence.The reported explanation is that window-level retrieval misses evidence needed for underspecified utterances, while repeated compression discards historical detail.Source: §C.3 · p. 22
  • No Memory cannot resolve history-dependent gaps, Full Memory can reduce window completeness and conciseness, and Ideal Memory still falls short of perfect gap resolution.The Full Memory drawbacks are especially noted in the workplace domain; Ideal Memory shows that generation can fail to express resolved information even when relevant evidence is supplied.Source: §C.3 · p. 22

D Model and Hardware Details

  • Experiments used three models for gap detection and summary generation, three others for benchmark construction and evaluation, and a single NVIDIA H200 GPU.Runs used NLTK punkt for sentence segmentation and tiktoken cl100k base for approximate token counting; all results are from a single experimental run.Source: D Model and Hardware Details · p. 22

E Scientific Artifacts

  • The experiments use existing research datasets and open-weight language models, with model checkpoints publicly accessible through Hugging Face.Benchmark statistics and implementation details are provided in Appendices A.1 and D.Source: E Scientific Artifacts · p. 22

F Use of Generative AI

Related papers

Loading 2608.09043v1…