PaperlayerView exact revision on arXiv

Source-linked AI summary

CoLink: Global Compression and Knowledge Linking for Multi-Turn Dialogue Modeling

Lian Yan, Zhenbo Xia, Haotian Wang, Chen Tang, Yi Guan, Jie Liu, Jingchi Jiang

arXiv:2610.34335v1cs.CLcs.AI

TL;DR

Long-horizon dialogue modeling is challenged by restricted context windows and difficulty retrieving relevant information from noisy histories. CoLink links historical topic graphs with external knowledge and compresses history and activated knowledge into compact representations. Across two datasets, it outperforms retrieval and local-token baselines under strong compression and can match or exceed full-history performance.

  • Problem

    Long-horizon dialogue modeling is challenged by restricted context windows and difficulty assigning attention to relevant history and knowledge amid conversational noise.

    Source: A. Problem Definition · p. 3

  • Method

    CoLink links historical topic graphs to external knowledge and uses Information Bottleneck objectives with competitive slots to compress history and knowledge.

    Source: §I · pp. 1–2

  • Results

    Across two datasets, CoLink outperforms structured-retrieval and local-token baselines by an average 2.59% under a 32× compression constraint and matches or exceeds Full History.

    Source: 1) Comparison Results · p. 8

  • Takeaways & Limitations

    Evaluations report that CoLink generalizes across domains and models while using only a small fraction of the original context length.

    Source: §VI · p. 12

Abstract

from arXiv · show

Extracting salient information from long-horizon dialogue histories remains a significant challenge for Large Language Models (LLMs) due to the restricted context windows and efficient retrieval challenges. Current methods typically rely on history organization or local token-level optimization, suffering from information loss, retrieval redundancy, and myopic attention. Inspired by the view in cognitive science that dialogue constitutes sense-making and memory association, we propose CoLink, a framework that enhances LLMs' global context awareness in domain-specific multi-turn dialogues through information linking and compression. CoLink first clusters historical contexts into topic graphs to establish cross-semantic associations with external knowledge. Guided by the adaptive control of thought, it then retrieves and activates knowledge via synergizing semantic relevance and graph-structural topology. Crucially, to achieve effective compression of history and knowledge, we ground multi-turn dialogues in information theory and derive a principled optimization objective for extracting the core sense-making information. Coupled with a competitive slot mechanism, CoLink condenses the global context into dedicated memory and knowledge slots, efficiently distilling lengthy, redundant inputs into an extremely compact latent representation. Extensive empirical results on both public and our MMD datasets validate CoLink's superiority and generalizability in knowledge linking and sense modeling.

I. INTRODUCTION

CoLink addresses long-dialogue context challenges by linking history to external knowledge and compressing both into globally coherent representations. It contrasts this approach with retrieval structures and local-token methods, and motivates its design through cognitive accounts of associative memory and abstraction.

I. INTRODUCTION

  • Long-context LLMs face restricted context windows, quadratic self-attention complexity, and redundant histories that can obscure critical semantic associations.Extracting contextually relevant memory and knowledge is important for multi-turn understanding and generation.Source: §I · p. 1

A. History Organization and Utilization

C. Cognitive Memory Mechanisms

  • Cognitive accounts motivate treating dialogue memory as abstract representations that activate relevant long-term knowledge rather than as verbatim histories.CoLink models external knowledge as long-term semantic memory and historical topic graphs as a memory index for activating context-relevant knowledge.Source: §I · p. 1, C. Cognitive Memory Mechanisms · pp. 2–3

I. INTRODUCTION

  • CoLink links historical topic graphs to external knowledge, activates relevant facts through semantic and structural associations, and compresses history and knowledge into dedicated slots.Its Information Bottleneck objective balances compactness, response prediction, and reconstruction.Source: §I · pp. 1–2

III. THEORETICAL FRAMEWORK AND OBJECTIVE DERIVATION

The paper frames multi-turn response generation as predicting a target sequence from dialogue history, the current query, and potentially auxiliary knowledge. It adopts the Information Bottleneck principle to compress input while retaining information relevant to the response.

A. Problem Definition

  • Multi-turn response generation predicts a target sequence from dialogue history and the current query, with auxiliary knowledge considered for domain-specific conversations.The formulation describes autoregressive prediction of the target sequence.Source: A. Problem Definition · p. 3
  • Assigning precise attention to the most relevant historical and knowledge contexts remains challenging as dialogue progresses amid conversational noise.Source: A. Problem Definition · p. 3

B. Information Bottleneck Preliminary

  • The Information Bottleneck objective minimizes information retained from the input while maximizing information relevant to the target response.The paper adopts this balance to suppress redundancy while preserving core semantics needed for generation.Source: B. Information Bottleneck Preliminary · p. 3

C. Global Context Sense-Making Strategy

CoLink uses separate memory and knowledge compressors under a shared information-theoretic strategy. The memory objective conditions compression on the current query and balances compactness with preservation of response-relevant information.

C. Global Context Sense-Making Strategy

1) Derivation of Memory Compression Objective:

  • History compression is conditioned on the current query so that retained context can be selected for the needs of that turn.For example, a medication-plan query calls for selectively integrating relevant history about symptoms, lifestyle, and examinations.Source: 1) Derivation of Memory Compression… · p. 4
  • The memory objective balances compactness, which penalizes redundant information, with preservation of salient information needed for generation.Minimizing I(Z; H|Q) pushes the compressed representation to depend less on history beyond what is relevant given the query.Source: 1) Derivation of Memory Compression… · p. 4
  • The preservation objective favors compressed representations that retain information useful for predicting the target response.Source: 1) Derivation of Memory Compression… · p. 4

2) Derivation of Knowledge Compression Objective:

CoLink compresses retrieved knowledge into a representation that preserves information relevant to generating the response, conditioned on the current query and compressed dialogue history. The method places this objective within a broader architecture that combines knowledge activation with compact memory and knowledge slots.

2) Derivation of Knowledge Compression Objective:

  • The knowledge compressor learns L as a minimal sufficient representation of retrieved paths K for generating response Y, conditioned on query Q and compressed history Z.Historical information extraction is conditioned on the current query and is independent of external knowledge.Source: 2) Derivation of Knowledge Compression… · p. 4
  • The objective minimizes mutual information between L and raw knowledge K given Q while maximizing L’s mutual information with response Y given Q and Z.The compactness term filters irrelevant knowledge noise; maximizing preservation is equivalent to maximizing the generator’s expected log-likelihood.Source: 2) Derivation of Knowledge Compression… · p. 4
  • The resulting compressed representations Z and L replace the full history H and knowledge K in the response-generation objective.The paper describes Z ≪ H and L ≪ K and reformulates generation using Z and L alongside the current turn and prior responses.Source: 2) Derivation of Knowledge Compression… · p. 4

IV. METHOD

  • CoLink combines topic-graph linking and adaptive knowledge activation with memory and knowledge slots, trained using KL divergence, reconstruction, and prediction losses.The framework overview identifies these components as its architecture for global compression and semantic alignment.Source: §IV · p. 4, Figure 3 · p. 5

A. Association and Activation

CoLink links dialogue-history topic graphs to an external knowledge graph, then activates candidate knowledge paths using semantic relevance and graph structure. Its topic graphs also serve as a dynamic contextual index for activation and compression.

1) Cross-Level Semantic Linking:

  • CoLink clusters dialogue history into themes, extracts relational triples, and aligns topic-graph entities with an external knowledge graph by cosine similarity.The resulting multi-layer semantic bridge links historical topics to external knowledge; the topic graphs expand as dialogue progresses.Source: 1) Cross-Level Semantic Linking · p. 5

2) Adaptive Control of Knowledge Activation:

  • For a new utterance assigned to a historical cluster, CoLink considers two-hop candidate paths built from external-graph entities aligned with that cluster’s topic graph.The path activation score combines semantic relevance to dialogue entities with structural linkage importance.Source: 2) Adaptive Control of Knowledge… · p. 5
  • Path activation incorporates baseline semantic similarity, temporal decay for distant utterances, and associative strength between aligned entities and candidate paths.The aligned entities and path entity sets contribute to the path’s activation level.Source: 2) Adaptive Control of Knowledge… · p. 5
  • Graph-structural relevance is initialized with a Gaussian kernel and iteratively updated by Random Walk with Restart over a subgraph anchored at an aligned entity.The resulting global relevance vector supplies the path-specific associativity used in activation.Source: 2) Adaptive Control of Knowledge… · p. 5
  • The topic graph acts as a dynamic contextual index, while the external knowledge graph provides consolidated long-term storage.Their interaction through context-adaptive activation is described as facilitating semantic integration and retrieval.Source: 2) Adaptive Control of Knowledge… · p. 5

B. Competitive Extraction of Salient Information

CoLink compresses variable-length history and knowledge into fixed-capacity latent slots. Its slot-attention mechanism aggregates information globally and uses the resulting slots as prefix tokens for the language model.

B. Competitive Extraction of Salient Information

  • CoLink abstracts variable-length history and knowledge into fixed, condensed bottleneck slots, using Slot Attention to distill long-range dependencies.For history compression, learnable memory slots instantiate Z; the knowledge compressor follows the same training framework.Source: B. Competitive Extraction of Salient… · pp. 5–6
  • The history compressor encodes a relevant historical cluster as a sequence, then projects inputs and slots into key, value, and query spaces.The projections use learnable linear transformations and Layer Normalization.Source: B. Competitive Extraction of Salient… · p. 6
  • Slots selectively aggregate input features through softmax attention, fuse updates with a GRU, and become the compressed representation after T iterations.The resulting slots are the latent representation Z.Source: B. Competitive Extraction of Salient… · p. 6
  • Each slot attends to all tokens, capturing globally salient information at a compression ratio of |H|/M; memory and knowledge slots are prepended as LLM prefix tokens.The paper says this prefixing enhances perception of critical context.Source: B. Competitive Extraction of Salient… · p. 6

C. Training Objectives and Optimization

CoLink’s compressors are trained with losses for compression, response prediction, and context reconstruction, followed by a two-stage training strategy. The paper also describes MMD, a knowledge-augmented multi-agent medical dialogue dataset, and reports quality assessments and benchmark use.

C. Training Objectives and Optimization

  • The composite training objective combines KL divergence, response prediction, and reconstruction losses, with hyperparameters balancing the terms.KL divergence encourages history compression relative to the query; prediction preserves response utility, while reconstruction addresses possible representation collapse and targets summaries produced via GPT-4.Source: C. Training Objectives and Optimization · p. 6
  • Training first freezes the LLM while optimizing compressors, then uses LoRA to jointly fine-tune the compressors and LLM.The strategy is presented as supporting training stability and semantic consistency with the backbone.Source: C. Training Objectives and Optimization · p. 6

A. Datasets

  • The MMD dataset uses a knowledge-augmented multi-agent debate framework with users, at least two domain experts, and a judge to address issues reported in existing medical dialogue datasets.Experts reason from a shared knowledge base, reflect on retrieved facts, and filter irrelevant knowledge; the judge emits a final response upon consensus.Source: A. Datasets · p. 6
  • The dataset section illustrates dialogue construction through an anonymized electronic health record, expert debate, and a judge’s consensus response.The example includes a user query and retrieved knowledge, followed by expert consensus and a response with follow-up questions.Source: A. Datasets · pp. 6–7, Figure 4 · p. 7
  • MMD contains 2,370 samples across 37 diseases, averages 2.77 debate rounds, and uses a 9:1 train/test split.Three general practitioners evaluated 800 samples; 94% were rated high quality, with Cohen’s Kappa of 0.86, and CMtMedQA was used to assess generalization.Source: A. Datasets · pp. 6–7
  • The paper provides a dataset-statistics table for MMD and CMtMedQA, but the supplied passage gives no individual table values.The prose identifies CMtMedQA as the public benchmark used for generalization assessment.Source: Table I · p. 7, A. Datasets · p. 7

B. Baselines

CoLink is evaluated against full-history and summary baselines, structured retrieval systems, and local-token attention methods, with implementations following their original papers.

B. Baselines

  • The comparison covers full-history concatenation and summaries of all history as standard baselines.Baseline implementations follow their original papers, including their information organization schemes and retrieval depths.Source: B. Baselines · p. 7
  • Structured retrieval baselines include memory, tree, and graph methods, plus NV-Embed for dense retrieval.The graph-based methods use OpenAI encoders for semantic-similarity retrieval.Source: B. Baselines · p. 7
  • Local-token attention comparisons use StreamLLM and CCA.Source: B. Baselines · p. 7
  • The hyperparameter settings are summarized in Table II.Source: Table II · p. 7

C. Experimental Details

Experiments compare CoLink with context strategies and evaluate its compression, transfer, and cross-domain performance. The reported results show gains over retrieval and local-attention baselines, including under strong compression.

1) Comparison Results:

  • 2.59% average gain over structured-retrieval and local-token-attention baselines across two datasets under 32× compression, while matching or exceeding Full History.The comparison uses Qwen2.5-1.5B as the backbone.Source: Table III · p. 8, 1) Comparison Results · p. 8
  • Rigid graph and tree retrieval can degrade performance, while chunk-based retrieval offers moderate improvements; summary histories lose information during condensation.The authors contrast these approaches with gradient-optimized slots for retaining holistic semantics.Source: 1) Comparison Results · p. 8
  • CoLink uses only 3.1% of the original context length, and its latent-space compression avoids costly entity extraction and graph construction for O(1) retrieval.The resource-consumption and compression-ratio comparison is summarized in Table IV.Source: 1) Comparison Results · p. 8, Table IV · p. 9

2) Generalizability of CoLink:

  • CoLink yields consistent gains over vanilla Llama-3.2-3B after memory-slot training on MMD, supporting transfer across backbone architectures.Table V reports transfer performance using first-stage memory-slot training.Source: 2) Generalizability of CoLink · p. 8, Table V · p. 9
  • 8.16% improvement over full-history concatenation on UltraData shows cross-domain performance beyond medical dialogue using memory slots alone.External knowledge is not natively required for this open-domain task.Source: 2) Generalizability of CoLink · p. 8, Table VI · p. 9
  • An MMD-trained model transferred to CMtMedQA without further finetuning consistently outperforms standard baselines across the different clinical dialogue contexts.Source: 2) Generalizability of CoLink · p. 8

3) Effectiveness Analysis:

CoLink’s competitive slots are analyzed as an alternative to sequential summarization: token contributions compete across slots, while gradients follow decoupled paths. Human and LLM-as-a-judge ratings also favor CoLink, particularly for contextual coherence.

3) Effectiveness Analysis:

  • Each input token allocates its weight across all memory slots, whereas standard summarization aggregates input tokens through a weighted average that can flatten as sequence length grows.The authors characterize the resulting effect as attention dilution and feature blurring.Source: 3) Effectiveness Analysis · p. 9
  • CoLink’s competitive routing gives loss gradients orthogonal, decoupled paths, while sequential summaries force gradients through a shared representation that can cause semantic interference.The analysis uses a first-order Taylor expansion of loss perturbation with respect to attention.Source: 3) Effectiveness Analysis · p. 9
  • CoLink’s memory slots show higher attention mass and gradient saliency toward the query and next-token prediction than summary history, excluding structural attention sinks.Figure 5 visualizes this comparison for a dialogue sample.Source: 3) Effectiveness Analysis · p. 9, Figure 5 · p. 10

4) Human and LLM-as-a-Judge Evaluation:

E. Ablation Studies

Ablations examine CoLink’s components, objective terms, and input encoding. Memory slots are especially consequential; knowledge-slot gains depend on the external knowledge graph, while encoding alignment and loss choices affect performance and training behavior.

E. Ablation Studies

  • The ablation program also compares compressing retrieved topic clusters with compressing full history; Table XIV identifies this comparison but the supplied passages do not state its outcome.Source: E. Ablation Studies · p. 10, Table XIV · p. 11

1) Ablation of Model Components:

  • Removing memory slots causes severe degradation, whereas knowledge activation and compression gains are relatively marginal with the misaligned medical knowledge graphs used for MMD and CMtMedQA.On GMD, which has a precisely aligned knowledge graph, IB-driven knowledge slots have a positive impact on diagnostic accuracy.Source: 1) Ablation of Model Components · p. 10, Table X · p. 10

2) Sensitivity to Loss Configuration:

3) Analysis of Input Encoding Space:

  • Using the LLM’s final-layer representation for compression input yields noticeable performance gains over an independent embedding layer or the LLM’s first hidden layer.The authors hypothesize that final-layer representations align compressed slots with the generator’s semantic manifold.Source: 3) Analysis of Input Encoding Space · p. 11, Table XIII · p. 11

4) Impact of History Clustering Strategies:

CoLink compresses dialogue history while retaining global context, and the reported evaluations include comparisons with full history, slot and clustering analyses, and a qualitative case study. Its conclusion also reports robustness and generalization, while noting weaker performance on precise lexical retrieval.

4) Impact of History Clustering Strategies:

  • Compressing retrieved topic clusters yields negligible performance variance from compressing full history in first-stage memory-slot training.The authors attribute this parity to noise filtering and the high information density of domain-specific dialogues, while noting clustering remains computationally advantageous.Source: 4) Impact of History Clustering… · p. 11

1) Impact of Slot Number:

  • CoLink’s performance varies non-monotonically with memory-slot count: too few slots cause semantic loss, while too many dilute information density and may leave slots underused or noisy.The study also reports that dynamic slots often collapse by masking most slots or retaining all of them, harming performance and leaving stable adaptive selection open.Source: 1) Impact of Slot Number · pp. 11–12

2) Case Study:

  • In the case study, CoLink aligns better with Full History than MemTree and GraphRAG, which rely on the immediate query and miss valuable historical context.The authors attribute CoLink’s precise, actionable medical advice to soft memory slots synthesizing global context without structural fragmentation.Source: Figure 7 · p. 12, 2) Case Study · p. 12

VI. CONCLUSION AND FUTURE WORKS

  • At 3.125% of the original context length, CoLink demonstrates superiority over strong baselines and matches or exceeds full visible history.The conclusion also describes its soft, learnable slots as preserving continuous semantic flow and avoiding the fragmentation of rigid approaches.Source: §VI · p. 12
  • Ablations and human evaluation support CoLink’s robustness, model-agnostic nature, and generalization capability.Source: §VI · p. 12
  • On MemDaily, CoLink underperforms text-based prompt baselines on precise lexical retrieval, including recall of specific entities such as phone numbers and job titles.The authors identify improving fidelity for precise information retrieval as future work.Source: §VI · p. 12
Loading 2610.34335v1…