MemFly: On-the-Fly Memory Optimization via Information Bottleneck

Zhenyuan Zhang; Xianzhang Jia; Zhiqin Yang; Zhenbo Song; Wei Xue; Sirui Han; Yike Guo

Source-linked AI summary

MemFly: On-the-Fly Memory Optimization via Information Bottleneck

Zhenyuan Zhang, Xianzhang Jia, Zhiqin Yang, Zhenbo Song, Wei Xue, Sirui Han, Yike Guo

arXiv:2602.07885v2Feb 8, 2026cs.AIcs.LG

TL;DR

LLM agents need long-term memory that balances efficient compression with precise retrieval for complex, extended reasoning. MemFly addresses this trade-off with information-bottleneck memory evolution and hybrid retrieval, achieving consistent improvements over state-of-the-art baselines across diverse backbone models.

Problem
Existing long-term memory frameworks struggle to compress redundant information efficiently while maintaining precise retrieval for downstream tasks.
Method
MemFly uses information-bottleneck optimization, a stratified Note-Keyword-Topic hierarchy, and hybrid semantic-symbolic-topological retrieval with iterative refinement.
Results
MEMFLY achieves consistent improvements across diverse backbone models, including 38.62% F1 on Qwen3-8B, 5.86 points above A-MEM.
Takeaways & Limitations
The results support structured memory organization as an effective approach that generalizes across heterogeneous backbone architectures.
Takeaways & Limitations
The current implementation prioritizes memory quality over construction speed, introducing moderate computational overhead, while multimodal and domain-specific evaluation remains open.

Abstract

from arXiv · show

Long-term memory enables large language model agents to tackle complex tasks through historical interactions. However, existing frameworks encounter a fundamental dilemma between compressing redundant information efficiently and maintaining precise retrieval for downstream tasks. To bridge this gap, we propose MemFly, a framework grounded in information bottleneck principles that facilitates on-the-fly memory evolution for LLMs. Our approach minimizes compression entropy while maximizing relevance entropy via a gradient-free optimizer, constructing a stratified memory structure for efficient storage. To fully leverage MemFly, we develop a hybrid retrieval mechanism that seamlessly integrates semantic, symbolic, and topological pathways, incorporating iterative refinement to handle complex multi-hop queries. Comprehensive experiments demonstrate that MemFly substantially outperforms state-of-the-art baselines in memory coherence, response fidelity, and accuracy.

1. Introduction

MemFly frames long-term agent memory as an online information bottleneck problem, addressing redundancy–fidelity tensions through on-the-fly optimization and structured retrieval. It combines gradient-free memory construction with tri-pathway retrieval and iterative refinement for complex reasoning.

Motivation: Long-term LLM agents need memory that tracks evolving entities, resolves temporal dependencies, and synthesizes evidence across conversations.
Limitations of Existing Frameworks: Retrieval-centric systems preserve details but accumulate redundancy and retrieval noise, while memory-augmented systems rely on LLM-driven summarization.
Memory Construction: A gradient-free optimizer extends Agglomerative Information Bottleneck principles to online memory construction through LLM-based semantic assessment.
Retrieval: Tri-pathway retrieval integrates semantic Topics, symbolic Keywords, and topological associative links, with iterative refinement for multi-hop queries.
MemFly Framework: MemFly formalizes agentic memory as an Online Information Bottleneck problem that unifies entropy accumulation and fidelity loss.
Memory Structure: MemFly uses a Note-Keyword-Topic hierarchy grounded in double clustering to preserve evidence structure.

2. Related Work

Related work has advanced retrieval through iterative refinement and graph-based organization, while separate memory-construction methods structure, compress, and evolve long-term interaction history. However, retrieval-centric systems generally treat memory as a passive index and can struggle with multi-hop evidence synthesis.

Retrieval-Centric Methods: RAG has evolved from passive retrieve-then-read pipelines toward active, iterative workflows using feedback for query refinement and hallucination filtering.These approaches optimize retrieval during inference rather than restructuring stored memory.
Retrieval-Centric Methods: Graph-based approaches organize knowledge for local retrieval and global summarization, while MemWalker uses explicit traversal to maintain structured memory representations.These methods extend retrieval beyond document-level matching through graph structure and traversal.
Limitations: Inference-time retrieval systems treat memory as a passive index and rely on query-centric embedding similarity, making them vulnerable to vector dilution during multi-hop evidence synthesis.Their primary intervention is refinement of the read path for specific queries, not evolution of the underlying memory structure.
Memory Construction: Memory-construction systems such as MemGPT and HiAgent use tiered storage hierarchies to organize interaction history across active working memory and archival storage.Their swapping mechanisms aim to emulate infinite retention.
Memory Construction: Other systems model biological or associative memory through forgetting curves, Zettelkasten-style linking, and user-centric profiling to support autonomous knowledge evolution.MemoryBank modulates information decay, while A-MEM and O-Mem promote associative updates; these methods mitigate the Goldfish Effect.

3. The MEMFLY Framework · 3.1. Problem Formulation · 3.2. Structural Prior

MEMFLY formulates long-term memory construction as an Information Bottleneck optimization that compresses interaction streams into relevance-maximizing states, realized online as dynamic graphs. A stratified Note-Keyword-Topic hierarchy supplies computational structure for fidelity, symbolic anchoring, and navigable retrieval.

3. The MEMFLY Framework: MEMFLY treats memory as a dynamic Information Bottleneck channel that compresses continuous interaction streams into compact, relevance-maximizing representations.The framework optimizes memory construction rather than maintaining a static repository.
3.1. Problem Formulation: The memory state M_t is instantiated as a dynamic graph G_t = (V_t, E_t, Φ_t), with nodes, topological connections, dense embeddings, and textual content.This graph realization supports computational memory operations while representing the agent’s evolving state.
3.1. Problem Formulation: The objective minimizes information retained from raw inputs while maximizing mutual information between memory state M_t and future reasoning tasks Y.Compression removes redundancy and irrelevant details, whereas relevance preserves critical downstream evidence.
3.1. Problem Formulation: Because future tasks Y are unobserved, MEMFLY uses structural proxies for semantic coherence and global navigability to estimate relevance.Keyword co-occurrence patterns capture coherence, while Topic hierarchy and associative links capture evidence-chain accessibility.
3.1. Problem Formulation: MEMFLY approximates intractable global optimization through online greedy agglomeration, making locally optimal memory decisions at each time step.The transition M_t+1 ← T(M_t, x_t) is governed by a policy π that minimizes incremental Lagrangian cost.
3.1. Problem Formulation: An LLM acts as a gradient-free policy that semantically assesses memory-unit redundancy and complementarity to approximate unavailable JS-divergence decisions.The method uses scores s_red(n_t, n_i) and s_comp(n_t, n_i), with higher redundancy indicating lower JS-divergence.
3.2. Structural Prior: To mitigate sparsity and noise from high-dimensional embeddings, MEMFLY imposes a Double Clustering-inspired structural prior.The prior organizes memory into a stratified Note-Keyword-Topic hierarchy.
3.2. Structural Prior: The hierarchy assigns Notes to fidelity, Keywords to symbolic anchoring, and Topics to navigation, with Topics enabling O(1) macro-semantic localization.Keywords stabilize semantic proximity through shared symbolic substructures, while Topics aggregate co-occurring keywords into navigable regions.

3.3. Memory Construction

MemFly constructs memory online through ingestion, gated structural updates, and topic evolution. It transforms inputs into structured notes, selectively merges, links, or appends information, and periodically restructures topics for macro-navigability.

Online Memory Construction: Memory construction is an online agglomerative process with ingestion, gated structural update, and topic evolution stages.This computation-on-construction mechanism makes memory updates tractable during streaming input.
Ingestion: Raw inputs are transformed into structured notes containing raw content, denoised context, embeddings, and extracted keywords.The transformation improves signal-to-noise ratio and increases I(nt; Y) relative to I(xt; Y).
Gated Structural Update: Before consolidation, dual sparse-dense indices retrieve a candidate neighborhood, after which an LLM scores redundancy and complementarity for each candidate pair.The scores lie in [0, 1], with redundancy measuring semantic overlap and complementarity measuring logical or topical connection between distinct information.
Gated Structural Update: Thresholded structural updates merge redundant units, link complementary units, or append novel units as autonomous memory elements.Merge reduces |Vt| while preserving distinct information; Link encodes conditional structure for multi-hop reasoning without increasing representational redundancy.
Topic Evolution: Periodic restructuring of the Keyword co-occurrence graph uses the Leiden algorithm to maintain O(1) macro-navigability.The topic layer is formalized as constrained graph partitioning over Gkw.

3.4. Memory Retrieval

MemFly retrieves memory through parallel semantic, symbolic, and topological pathways, then fuses their evidence with Reciprocal Rank Fusion. For complex queries, iterative refinement synthesizes sub-queries and repeats retrieval until the evidence is sufficient or reaches the iteration limit.

Hybrid retrieval: MemFly uses tri-pathway hybrid retrieval that decomposes queries into complementary semantic signals and traverses the memory graph in parallel.The pathways are macro-semantic localization, micro-symbolic anchoring, and topological expansion.
Hybrid retrieval: Macro-semantic localization finds top-Ktopic relevant Topic centroids and retrieves notes through hierarchical traversal.It uses the topical embedding htopic and centroid embeddings µC.
Hybrid retrieval: Micro-symbolic anchoring matches query entities against the keyword index to retrieve notes through keyword membership.This pathway targets precision for entity-centric queries using keyword embeddings ek′.
Hybrid retrieval: Topological expansion follows ERELATED edges from anchor nodes to retrieve logically related evidence that may be vectorially distant.This pathway supports connectivity for multi-hop reasoning.
Evidence fusion: Reciprocal Rank Fusion combines candidates across pathways, prioritizing evidence that consistently ranks highly without score normalization.The fused evidence pool is constrained by the predefined budget Kfinal.
Iterative Evidence Refinement: Iterative Evidence Refinement evaluates evidence completeness, synthesizes refined sub-queries for identified gaps, and repeats tri-pathway retrieval until sufficiency or Imax is reached.The evidence pool is updated after each re-execution of the retrieval function R(q).

4. Experiments

MemFly is evaluated on the LoCoMo benchmark across five reasoning categories using F1 and BLEU-1, with Recall and Hit Rate added for ablations. Across closed- and open-source backbones, it achieves the strongest average results, while ablations show the importance of gated memory updates and complementary retrieval pathways.

Evaluation Setup: MemFly is evaluated on LoCoMo, which contains long-horizon conversations with interleaved topics and evolving entity states across five reasoning categories.The categories are Multi-Hop, Temporal, Open Domain, Single Hop, and Adversarial.
Evaluation Setup: The evaluation uses F1 Score for token-level overlap and precision, BLEU-1 for lexical fidelity, and ablation metrics Recall and Hit Rate.Recall measures the proportion of ground-truth evidence retrieved, while Hit Rate records whether any relevant evidence appears among candidates.
Overall Performance: MEMFLY achieves the highest average F1 and BLEU-1 across four backbone models, including 43.76% F1 on GPT-4o-mini and 44.39% on GPT-4o.It outperforms the strongest baseline by 1.79 and 0.27 points on those closed-source models, and reaches 38.62% F1 on Qwen3-8B, 5.86 points above A-MEM.
Category-wise Analysis: MEMFLY records its largest category gain on Open Domain queries, achieving 25.74% F1 on GPT-4o versus 17.73% for MEM-0.It also achieves 42.09% and 42.25% F1 on Single Hop tasks across the two Qwen models.
Ablation Studies: Removing gated memory updates reduces average F1 from 38.62% to 27.97% and Recall from 62.22% to 42.11%, while removing Topic, Keyword, or iterative refinement yields 36.79%, 32.69%, and 32.94% F1, respectively.The w/o Denoise construction variant achieves 36.07% F1, indicating a smaller and relatively stable degradation than removing the update mechanism.

5. Conclusion

MEMFLY formulates agentic long-term memory as an Information Bottleneck problem, using gradient-free consolidation, a stratified hierarchy, and tri-pathway retrieval with iterative refinement. Its current implementation prioritizes memory quality over construction speed, while multimodal and domain-specific evaluation remains future work.

Conclusion: MEMFLY formulates agentic long-term memory as an Information Bottleneck problem and consolidates redundant information while preserving task-relevant evidence.It uses an LLM-based gradient-free optimizer and a stratified Note-Keyword-Topic hierarchy.
Conclusion: Tri-pathway retrieval with iterative refinement exploits the stratified memory structure for complex reasoning.The retrieval mechanism integrates the hierarchy during complex reasoning.
Limitations: The current implementation prioritizes memory quality over construction speed, introducing moderate computational overhead.This trade-off is identified as a limitation of the current implementation.
Limitations: Extending evaluation to multimodal and domain-specific scenarios remains an avenue for future investigation.The limitation concerns the scope of the current evaluation.

A. Prompting Templates

MEMFLY’s appendix specifies structured-JSON prompting templates for semantic ingestion, gated structural updates, and iterative evidence refinement. These prompts support dual symbolic-semantic access, redundancy-aware graph operations, and targeted multi-hop evidence expansion.

A. Prompting Templates: All MEMFLY prompts are designed to elicit structured JSON outputs for reliable parsing.
A.1.1. SEMANTIC INGESTION PROMPT: The semantic ingestion prompt transforms raw conversational input x_t into a structured Note n_t = (r_t, c_t, h_t, K_t).
A.1.1. SEMANTIC INGESTION PROMPT: Extracted keywords anchor the existing Keyword index K, while encoded context h_t = Embed(c_t) enables symbolic and semantic retrieval pathways.
A.1.2. GATED STRUCTURAL UPDATE PROMPT: The gated structural update prompt evaluates each candidate Note using redundancy and complementarity scores to select Merge, Link, or Append.
A.1.2. GATED STRUCTURAL UPDATE PROMPT: It classifies relations as SUPPORTS, CONFLICTS, or RELATED TO, assigns connection strength from 0.0–1.0, and returns strictly valid JSON.
A.1.2. GATED STRUCTURAL UPDATE PROMPT: When merging is triggered, the generated unified context preserves distinct information from both units while eliminating redundancy.
A.2.2. ITERATIVE EVIDENCE REFINEMENT PROMPTS: The Iterative Evidence Refinement prompts evaluate evidence sufficiency, identify missing information, and generate targeted single sub-queries for subsequent retrieval.

A.2.1. QUERY INTENT ANALYSIS PROMPT

The query-intent prompt analyzes each user query by identifying its taxonomy topic, extracting canonical entities, and detecting temporal intent. Its structured outputs map to semantic and symbolic retrieval pathways through topic and keyword embeddings.

Query Intent Analysis Prompt: The prompt identifies the query’s target taxonomy category, extracts key entities, and detects time-related intent.It frames the topic as the subject itself rather than the user’s intent.
Query Intent Analysis Prompt: The topic description must be a strict noun phrase naming the category, limited to fewer than 8 words.The prompt excludes formulations such as “how to...” or “techniques for...”.
Query Intent Analysis Prompt: The keyword output contains 3-5 core entities, technical terms, or concepts converted to canonical singular forms.The expected output is a JSON object with topic_desc and keywords fields.
Mapping to Paper Notation: The topic description is embedded as htopic for Pathway 1 topic matching, while keywords form Hkeys for Pathway 2 keyword matching.These representations drive macro-semantic localization and micro-symbolic anchoring, respectively.

B. Dataset Statistics

The LoCoMo benchmark spans five reasoning categories targeting distinct memory capabilities, with an imbalanced distribution addressed through sample-size-weighted average scores. These categories cover multi-hop synthesis, temporal reasoning, open-domain retrieval, single-hop matching, and adversarial distractor handling.

Category Coverage: The benchmark evaluates Multi-Hop, Temporal, Open Domain, Single Hop, and Adversarial reasoning categories designed to test distinct memory capabilities.Multi-Hop synthesizes evidence across memory units; Temporal tests time-dependent reasoning; Open Domain evaluates conversational knowledge retrieval; Single Hop assesses direct entity matching; and Adversarial introduces distractors.
Category Distribution: Single Hop comprises 42.3% of samples, while Open Domain comprises 4.8%, making them the largest and smallest categories, respectively.The category distribution is explicitly described as imbalanced.
Evaluation Aggregation: Average scores in Tables 1 and 2 use weighted averages based on category sample sizes to account for distributional imbalance.This weighting makes larger categories contribute proportionally to overall evaluation.

C. Hyperparameter Settings

MemFly’s hyperparameters are divided between memory construction and retrieval, covering structural updates, multi-path evidence selection, and iterative refinement. Tuned on held-out validation data, the settings remain robust to moderate variation, with graceful degradation within ±20%.

Hyperparameter Settings: The hyperparameters are organized into memory construction and memory retrieval phases.Table 5 summarizes the settings used across both phases.
Memory Construction: Memory construction uses merge threshold τm = 0.7 and link threshold τl = 0.5 for gated structural updates.The higher merge threshold consolidates highly redundant information, while the lower link threshold captures complementary relationships.
Memory Retrieval: Memory retrieval sets Ktopic = 3, Kkey = 10, Kfinal = 20, and Imax = 3 to balance coverage, anchoring, context size, and refinement effort.These parameters support exploration of relevant topic clusters, entity-centric queries, bounded evidence, and limited retrieval loops.
Validation and Robustness: Performance degrades gracefully when hyperparameters deviate within ±20% of the reported settings.All hyperparameters were tuned on a held-out validation set, and the framework was relatively robust to moderate variations.