PaperlayerView exact revision on arXiv

Source-linked AI summary

From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents

Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, Min Zhang, Shu-Tao Xia

arXiv:2603.01455v3cs.CVcs.AIcs.CLcs.IRcs.MM

TL;DR

Long-horizon video agents struggle to retain both visual detail and efficient semantic context because existing memory systems are either redundant or lossy. MM-Mem organizes multimodal memory into a Fuzzy-Trace Theory-inspired pyramid and controls its construction and retrieval with SIB-GRPO and entropy-driven drilling. Across 4 benchmarks, it achieves state-of-the-art performance and robust generalization across offline and streaming settings.

  • Problem

    Long-horizon video understanding is limited by context windows and static memory mechanisms, while vision-centric and text-centric approaches respectively incur redundancy or lose visual detail.

  • Method

    MM-Mem uses Sensory Buffer, Episodic Stream, and Symbolic Schema layers, with SIB-GRPO for redundancy-aware compression and entropy-driven top-down retrieval.

  • Results

    MM-Mem achieves state-of-the-art performance and robust generalization across 4 benchmarks covering offline and streaming long-video understanding.

  • Takeaways & Limitations

    MM-Mem structurally separates visual verbatim details from semantic gist schemas to support precise detail verification without the cognitive overload typical of vision-centric methods.

  • Takeaways & Limitations

    The hierarchical architecture incurs higher construction-phase computational cost than flat, compression-heavy models, and the system depends on upstream vision encoders.

Abstract

from arXiv · show

While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy. Extensive experiments across 4 benchmarks confirm that MM-Mem achieves state-of-the-art performance on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code and associated configurations are publicly available at https://github.com/EliSpectre/MM-Mem.

1. Introduction

Long-horizon multimodal agents must balance fine-grained visual evidence with efficient semantic memory. MM-Mem addresses this through a Fuzzy-Trace Theory-inspired pyramid, adaptive information-bottleneck construction, and entropy-driven retrieval, achieving strong results across offline and streaming benchmarks.

  • Human memory separates abstract gist from fine-grained verbatim traces, retaining specific visual evidence while managing long-term semantic context.
  • Vision-centric methods incur redundancy and latency from dense visual memories, whereas text-centric methods lose visual cues through lossy captioning.
  • MM-Mem organizes memory bottom-up across a Sensory Buffer, Episodic Stream, and Symbolic Schema, progressively transforming perceptual signals into cognitive knowledge.
  • SIB-GRPO uses an Information Bottleneck objective to balance semantic preservation against redundancy reduction during adaptive memory construction.
  • Entropy-driven retrieval starts from abstract schemas and drills down to fine-grained details when uncertainty is high, enabling targeted verification.
  • MM-Mem achieves state-of-the-art performance across 4 benchmarks spanning offline long-video understanding and online streaming settings.

2. Related Work

Related work includes multimodal memory systems, LLM-agent memory mechanisms, and multimodal-agent frameworks, but existing approaches leave efficiency, cross-modal alignment, and open-ended generalization unresolved. MM-Mem combines textual coarse localization with visual fine-grained retrieval to address this balance.

  • Vision-centric long-video methods improve visual coverage through dense sampling or token compression but can incur high computational cost.
  • MM-Mem unifies high-level textual memory for coarse localization with low-level visual memory for fine-grained retrieval.
  • LLM-agent memory research spans hierarchical caches, forgetting-curve management, associative links, and reinforcement-learning-based control.
  • Existing multimodal-agent systems often rely on predefined structures and fixed workflows, which may constrain generalization in open-ended long-horizon environments.

3. Method

MM-Mem builds a three-level multimodal memory pyramid that progressively distills visual evidence into compact episodic traces and symbolic knowledge. SIB-GRPO formulates memory construction as semantic compression, while multimodal grounding preserves access to concrete visual details.

  • Multimodal distillation: The construction process preserves integrated multimodal representations while progressively shifting from vision-dominant verbatim traces toward text-dominant semantic gist.This crossmodal progression implements the paper’s Fuzzy-Trace-inspired distinction without isolating modalities into separate tracks.
  • Sensory Buffer: The Sensory Buffer segments videos into salient short sub-clips and stores visual representations, associated text traces, and temporal locations.Text remains auxiliary at this vision-dominant layer, providing semantic anchors for indexing dense visual details.
  • Episodic Stream: The Episodic Stream uses ADD_NEW, MERGE, and DISCARD decisions to consolidate sensory entries into a compact chronological sequence.The resulting stream summarizes retained visual representations into event-level episodic units.
  • Symbolic Schema: The Symbolic Schema forms a knowledge graph whose grounding edges link text-dominant concepts back to episodic units containing visual evidence.These pointers support high-level reasoning while enabling drill-down to concrete details.
  • Pyramidal memory structure: MM-Mem organizes memory into a Sensory Buffer, Episodic Stream, and Symbolic Schema, transforming perceptual signals into increasingly abstract knowledge.The Sensory Buffer retains fine-grained evidence, the Episodic Stream compresses events, and the Symbolic Schema consolidates structured knowledge over longer time scales.
  • Semantic compression: SIB-GRPO trains a memory manager to retain task-relevant semantics while discarding redundant, low-novelty details during Sensory-to-Episodic conversion.The stochastic encoder maps sensory content and the latest episodic node to an episodic representation, while the objective balances prediction, compression, length, and teacher-prior regularization.

4. Experiment

MM-Mem is evaluated across offline, streaming, and egocentric long-video settings, with comparisons spanning accuracy, ablations, memory topology, and efficiency. Results show consistent gains over prior systems, while qualitative analyses illustrate hierarchical retrieval and multimodal memory organization.

  • Long Video Understanding: MM-Mem surpasses prior agent systems by 5.1% relatively on Video-MME and 7.1% on MLVU M-Avg.It also surpasses compared open-source MLLMs and remains competitive with proprietary models such as Gemini 1.5 Pro.
  • Online Streaming Video Understanding: MM-Mem improves over Flash-VStream by 5.9% in Accuracy and 5.2% in Score on VStream-QA-Ego.
  • Egocentric Long Video Understanding: MM-Mem achieves 30.28% accuracy on HD-EPIC++, exceeding Qwen3-VL-8B by 4.40 points.It also surpasses LLaVA-Video-7B and VideoLLaMA 3-7B by 4.91 and 9.92 points, respectively.
  • Effectiveness of SIB-GRPO and Pyramid Memory: Removing SIB-GRPO and then pyramid memory degrades performance, with the largest losses on Long and Overall splits.The ablation attributes complementary roles to semantic consolidation and organization across temporal and semantic granularities.
  • Topology of the Cognitive Memory Space: Memory topology separates domain-specific visual details in the Sensory Buffer and forms semantic clusters in the Episodic Stream.
  • Efficiency and Deployment Analysis: MM-Mem requires 5.35s per minute for inference, 19.54s per minute for memory construction, and 17.8 GB peak VRAM on an NVIDIA A100.Offline construction can be amortized across multiple queries for the same video, while compact textual memory reduces deployment cost.
  • Qualitative Analysis: Coarse-to-fine retrieval starts with Symbolic Schema reasoning, drills into Episodic Stream evidence, and reaches the Sensory Buffer for precise verification.The examples cover temporal reasoning, action recognition, and detail-sensitive retrieval.

5. Conclusion

MM-Mem is a Fuzzy-Trace-Theory-grounded pyramidal multimodal memory framework that separates verbatim visual details from gist semantic schemas. It combines information-theoretic compression with entropy-driven retrieval, while experiments report state-of-the-art performance and robust generalization; the authors identify computational, perceptual, and unsupervised-scenario limitations.

  • Conclusion: MM-Mem structurally decouples verbatim visual details from gist semantic schemas to bridge fine-grained perception and high-level cognition.The framework uses SIB-GRPO for dynamic, redundancy-aware compression and entropy-driven retrieval from schemas to sensory details.
  • Conclusion: The framework achieves state-of-the-art performance and robust generalization in extensive experiments.The conclusion presents MM-Mem as cognitive infrastructure for long-horizon autonomous agents.
  • Limitations: The hierarchical construction phase incurs higher computational cost than flat, compression-heavy models.The authors suggest asynchronous processing, parallelization, and future pipeline distillation to reduce latency for edge deployment.
  • Limitations: MM-Mem depends on upstream vision encoders and captioners, and stronger end-to-end perception backbones may improve robustness against visual artifacts.
  • Limitations: The current memory manager uses task-driven reinforcement learning, while future work targets unsupervised scenarios and dynamic memory updating and forgetting.
  • Ethics Statement: Real-world deployment requires privacy safeguards, bias monitoring, diverse benchmarks, and caution against high-stakes use.The ethics discussion recommends data minimization, local storage, rigorous access controls, and monitoring of selective memory construction.

A. Proof of the Variational IB Bounds

The variational IB derivation models a stochastic memory manager that maps sensory input to episodic representations, then uses variational decoder and prior distributions to obtain tractable bounds. The episodic stream is determined by discrete update actions under fixed transition rules.

  • Variational IB formulation: The stochastic encoder pθ(m ∣x) maps sensory input X to episodic representation M under the Markov structure Y ↔X ↔M.
  • Variational IB formulation: A variational decoder qϕ(y ∣m) approximates pθ(y ∣m), while a variational prior r(m) approximates the intractable marginal pθ(m).
  • Proof steps: The proof rewrites mutual information, applies a variational lower bound using KL non-negativity, and matches the resulting expression to the training sampling form.
  • Variational objective: Because H(Y) is constant with respect to (θ, ϕ), minimizing I(X; M) −βI(M; Y) is equivalent to maximizing the corresponding variational objective.
  • Episodic-stream determinism: For fixed sensory buffers and update rules, the action sequence uniquely determines the episodic stream through deterministic transitions.The action space contains ADD_NEW, MERGE, and DISCARD operations.

B. Dataset and Benchmark Details

The benchmark section establishes a fixed HD-EPIC split, introduces HD-EPIC++ with dense procedural annotations, and constructs a hard-negative multiple-choice VQA benchmark. Its design targets long-horizon procedural, temporal, spatial, and multimodal reasoning.

  • Motivation and split protocol: The HD-EPIC videos are reorganized into 105 training and 51 test videos, totaling 156 videos, with SIB-GRPO trained on the training split.Benchmark performance is reported on the held-out test split.
  • Dataset overview: HD-EPIC++ extends HD-EPIC with denser, fine-grained annotations for recognition, temporal grounding, and structured reasoning over procedures, entities, and interactions.
  • Dense annotations: The dataset covers seven annotation types: Recipe, Ingredient, Nutrition, Fine-Grained Action, 3D Perception, Object Motion, and Gaze.
  • VQA benchmark construction: The VQA benchmark uses 30 question prototypes instantiated into 26,650 five-way multiple-choice questions with hard negatives sampled from annotations.
  • Scalability and intended impact: HD-EPIC++ is designed to evaluate long-horizon procedural understanding, entity and state tracking over time, and grounded multimodal reasoning under realistic distractors.The dense annotations could support approximately 100,000 unique questions through additional instantiations.
  • Sensory evidence construction: Key sub-clips preserve salient temporal evidence while minimum-separation filtering suppresses near-duplicates and excessive overlap.Each retained sub-clip is paired with visual and textual traces and a temporal location.

D. Implementation Details

Implementation uses SIB-GRPO fine-tuning with LoRA adaptation and follows specified benchmark evaluation protocols. VStream-QA uses an automatic judge that reports both correctness accuracy and average semantic scores.

  • Training configuration: The model is fine-tuned with SIB-GRPO on HD-EPIC++ and adapted using LoRA within the SWIFT framework.Training and inference hyperparameters are summarized in Table 5.
  • Benchmarks: HD-EPIC++ is an egocentric long-horizon kitchen benchmark covering action perception, temporal reasoning, 3D spatial understanding, object motion, gaze, and VQA.
  • VStream-QA evaluation: VStream-QA evaluation follows the Flash-VStream protocol and uses GPT-4o-mini as an automatic judge for open-ended questions.
  • Answer Agent: The Answer Agent selects the best option for each video multiple-choice question and outputs only its corresponding letter from A–D.
  • VStream-QA evaluation: VStream-QA reports Accuracy for judged correctness and Score as the average assigned value from 0 to 5.
  • Evaluation prompts: The evaluation agent accepts meaningful semantic matches, including synonyms and paraphrases, when judging generated answers against ground truth.
  • Evaluation prompts: The VStream-QA judge returns a yes/no correctness label and an integer similarity score for each video question–answer pair.

F.1. Ablation Studies

Ablations show that MM-Mem’s visual memory, hierarchical layers, and SIB-GRPO each contribute to performance, with visual evidence especially important for long videos.

  • Ablation summary: Removing any component consistently degrades performance, indicating that each module contributes positively.
  • Visual vs. Text Memory: Removing Visual Memory drops Overall from 78.1 to 70.9 and Long from 75.7 to 66.1, a -9.6 Long decrease.Removing Text Memory causes a smaller Overall drop from 78.1 to 75.8, while visual evidence remains especially important for long videos.
  • Training / Memory Management: Removing SIB-GRPO reduces Overall to 75.3 (-2.8) and Long to 71.9 (-3.8), indicating greater benefit as temporal context grows.
  • Hierarchical Memory Components: Removing Sensor reduces Overall to 73.6 and Long to 70.2, while removing Episodic reduces them to 75.0 and 71.2, respectively.Removing Symbolic produces a smaller Overall drop to 76.4, but remains helpful for higher-level temporal and relational reasoning.
  • Full model: 78.1 overall accuracy is achieved by full MM-Mem, with 82.8, 75.8, and 75.7 on Short, Medium, and Long subsets.The reported results include uncertainty estimates of ±0.2, ±0.2, ±0.3, and ±0.2, respectively.

F.3. Results on VSteam-QA

MM-Mem performs strongly on streaming egocentric videos but is weaker on movie-style streaming, indicating split-dependent effectiveness.

  • VS-Ego: MM-Mem achieves the highest VS-Ego Accuracy of 62.5 and Score of 4.1, exceeding Flash-VStream by +3.5 and +0.2.
  • VS-Movie: On VS-Movie, MM-Mem reaches 52.1 Accuracy and 3.2 Score, below the strongest baselines at approximately 54.0–56.1 Accuracy and 3.4 Score.
  • VS-Movie: MM-Mem trails Flash-VStream on VS-Movie by -4.0 Accuracy and -0.2 Score, at 52.1 versus 56.1 and 3.2 versus 3.4.The passage attributes this gap to cues such as scene cuts, dialogue patterns, and narrative coherence not fully captured by the current memory design.
  • Cross-split observation: Across splits, MM-Mem shows clear gains on VS-Ego but a smaller advantage on VS-Movie, suggesting better generalization to egocentric streaming under the current setup.

F.4. SFT vs. SIB-GRPO RL

SIB-GRPO improves HD-EPIC++ performance more than supervised fine-tuning over the same Qwen3-VL-8B baseline, supporting its stronger contribution in this evaluation.

  • Performance comparison: 30.28 accuracy from SIB-GRPO exceeds the Qwen3-VL-8B baseline by +3.03, whereas SFT reaches 27.25, or +1.37.
  • Performance comparison: SIB-GRPO provides a +1.66 accuracy gain over SFT, whose improvement is described as clear but limited.
  • Interpretation: The reported margin indicates that preference-driven reinforcement learning complements supervised learning for HD-EPIC++ video understanding and action grounding.
Loading 2603.01455v3…