PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Learner with Grow-and-Refine Multimodal Semantic Memory

Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu, Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He, Xiaofan Li, Na Zhao, Jingdong Wang, Zechao Li

arXiv:2511.21678v2cs.AIcs.LG

TL;DR

Existing memory-augmented MLLMs can retain logical summaries while losing visual grounding needed for multimodal reasoning. ViLoMem stores and retrieves visual and logical error schemas separately, and consistently improves results across six multimodal benchmarks, especially on mathematical reasoning tasks.

  • Problem

    Existing memory-augmented models retain high-level logical summaries but discard visual grounding and perceptual cues essential to multimodal reasoning.

    Source: §1 · p. 1

  • Method

    ViLoMem separately encodes visual distraction patterns and logical hallucination errors in structured schemas, coordinating their retrieval and updating the relevant stream from experience.

    Source: §1 · p. 2

  • Results

    Across six multimodal benchmarks, ViLoMem consistently improves, with GPT-4.1 gaining +6.48 on MathVision and particularly pronounced gains on mathematical reasoning tasks.

    Source: §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    ViLoMem enables progressive mistake reduction and cross-domain knowledge transfer in multimodal tasks.

    Source: §1 · p. 2

    On MMMU (1,041 samples, GPT-4.1 solver), 66 regression cases included 22 caused by generic visual memory and 44 by empty retrieval.

    Source: §6.5 · p. 14

Abstract

from arXiv · show

MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo -- solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a single-modality trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both multimodal and integrated, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce ViLoMem, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge -- preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across six multimodal benchmarks, ViLoMem consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction-hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning. Our project page is available at https://weihao-bo.github.io/ViLoMeo-page.

1. Introduction

ViLoMem addresses the limits of solving multimodal queries in isolation and of memories that retain reasoning summaries but discard visual grounding. It separately stores visual and logical error patterns, retrieves them for new problems, and updates memory from experience.

1. Introduction

  • Existing memory-augmented models store past interactions as high-level logical summaries but discard visual grounding and perceptual cues essential to multimodal reasoning.MLLMs also approach queries de novo, re-deriving insights and repeating familiar errors.Source: §1 · p. 1
  • Visual perception remains weaker than language reasoning; diagram errors can persist in intermediate steps, and visual attention errors can cause downstream logical hallucinations.Across six benchmarks, visual error summaries consistently exceed logical memory errors.Source: §1 · pp. 1–2
  • Logic-only memory is insufficient because abstract rules must be aligned with the correct visual counterparts, as in applying an area formula to a triangle’s shape.The paper relates this progressive integration to human semantic memory’s visual and abstract representations.Source: §1 · p. 2
  • ViLoMem separately schemas visual distraction patterns and logical hallucination errors, then uses question-aware visual attention and task-positioned logical retrieval.Visual retrieval uses keyword-guided cross-modal attention maps; logical retrieval first identifies the problem’s subject and reasoning requirements.Source: §1 · p. 2
  • ViLoMem automatically attributes successes or failures to the visual or logical stream and updates the corresponding schemas without human supervision.The authors state that it enables progressive mistake reduction and cross-domain knowledge transfer.Source: §1 · p. 2

2. Related Work

Prior work explores both prompt-based self-improvement and persistent memory for agents. The paper identifies a gap in long-term memory for multimodal tasks: existing approaches tend to retain logic while neglecting visual information.

2.1. Context Engineering

  • Context-engineering methods refine prompts from feedback, but their single-interaction context limits long-term accumulation and iterative refinement can remove important details.Examples include ReAct, Reflexion, TextGrad [49], and GEPA [2].Source: §2.1 · p. 2

2.2. Long-term Memory

  • Long-term memory approaches retain knowledge from experience; Dynamic Cheatsheet [33] stores successful and unsuccessful strategies, while ACE [53] incrementally curates a context playbook.Other memory-population approaches use formative experience, reinforcement-learning exploration, or human feedback.Source: §2.2 · p. 2
  • The paper characterizes existing memory frameworks as logic-centric, contrasting them with a hub-and-spoke account of modality-specific visual and logical representations integrated by a semantic hub [9, 22, 23].ViLoMem takes inspiration from this account with separate visual and logical memory modules, a semantic hub, and an executive verifier.Source: §2.2 · pp. 2–3

3. Method

ViLoMem uses separate visual and logical memory banks in a closed loop: it retrieves guidance, solves and verifies each problem, then updates memory from detected errors. Specialized generation and retrieval coordinate the two streams to guide later solutions.

3. Method

  • ViLoMem retrieves from visual and logic memory banks, uses both to solve each image-question problem, verifies the answer, and updates memory when an error is detected.The two banks store textual reasoning guidelines and visual guidelines paired with source images.Source: §3 · p. 3

3.1. Memory Generation

  • Parallel memory generation attributes errors to visual or logical causes and creates structured memory units for the corresponding streams.The framework uses an MLLM for visual analysis and an LLM for logical analysis.Source: §3.1 · p. 3, Figure 2 · p. 4

3.1.1. Visual Memory Generation

  • The visual module analyzes the image, question, erroneous reasoning, and ground truth to classify visual misinterpretation and generate a corrective observation guideline.Visual errors include object confusion, overlooked symbols, and misunderstandings of spatial relationships; similar memories can be merged, while a new entry is created otherwise.Source: §3.1.1 · p. 3

3.1.2. Logical Memory Generation

  • The logic module examines textual reasoning without visual information, classifies reasoning failures, and generates an abstract logic guideline for memory updating.It targets errors such as computational mistakes, formula misapplications, and logical fallacies; its memory uses a similarity-based merge/create process.Source: §3.1.2 · p. 4

3.2. Memory Retrieval and Utilization

  • For a new image-question problem, ViLoMem retrieves memories from both banks using specialized procedures: a two-stage multimodal-to-text pipeline for visual memories and enriched-query analysis for logical memories.Retrieved guidance is supplied to the solver, and verified errors trigger parallel memory updates.Source: §3 · p. 3, §3.2 · p. 4

3.2.1. Visual Memory Retrieval

  • Visual retrieval first finds image-similar memories, then reranks candidates by text similarity to the enriched query and filters them by a threshold and top-k selection.Retrieved visual memories can also generate question-aware attention maps highlighting historically error-prone regions in the query image.Source: §3.2.1 · pp. 4–5

3.2.2. Logical Memory Retrieval

  • Logical retrieval constructs an enriched query from the question and structured domain information, then selects text-similar guidelines above a threshold.The retained guidelines are ranked by similarity and limited to the top-k.Source: §3.2.2 · p. 5

3.2.3. Solution Generation with Dual Memory

  • The MLLM solver generates an answer from the original image and question together with retrieved logical and visual memories.Logical guidelines provide reasoning frameworks, while visual guidelines supply perceptual priors.Source: §3.2.3 · p. 5

4. Experiments

Experiments evaluate ViLoMem across six multimodal benchmarks and three MLLMs. They report consistent accuracy gains, complementary memory streams, and tests of memory transfer and scaling.

4.1. Experimental Setup

  • The setup covers six benchmarks spanning hallucination robustness, multimodal mathematics, and vision-dependent knowledge, evaluated with GPT-4.1 and two Qwen3-VL models using pass@1.The benchmarks are HallusionBench, RealWorldQA, MathVista, MathVision, MMMU, and MMStar.Source: §4.1 · pp. 5–6

4.2. Main Results on Multimodal Benchmarks

  • ViLoMem improves all evaluated models, with GPT-4.1 gains of +6.48 on MathVision and +2.61 on MathVista, and Qwen3-VL-8B-Instruct gains of +4.38 on MMMU and +2.74 on RealWorldQA.Case studies show visual memory supplying task-relevant cues when logical memory retrieves visually irrelevant guidelines; when the problem already provides complete visual descriptions, logical memory alone can suffice.Source: §4.2 · pp. 6–7, Figure 3 · p. 6

4.4. Memory Usage Analysis

  • Visual errors account for 59%–93% of stored cases, yet retrieval uses both streams comparably across benchmarks and models.Retrieval operations also significantly exceed memory generation events.Source: Figure 4 · p. 7, §4.4 · p. 7

4.3. Ablation Study

  • Removing either memory stream degrades GPT-4.1 performance; logical memory removal causes larger drops on MathVista, while visual memory removal degrades both tested benchmarks.The ablation indicates that visual and logical streams capture distinct error patterns rather than redundant ones.Source: Table 2 · p. 6, §4.3 · p. 7

4.5. Cross-Model Memory Transfer

  • Cross-model memory transfer benefits Qwen3-VL-8B-Instruct most, improving MMMU by +1.36 and MathVista by +1.33 over its self-generated memory.Larger models show comparable or slightly reduced performance with transferred memories.Source: Table 3 · p. 7, §4.5 · p. 8

4.6. Cross-Benchmark Memory Generalization

  • Cross-domain memories help MathVision and RealWorldQA, but conflict on MathVista and HallusionBench, supporting the need for task-aligned memory banks.The experiments exclude each target benchmark’s task-specific memory and retrieve from memories accumulated across the other benchmarks.Source: §4.6 · p. 8, Table 4 · p. 8

4.7. Memory Scalability

  • On unseen WeMath, accuracy rises from 72.53 to 74.58 as memory grows from 15k to 150k tokens, and cross-benchmark memory scores 74.58 versus 73.85 for direct memory.The pool is accumulated sequentially from four math-domain benchmarks and contains approximately 3k samples and 150k memory tokens.Source: §4.7 · p. 8, Table 5 · p. 8

5. Conclusion

ViLoMem combines visual and logical memory streams with specialized retrieval and grow-and-refine updates. Across six benchmarks, it improves performance, with complementary streams, cross-domain transfer patterns, and potential for lifelong learning.

5. Conclusion

  • ViLoMem separately models visual distraction and logical hallucination errors, coordinating the streams through specialized retrieval and grow-and-refine updates.The conclusion describes progressive error reduction without catastrophic forgetting.Source: §5 · p. 8
  • Across six multimodal benchmarks, ViLoMem consistently improves performance, with particularly pronounced gains on mathematical reasoning tasks.The conclusion attributes the especially strong gains to visual-logical coupling being acute in those tasks.Source: §5 · p. 8
  • Cross-domain transfer is heterogeneous: task-aligned domains benefit from shared memory, while domain-mismatched tasks show mild interference.Cross-model experiments also report transfer of error patterns and reasoning strategies from stronger models to smaller ones.Source: §5 · p. 8

6. Additional Results and Ablation Study

Additional evaluations report benefits across model architectures, attention variants, and case studies, while exposing limits of attention maps and text-only memory retrieval. Two-stage retrieval and selective updates also compare favorably on efficiency and accuracy.

6.2. Attention Mechanism Ablation

  • Attention maps yield gains on hallucination and general-reasoning benchmarks, but plateau or marginally decline on MathVista and MathVision.The authors attribute the math limitation to imprecise fine-grained visualizations and the demands of interpreting heatmap overlays.Source: §6.2 · p. 12, Table 7 · p. 13

6.3. Additional Case Study

  • In vision-intensive cases, visual memory supplies concrete viewing strategies and attention maps focus on queried regions, while logical memory may offer no useful guidance.For geometry and chart reading, visual memory guides inspection and logical memory supplies reusable measurement or graph-interpretation rules.Source: §6.3 · p. 12

6.4. Comparison with Existing Memory Methods

  • Text-only retrieval can surface visually dissimilar examples with similar questions, misdirecting attention and sometimes underperforming the baseline, especially for smaller models.The comparison adapts Dynamic-Cheetsheet’s retrieval-based configuration and ACE-like methodology for multimodal use.Source: §6.4 · p. 12

6.1. Integration with more models

  • ViLoMem consistently improves performance over baseline and step-by-step configurations across GLM-4.1v, InternVL3-38B, and Gemini 2.5.Models with extended deliberative reasoning are reported as more compatible with step-by-step memory retrieval.Source: §6.1 · p. 12, Table 6 · p. 13

6.4. Comparison with Existing Memory Methods

  • On WeMath, two-stage retrieval reaches 74.58% accuracy and reduces latency 22.3× versus text-only retrieval.As memory grows from 87k to 150k tokens, text-only latency rises from 22.8ms to 115ms, while two-stage latency rises from 20.4ms to 61.6ms.Source: Table 8 · p. 13, §6.4 · p. 13
  • On MathVista, ViLoMem achieves +3.01 accuracy while reducing retrieval latency by 63.1% and storage cost by 66.8% versus Dynamic-Cheetsheet.ViLoMem updates memory only for incorrect answers, compared with Dynamic-Cheetsheet’s updates for every case (221K vs. 73K tokens).Source: Table 9 · p. 14, §6.4 · p. 14

6.5. Failure Case Analysis

On MMMU, the analysis identifies two sources of regression when ViLoMem fails on baseline-correct cases: weakly relevant visual memories and empty retrieval.

6.5. Failure Case Analysis

  • Among 66 regression cases on MMMU, 33.3% (22 cases) involved generic visual cues weakly matched to the image and question, while 66.7% (44 cases) had no retrieved memory.The analysis covers 1,041 samples with GPT-4.1 as solver; overall ViLoMem achieved a net gain of +8.86% over baseline.Source: §6.5 · p. 14
  • No failures occurred when both visual and logical memories were successfully retrieved.The authors note that empty retrieval leaves the solver with a step-by-step prompt that may differ from the baseline’s default prompting.Source: §6.5 · p. 14

6.6. Visual Memory Characterization

MMMU results show that memory-stream contributions vary by discipline, with visual memory more beneficial in several technical fields and logical memory stronger in medicine. On MathGlance, ViLoMem also improves accuracy over the no-memory baseline.

6.6. Visual Memory Characterization

  • Across 835 subject-matched MMMU samples, the analysis compares full-system gains over baseline with the independent contributions of visual and logical memory.The table defines ∆(vis) as ViLoMem minus the no-visual variant and ∆(log) as ViLoMem minus the no-logic variant.Source: Table 10 · p. 14, §6.6 · p. 14

Visual memory dominates in Tech & Engineering

  • Visual memory contributes more than logical memory overall (+9.8 vs. +5.4), with the largest visual benefits in Energy & Power (+20.0) and Math (+19.4).Agriculture (+14.3) and Mechanical Engineering (+12.5) also benefit strongly; these subjects include specialized diagrams and plots.Source: Visual memory dominates in Tech &… · p. 14, §6.6 · p. 14
  • In Health & Medicine, logical memory contributes more (+8.1 vs. +1.0 for visual memory), while both streams reduce Business performance.Diagnostics & Laboratory Medicine gains +24.2 overall, primarily from logical memory (+9.1); Business declines are ∆(vis)=−3.6 and ∆(log)=−4.2.Source: Visual memory dominates in Tech &… · p. 14

6.7. Results on Visual Perception Benchmarks

  • On MathGlance, ViLoMem improves average accuracy by +1.87% over no memory on plane/solid geometry and graph tasks, using 166 visual and 16 logic memory entities.OCRBench and DocVQA are not directly applicable because their character-matching evaluation differs from the binary correctness judgments required for error-driven memory generation.Source: §6.7 · p. 14

7. Additional Experimental Details

The section documents model deployment, decoding and memory-generation settings, visual-attention-map construction, and evaluation procedures, including a caveat about API safety filters.

7. Additional Experimental Details

  • Open-source checkpoints generally run through vLLM, while Qwen3-VL-235B-A22B-Instruct and proprietary models are evaluated through APIs.Provider safety filters may reject some API-based images or prompts, adding a small amount of noise to reported scores.Source: §7 · pp. 14–15
  • Unless otherwise specified, models use temperature 0.7 and an 8,192-token generation limit; problem analysis and memory generation use limits of 1,024 and 2,048 tokens, respectively.Baseline questions are given without extra prompts, while Step prepends a step-by-step system prompt.Source: §7 · p. 15
  • Attention maps follow Zhang et al. with Qwen2.5-VL-3B, producing token-level image saliency overlaid as heatmaps to interpret visual-memory retrieval.Source: §7 · p. 15
  • VLMEvalKit is the primary evaluation framework, with Math-Verify and Qwen3-8B-Instruct judging used when automatic matching fails or answers are ambiguous.The judge assesses semantic correctness against the reference answer.Source: §7 · p. 15

8. Prompt Templates

The section supplies prompt templates for the Step configuration, problem analysis, and logical and visual memory generation, alongside a judge prompt and representative memory-process examples.

8. Prompt Templates

Loading 2511.21678v2…