PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner

Jieke Wang, Tiancheng Shen, Yibo Yang, Ming-Hsuan Yang

arXiv:2608.23268v1cs.CV

TL;DR

Scientific and mathematical reasoning remains challenging for multimodal models, while parameter adaptation is unavailable for closed-weight or on-device backbones and stateless prompting cannot accumulate solved experience. DG-Mem adds dual-grained external memory to a frozen MLLM, using exemplars, reflection-derived schemas, incremental categorization, and Shapley attribution; it consistently outperforms no-memory and competitive memory baselines across three benchmarks and four backbones.

  • Problem

    Multimodal models still falter on scientific and mathematical reasoning, while parameter-level adaptation requires unavailable gradient access and stateless prompting cannot reuse solved problems.

  • Method

    DG-Mem pairs a frozen MLLM with exemplar and IF-THEN schema memories, uses reflections for schema synthesis and online categorization, and re-weights retrieval with Shapley-derived per-rule utilities.

  • Results

    DG-Mem consistently outperforms memory-less baselines and ViLoMem across four backbones and three multimodal reasoning benchmarks.

  • Takeaways & Limitations

    The gradient-free design supports deployment on closed-weight and on-device multimodal backbones while combining instance grounding with abstract schemas.

  • Takeaways & Limitations

    Shapley utilities remain sensitive to stochastic backbone rewards, and freezing memory at test time excludes continual learning from test-time errors.

Abstract

from arXiv · show

Frontier multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning. Parameter-level adaptation is unavailable for closed-weight or on-device backbones, and stateless prompting forfeits any compounding benefit from problems already solved. We propose \textbf{DG-Mem}, a dual-grained agentic memory framework that augments a frozen MLLM with a non-parametric, externally stored memory built once from training-time rollouts and consulted read-only at test time. Motivated by the Complementary Learning Systems (CLS) account of human memory, DG-Mem factors its store into an instance-grounded exemplar memory and a category-level schema memory of IF-THEN rules, with a transient reflection store mediating their construction so that schemas are synthesized only from abstract reflections, never from exemplar text. Two design choices distinguish DG-Mem: an online concept categorizer that grows the category space incrementally during training rather than committing to a predefined taxonomy, and a Shapley context attribution procedure that decomposes correctness across the entire retrieved rule set and yields a per-rule utility that re-weights retrieval at test time. The pipeline introduces no gradient updates and is deployable on closed-weight or on-device backbones. Across MathVista, MMMU, and MMMU-Pro on four open-weight and proprietary backbones (Qwen3.5-27B, Qwen3.5-122B-A10B, GPT-5-Nano, Gemini-3-Flash), DG-Mem improves consistently over no-memory and competitive memory baselines.

1 Introduction

DG-Mem addresses the limits of stateless prompting and parameter-level adaptation with dual-grained external memory for frozen multimodal backbones. It combines exemplars, abstract schemas, incremental categorization, and Shapley-based rule attribution, improving consistently across multiple benchmarks and backbones.

  • Motivation: Frozen or proprietary MLLMs remain weak on scientific and mathematical reasoning, while parameter-level adaptation requires gradient access and stateless prompting cannot reuse solved problems.External agentic memory offers a non-parametric store that conditions a frozen backbone across problems.
  • Motivation: Existing memory systems either preserve instance-bound episodes or distill experience into flat rules, trading concrete grounding against compact abstraction.The paper identifies this granularity trade-off as a central limitation of prior approaches.
  • Framework: DG-Mem separates instance-grounded exemplars from category-level IF-THEN schemas, synthesizing schemas exclusively from transient abstract reflections.This separation operationalizes CLS-style consolidation while preventing exemplar text from entering schema synthesis.
  • Framework: An online concept categorizer grows the category space during training, while Shapley context attribution distributes credit across retrieved rule sets and produces per-rule utilities for retrieval.These mechanisms address incremental taxonomy construction and multi-memory credit assignment.
  • Results: Across four backbones and three multimodal benchmarks, DG-Mem consistently beats the No-memory baseline and both offline and online ViLoMem.The evaluated backbones are Qwen3.5-27B, Qwen3.5-122B-A10B, GPT-5-Nano, and Gemini-3-Flash; benchmarks are MathVista, MMMU, and MMMU-Pro.

2 Related Works

Related work spans memory granularity and construction regime. Prior systems preserve episodic cases, distill rules, or learn memory operations, whereas the comparison framework distinguishes multimodality, memory grains, credit assignment, and gradient-free operation.

  • Granularity: Agentic memory methods differ in what they store, including episodic trajectories, case banks, unstructured notes, and prescriptive playbooks.Trajectory and case stores retain concrete experience, while playbooks distill it into generic rules.
  • Comparison Dimensions: Table 1 compares methods by multimodal handling, instance-grounded exemplars, abstract schema rules, per-rule credit assignment, and gradient-free operation.Each dimension is marked with ✓ or × according to whether the method satisfies it.
  • Construction Regime: Construction regimes include learned memory operations, lightweight memory managers, latent memory tokens, and offline distillation from frozen-backbone rollouts.Several learned regimes require gradients and may use opaque latent representations or fixed cognitive taxonomies.

3 Methods

DG-Mem constructs a frozen-backbone memory offline using complementary exemplar, reflection, and schema stores, then applies utility-aware retrieval at test time. Its pipeline grows categories online and uses Shapley attribution to assign credit across retrieved rules.

  • Memory architecture: DG-Mem augments a frozen MLLM with externally stored exemplar and schema memories, while a transient reflection store mediates offline construction and remains unretrieved at inference.The memory is constructed once from training rollouts and consulted read-only at test time.
  • Memory construction: For each training problem, N temperature-sampled trajectories are verified as correct or incorrect, providing the sole training signal without reasoning annotations beyond the gold answer.Verification uses a cascade of native matching, symbolic parsing, and LLM judging.
  • Exemplar memory: The exemplar reflector produces a transferable reasoning strategy and verification check, indexed with a multimodal embedding to distinguish visually different problems sharing textual concepts.The verification check identifies the dominant error mode in incorrect trajectories.
  • Schema memory: Schemas are synthesized per online-grown category from abstract reflections, with redundancy removed and instance-specific phrasing elevated into category-wide prescriptive rules.Exemplar-level text and trajectories are excluded from schema synthesis, while aggregation across problems can dilute spurious single-instance rules.
  • Memory access: At test time, schema retrieval combines normalized cue similarity with accumulated rule utility, while exemplar retrieval uses multimodal cosine similarity and a similarity threshold.The interpolation parameter α recovers similarity-only retrieval at α = 1 and utility-only ranking at α = 0.
  • Utility training: Shapley attribution evaluates empty, full, and sampled intermediate rule subsets, converting binary rollout rewards into per-rule marginal contributions that update schema utilities.Exact Shapley computation is available when B ≥ 2^K; the exact limit has O(K!) cost.

4 Experiments

DG-Mem is evaluated as a frozen-backbone memory system on held-out multimodal reasoning benchmarks, with controlled comparisons against no memory, ViLoMem, and component ablations. Results show consistent benefits from combining exemplar and schema memories, while Shapley utility further improves retrieval.

  • Experimental Setup: Evaluation uses fixed 3:1 train/test splits, held-out test partitions, top-1 accuracy, and a memory store frozen during testing.Only gold training answers are required; no chain-of-thought or per-step annotations are used.
  • Experimental Setup: DG-Mem is tested with Qwen3.5-27B, Qwen3.5-122B-A10B, GPT-5-Nano, and Gemini-3-Flash against No-memory and offline or online ViLoMem.The same backbone is used throughout each agent graph, except for the fixed verification-cascade judge.
  • Ablation Studies: Removing exemplar memory reduces accuracy by −3.2, −2.6, and −3.7 on MathVista, MMMU, and MMMU-Pro, respectively.Exemplars preserve diagrammatic configurations that abstract IF-THEN rules may omit.
  • Ablation Studies: Exemplar-only trails the full method by −4.0, −0.7, and −3.7 on MathVista, MMMU, and MMMU-Pro, while the full method exceeds both single-grain ablations.The two memory grains therefore address non-overlapping failure modes rather than redundant signal.
  • Ablation Studies: Using similarity-only schema retrieval costs −0.6, −0.7, and −2.1 relative to the full method on MathVista, MMMU, and MMMU-Pro.The consistent loss supports re-weighting retrieved rules with Shapley-derived utility rather than embedding similarity alone.

5 Limitations

The paper identifies attribution stochasticity and frozen test-time memory as limitations that bound DG-Mem’s current scope and motivate follow-up work.

  • Finite-budget Shapley attribution can produce noisy marginal utilities because identical-context backbone calls may vary across executions.The resulting utility sign can depend on data order, despite averaging across problems and ranking rather than thresholding utilities.
  • Freezing memory during evaluation isolates retrieval and conditioning effects but excludes continual learning from test-time errors.The authors leave combining offline consolidation with bounded online updates as an open design question.

6 Conclusions

DG-Mem equips frozen multimodal backbones with separated exemplar and schema memories, dynamically categorizes concepts, and uses Shapley attribution to improve retrieval ranking. Across the evaluated benchmarks and backbones, it consistently outperforms memory-less and competitive single-grain baselines.

  • DG-Mem combines instance-grounded exemplars with category-level IF-THEN schemas separated by a transient reflection store.Schemas are synthesized from reflections rather than exemplar text, while the backbone remains frozen.
  • Online incremental categorization avoids a fixed manual taxonomy, while Shapley attribution ranks retrieved rules by empirical utility rather than similarity alone.These mechanisms address concept-space growth and multi-rule credit assignment within the memory pipeline.
  • DG-Mem consistently outperforms memory-less baselines across four backbones and three multimodal reasoning benchmarks, and surpasses ViLoMem in competitive configurations.The conclusion also reports that ablations support distinct roles for exemplar and schema memories and a consistent utility-reweighting margin.

A.1 Datasets and Splits

The evaluation uses fixed train/test partitions shared across backbones and methods, enabling head-to-head comparisons while keeping test problems held out from memory construction and attribution.

  • A fixed 3:1 train/test partition is used for each benchmark, with the same split file shared by every backbone and method.This design supports head-to-head comparisons among No-memory, ViLoMem, and DG-Mem.
  • Only gold answers from training problems provide supervision; no chain-of-thought or step-level annotations are used.
  • Training problems construct memory and estimate utility, while held-out test problems are evaluated with memory frozen.

A.2 Test-Time Retrieval

At test time, DG-Mem first activates categories, retrieves high-scoring schema rules, and separately retrieves sufficiently similar exemplars; retrieval uses fixed similarity and interpolation settings.

  • For each query, the concept summarizer runs once before category retrieval and rule selection within active categories.Categories must pass a 0.40 similarity gate, and up to 2 rules per active category are retrieved.
  • Schema rules are ranked using similarity and utility interpolation, with a 0.35 minimum rule-similarity threshold.The interpolation coefficient α is 0.8.
  • Exemplars are retrieved by multimodal cosine similarity against the query representation and injected only when they clear the similarity threshold.If no exemplar qualifies, no exemplar is added to the context.

B Additional Ablation Studies

Additional ablations on GPT-5-Nano and Qwen3.5-122B-A10B show that both memory grains contribute, while their combination provides complementary benefits across benchmarks.

  • Each grain alone improves over the No-memory baseline on every benchmark across GPT-5-Nano and Qwen3.5-122B-A10B.The component sweeps are reported in Tables 6 and 7, with each row disabling one mechanism.
  • The full method exceeds the better of the two single-grain ablations on every cell.This pattern holds across both additional backbones and all benchmark cells.
  • The results indicate that exemplar and schema memories cover non-overlapping failure modes rather than redundant signal.This interpretation is stated directly alongside the consistent ablation pattern.

C Full Prompts

The appendix specifies the prompts and operational rules that implement DG-Mem, covering solver execution, concept summarization, reflection, exemplar construction, and online categorization.

  • C Full Prompts: The full-prompt appendix presents each template separately and identifies system, user, and shared or utility templates by color.Runtime placeholders are substituted with fields such as the question and formatted rollout details.
  • C.1 Solver Prompt: The solver prompt asks the frozen MLLM to solve a multimodal problem step by step and return a boxed final answer, with retrieved memory optionally prepended.The user turn contains the image and question text.
  • C.2 Concept Summarizer (K): The concept summarizer produces K(q), a concise subject-and-concepts signature used for online categorization during training and retrieval at test time.It is instructed not to solve the problem and to output the subject and key concepts.
  • C.3 Question-Level Reflector (ReflR, writes to MR): The question-level reflector converts labeled rollout groups into abstract IF-THEN rules for the transient reflection store MR.Rules target positive heuristics, visual traps, and logical or calculation traps while remaining tied to key concepts.
  • C.3 Question-Level Reflector (ReflR, writes to MR): The reflector constrains rules to concise, transferable statements and returns two to five rules in a JSON object.Problem-specific numbers, option letters, variable names, and entity names are prohibited.
  • C.4 Compact Reflector (ReflE, writes to ME): The compact reflector creates each exemplar’s transferable reasoning_strategy and verification_check from rollout text.It is the only prompt allowed to reference rollout text directly; schema synthesis instead receives abstract rules.
  • C.4 Compact Reflector (ReflE, writes to ME): The exemplar prompt requires procedures that work for different problems and images, emphasizing what to do and how to verify rather than specific answers.Specific numerical values, option letters, and problem entities must not appear in the output.
  • C.5 Online Concept Categorizer (Cat): The online concept categorizer assigns a problem to a strongly matching existing category or creates a descriptive new category when no good fit exists.It may update representative concepts when assigning a problem to an existing category.
Loading 2608.23268v1…