PaperlayerView exact revision on arXiv

Source-linked AI summary

Task-Focused Memorization for Multimodal Agents

Tao Zou, Yichen He, Tian Qiu, Yuan Lin, Hang Li

arXiv:2605.31075v1cs.CV

TL;DR

Multimodal agents must select useful information from unbounded observations, yet existing memory formation is not explicitly aligned with changing task demands. TaskMem learns memorization through two-phase reinforcement learning, and improves VQA accuracy across three streaming benchmarks by 6.3%, 7.0%, and 5.3%, respectively.

  • Problem

    The central gap is determining which information multimodal agents should retain so memory remains relevant to their roles and future environment tasks.

  • Method

    TaskMem trains a memorization policy in two phases: first for accurate, non-redundant, well-formatted memories, then for task relevance using online adapter tuning and environment feedback.

  • Results

    6.3%, 7.0%, and 5.3% VQA accuracy improvements are reported on VideoMME, EgoLife, and EgoTempo, respectively, with consistent gains across the three benchmarks.

  • Takeaways & Limitations

    TaskMem makes memory formation an active, task-focused policy that adapts memory content to deployment environments while preserving fundamental memory quality.

  • Takeaways & Limitations

    Phase Two must adapt from sparse recent-task feedback while avoiding catastrophic forgetting and remaining computationally efficient at deployment time.

Abstract

from arXiv · show

Long-term memory is essential for multimodal agents to build coherent experience, accumulate world knowledge, and achieve continual learning. However, constructing effective memory goes beyond memory module design and basic requirements such as accuracy and fidelity; the key challenge lies in determining what to memorize. Multimodal agents, such as embodied agents, continuously perceive, reason, and act in real or virtual environments, receiving an unbounded stream of multimodal observations. From this combinatorial explosion of information, an agent must selectively retain content that is relevant to its role in the environment and valuable for future tasks. To bridge this gap, we frame memory generation as a learnable memorization policy and introduce TaskMem (Task-focused Memorization Policy Learning), a reinforcement-learning-based framework that enables the policy to dynamically adjust its focus to the demands of real tasks encountered in the environment. TaskMem adopts a two-phase training paradigm: Phase One learns how to memorize by optimizing memory quality under fundamental fidelity requirements; Phase Two occurs after deployment, where the agent learns what to memorize by tuning an adapter on its base MLLM, using recent environment tasks to define a reward model that guides the memorization policy toward task-relevant content. To evaluate our approach, we reformulate VideoMME, EgoLife, and EgoTempo into streaming benchmarks that simulate a realistic setting in which an agent processes streaming observations and handles tasks arriving online. To isolate memory assessment, the questions must be answered using only the agent's memory, without access to raw video. Built on Qwen3-VL-30B-A3B, TaskMem improves VQA accuracy by 6.3%, 7.0%, and 5.3% on these benchmarks, respectively.

1 Introduction

Multimodal agents need long-term memory, but the central challenge is selecting information that remains relevant to their roles and future tasks. TaskMem addresses this by learning a task-focused memorization policy and evaluates it in streaming VQA settings.

  • Long-term memory supports multimodal agents’ cross-modal coherence, world knowledge, continual learning, and long-horizon decision-making.
  • The core challenge is autonomously deciding what information to store from unbounded multimodal observations.
  • Existing memory-generation methods are largely heuristic and do not explicitly optimize which information should be memorized for environmental tasks.
  • TaskMem frames memory generation as a learnable policy that selects information from streaming inputs using recent memory history.
  • TaskMem uses two optimization phases: Phase One improves correctness, non-redundancy, and format compliance, while Phase Two aligns memory with deployment tasks.Phase Two tunes a lightweight adapter using task-level rewards and augmented pairwise preferences.
  • 6.3%, 7.0%, and 5.3% accuracy gains are reported on VideoMME, EgoLife, and EgoTempo, respectively, over the base model.The benchmarks are reformulated as sequential task streams in which questions are revealed after video processing.

2 Approach

TaskMem formulates episodic memorization over streaming video as a policy that selects memory content from recent observations and prior memories. Its two-phase training first establishes fundamental memory quality, then adapts memory focus to deployment tasks through online preference optimization and lightweight tuning.

  • Problem Formulation: TaskMem processes streaming video with a sliding-window context of recent segments and previously generated memories, producing an episodic memory for the current segment.The policy conditions on q_t and selects m_t for v_t; the formulation can extend to semantic and visual memory.
  • Phase One: How to Memorize: Phase One uses reinforcement learning before deployment to optimize factual accuracy, non-redundancy, formatting, and content richness without curated supervised training data.The trajectory-level reward combines format, reasoning-length, quality, and richness components, with format correctness treated as a prerequisite.
  • Phase Two: What to Memorize: Phase Two uses recent environment tasks and feedback to construct pairwise preferences that steer memory generation toward task-relevant content.A reward model infers task intent and compares candidate memories; these comparisons produce preference data for policy optimization.
  • Phase Two: What to Memorize: Phase Two must address sparse feedback, catastrophic forgetting, and computational efficiency because adaptation uses only a small number of recent deployment tasks.The paper identifies examples such as ten questions as a limited learning signal and motivates fast, resource-efficient updates.
  • Phase Two: What to Memorize: Phase Two applies Direct Preference Optimization while updating only adapter parameters and keeping the backbone model fixed.The lightweight adapter is an additive trainable vector inserted at a selected transformer layer to address catastrophic forgetting and deployment efficiency.

3 Experiments

TaskMem is evaluated in streaming VQA settings through two-stage GSPO training, including off-policy and on-policy histories, reward stabilization, and task-focused Phase Two adaptation. Across VideoMME, EgoLife, and EgoTempo, it improves memory-based VQA performance and remains competitive with strong baselines.

  • Evaluation setup: TaskMem reformulates VQA benchmarks into streaming task environments and evaluates answers using generated memory.Videos are processed sequentially, with questions revealed after processing; the evaluation isolates memory by requiring answers from memory alone.
  • Phase One Training: Phase One trains the memorization policy with GSPO using off-policy histories synthesized by Gemini-2.5-Pro and on-policy histories generated by the current policy.On-policy training samples candidate memories, scores them with reward models, updates the policy, and maintains diverse contexts through batch adjustment.
  • Phase One Training: Disabling the richness reward causes reward hacking: memory length decreases rapidly while apparently higher rewards reflect shorter, less substantive memories.The comparison uses TaskMem, a version without richness reward, and a version without NSR control.
  • Phase One Training: NSR can destabilize training after average rewards become high, so TaskMem disables NSR for positive-reward samples.The method combines Positive Sample Reinforce and Negative Sample Reinforce, with the latter restricted to negative-reward samples for stability.
  • Phase Two Training: Phase Two improves task relevance while maintaining stable memory quality and low redundancy; a scaled 10-step adapter matches 40-step performance while reducing training data and time by approximately 75%.The adapter direction converges early, with step-10 and step-40 cosine similarity reaching 0.8, while later training mainly increases adapter norm.
  • Main Results: 6.3%, 7.0%, and 5.3% accuracy improvements are reported on VideoMME, EgoLife, and EgoTempo, respectively, with Phase One and Phase Two both improving over the base model.TaskMem outperforms baselines on VideoMME and EgoLife, remains competitive on EgoTempo, and achieves higher precision than GPT-5.2.

4 Case Study

The case studies show that TaskMem’s Phase Two training shifts episodic memory toward task-relevant details while retaining accurate, coherent descriptions. A long-video example further illustrates detailed, low-hallucination, streaming memory tailored to counting tasks.

  • Task-Focused Memory: Phase Two shifts memory from general event narration toward fine-grained details relevant to the trained task.The reported cases compare Phase One and Phase Two generations across object recognition and other tasks.
  • Object Reasoning: In the object-reasoning case, Phase Two adds more detailed scene, object, and appearance information than Phase One.
  • Long-Video Example: TaskMem’s five-minute-video memory contains 4620 tokens and is described as detailed yet low-hallucination.
  • Long-Video Example: The long-video example maintains coherent narrative flow across clip boundaries and consistent global entity IDs such as [face_1].
  • Task Relevance: For counting tasks, the policy records local counts within clips to support solving the global counting problem.

5 Related Work

Prior multimodal-agent memory systems use diverse storage and representation paradigms, but commonly predefine memory content through prompting or post-training. TaskMem instead targets continual adaptation of memory formation using environmental task feedback.

  • Prior work explores memory banks, sparse representations, and textual, object-centric, episodic, semantic, and visual memories for long-horizon reasoning.
  • Existing approaches typically predefine what is stored through prompting or post-training.
  • TaskMem focuses on dynamically adapting memory content to environmental demands through continual feedback.
  • Unlike direct test-time-training signals, deployment provides only task interaction rather than direct observations of memory quality.

6 Conclusion

TaskMem trains a reinforcement-learning memorization policy in two phases: first for fundamental memory quality, then for task relevance. Across three streaming VQA benchmarks, it consistently outperforms the evaluated baselines.

  • TaskMem trains a memorization policy with reinforcement learning to generate task-relevant episodic memory.
  • Phase One optimizes accuracy, non-redundancy, formatting, and content richness, while Phase Two tunes a lightweight adapter toward task-relevant content.
  • TaskMem consistently outperforms all evaluated baselines across VideoMME, EgoLife, and EgoTempo under streaming VQA evaluation.
  • The paper identifies extending beyond episodic memory to semantic and visual memory, and to more interactive embodied environments, as future work.

A Implementation Details of Tools

The implementation extracts face and speaker representations and uses gated, multi-component rewards for Phase One training. Valid memories are assessed for format, quality, reasoning length, and richness.

  • Representation Extraction: Face recognition annotates detected faces with bounding boxes and persistent face IDs on middle frames from one-second clips.
  • Representation Extraction: Speaker identification links recognized audio segments to the most likely face ID, using global-memory voice retrieval when local identification fails.
  • Reward Design: The Phase One reward is format-gated: invalid outputs receive only a format penalty, while valid outputs receive thinking-length, quality, and richness rewards.
  • Reward Design: The thinking-length reward penalizes overlong reasoning traces; experiments set the length threshold Lthink to 1200.
  • Reward Design: The quality reward evaluates factual grounding, coherence with prior memory, non-redundancy, textual form, and token-budget compliance.
  • Reward Design: Richness is rewarded only after format and quality validity, encouraging non-redundant, content-rich memories rather than merely acceptable short outputs.

C.1 Reward Model Implementation

Phase One combines rule-based checks with external evaluators to reward valid, faithful, non-redundant, and content-rich memories.

  • C.1 Reward Model Implementation: Format validity and thinking length are scored deterministically from the parsed reasoning trace and final memory output.The format checker verifies required thinking tags and JSON schema compliance; length is computed from reasoning-trace token length.
  • C.1 Reward Model Implementation: Quality validity requires faithful visual grounding, contextual coherence, textual validity, non-redundancy, and compliance with the memory token budget.Gemini-2.5-Flash assesses faithfulness and coherence, while GPT-4o checks textual validity and non-redundancy.
  • C.1 Reward Model Implementation: Richness is rewarded by ranking format-valid and quality-valid memories for non-redundant, visually grounded, content-rich episodic information.The rankings are converted into scalar rewards, with fixed prompts and deterministic decoding across experiments.

C.2 Training Hyperparameters of GSPO

The supplied passages identify the GSPO hyperparameter table and describe the fixed candidate cache used to derive task-relevance preferences during Phase Two.

  • C.2 Training Hyperparameters of GSPO: Table 9 reports the hyperparameters used for GSPO training in Phase One.The passage points to Table 9 but does not provide the individual hyperparameter values.
  • C.2 Training Hyperparameters of GSPO: Phase Two precomputes and fixes candidate memories generated by the Phase One policy for each streaming context.The cache stores candidate sets for current clips and is not resampled during Phase Two.
  • C.2 Training Hyperparameters of GSPO: Recent questions from the deployment environment define pairwise task-relevance comparisons between candidate memories without using ground-truth answers.The evaluator compares candidates generated for the same context under the task distribution represented by recent questions.
  • C.2 Training Hyperparameters of GSPO: Only consistent, non-tied comparisons retained after swapping presentation order become relative preferences rather than absolute scalar rewards.The task reward model uses GPT-4o to compare candidate memories for supporting recent environment questions.

D.3 Preference Data Construction

Preference data construction filters candidate-memory comparisons for reliable task-relevance differences while controlling for basic memory quality, then selects a strong pair from an acyclic preference graph.

  • D.3 Preference Data Construction: DPO pair construction requires a clear task-relevance difference and no substantial basic-quality difference between the two memories.This keeps the preference focused on task relevance rather than general memory quality.
  • D.3 Preference Data Construction: Reliable win/loss comparisons are retained, while tied or order-inconsistent results are discarded before graph construction.The retained preferences are aggregated into a directed graph over candidate memories.
  • D.3 Preference Data Construction: The preference graph is converted to a DAG by removing cycle-involved comparisons, with longer paths representing larger task-relevance gaps.An edge y →x means x is preferred to y for the current task distribution.
  • D.3 Preference Data Construction: The final pair maximizes DAG path distance subject to preference, non-inferior preferred-memory quality, and exclusion of pairs that are both low-quality.Ties in path distance are broken by choosing the smaller memory-length difference.
  • D.3 Preference Data Construction: 29.17% of sampled contexts yield valid preference pairs from N = 8 candidates, covering roughly 100 videos for Phase Two training.The implementation samples eight candidate memories per context before filtering.

D.4 Training Metric Implementation

Phase Two evaluation checks whether task-focused adaptation preserves memory fidelity and non-redundancy while improving relevance relative to the Phase One reference policy.

  • D.4 Training Metric Implementation: Accuracy counts a memory as correct only when it is faithful to the current clip and subtitles and satisfies the memory token budget.Gemini-2.5-Flash judges faithfulness, while a rule-based check verifies the predefined length limit.
  • D.4 Training Metric Implementation: Non-redundancy measures whether generated memory adds new information without repeating historical memories while remaining well-formed and stylistically consistent.GPT-4o compares each generated memory against previously stored content.
  • D.4 Training Metric Implementation: Relevance compares current-policy memories with Phase One reference-policy memories using recent environment questions as the task context.GPT-4o judges win, tie, or loss under the same task-relevance prompt used by the task reward model.
  • D.4 Training Metric Implementation: Table 10 reports the DPO training hyperparameters.The supplied passages identify the table but do not provide its individual hyperparameter values.

E.1 Robustness to the Choice of Answer Generator

TaskMem’s gains remain consistent when the answer generator changes, although absolute scores differ between GPT-4o and Gemini-2.5-Pro. With fixed memories, both coverage and precision improve, indicating robustness to the QA model.

  • Evaluation setup: GPT-4o and Gemini-2.5-Pro are compared under the same memory-based QA protocol while generated memories remain fixed.The comparison isolates answer-generator effects from changes in memory quality.
  • Results: 6.3 points with GPT-4o and 5.6 points with Gemini-2.5-Pro are TaskMem’s accuracy improvements over Qwen3-VL-30B-A3B.The relative improvement persists despite different absolute scores.
  • Interpretation: Gemini-2.5-Pro’s higher absolute scores mainly reflect higher coverage, while its precision remains comparable.Given identical memories, Gemini-2.5-Pro more often identifies sufficient evidence and attempts an answer.

E.3 Hard Case Analysis

Error analysis of 50 incorrectly answered questions identifies missing fine-grained evidence, inaccurate visual descriptions, and failures to integrate relational and temporal structure. These cases motivate richer visual memory that preserves task-relevant spatial, relational, and temporal cues.

  • Analysis procedure: 50 incorrectly answered questions were manually inspected for evidence sufficiency, factual accuracy, and QA failures despite relevant evidence.The analysis organizes hard cases around what the generated memory preserved and whether the QA model could use it.
  • Failure modes: Three major failure modes are missing fine-grained visual evidence, inaccurate or misleading visual descriptions, and incomplete integration of visual information.The sampled errors distinguish missing evidence from inaccurate descriptions and integration failures.
  • Implications: Future work should incorporate richer visual information and preserve spatial, relational, and temporal cues from cross-modal information.The analysis leaves open how visual memory should be structured, organized, and selective about retained details.
  • Failure modes: TaskMem’s hard cases include memories that capture local visual cues but fail to preserve relational and temporal information needed to answer questions.This indicates that low-level visual perception can be present without sufficient task-relevant semantic integration.
Loading 2605.31075v1…