PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Latent Memory Routing for Vision-Language Reasoning

Hao-Xuan Ma, Jin-Fei Qi, Yicheng Xiao, Han-Jia Ye

arXiv:2609.05539v1cs.CVcs.AI

TL;DR

Longer generations can make MLLMs lose track of early visual evidence and intermediate constraints carried in a single growing context. DLMR adds disentangled visual and reasoning memories with a router that selects what and how much to reuse. The paper reports gains across general VQA and reasoning benchmarks and improved token efficiency in text-only reasoning.

  • Problem

    MLLMs can lose access to early visual evidence and intermediate constraints during long generations in a monolithic growing context.

    Source: §1 · p. 1

  • Method

    DLMR equips a frozen MLLM with separate latent memories for visual evidence and reasoning constraints, plus an injector and router that select the memory and token budget.

    Source: §1 · p. 2

  • Results

    DLMR delivers consistent gains on general VQA and reasoning benchmarks across multiple backbones, with the largest improvements on tasks requiring reliable visual revisiting and constraint maintenance over long generations.

    Source: §6 · p. 8

  • Takeaways & Limitations

    DLMR also transfers to text-only reasoning and improves token efficiency.

    Source: §6 · p. 8

    Potential applications are framed as conditional on responsible deployment.

    Source: Impact Statement · p. 9

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have recently made strong progress in vision-language reasoning, yet their performance often degrades as generations grow longer. A key factor is that they frequently lose track of earlier visual evidence and intermediate constraints under a monolithic growing context. Inspired by how humans separately recall what they see and what they infer when solving complex tasks, we propose DLMR, a parameter-efficient mechanism that equips MLLMs with Dual Latent Memories: a visual memory that compresses image evidence and a reasoning memory that tracks intermediate conclusions and constraints. A Router then dynamically decides which memory and how much to reuse during inference, preserving visual grounding while maintaining coherent long-horizon reasoning. DLMR is trained in three stages, from latent memory construction to selective router learning, while keeping the base MLLM frozen, yielding substantial gains on both general and reasoning benchmarks with only a small number of additional trainable parameters. Analyses further show interpretable, state-dependent routing with specialized memory roles and reduced decoding tokens over long generations. Code is available at https://github.com/Hunter-Wrynn/DLMR.

1. Introduction

Long-horizon decoding can weaken access to early visual evidence and intermediate constraints in MLLMs’ single growing context. DLMR adds separate latent memories and a learned router to selectively reuse visual evidence or reasoning state, with reported gains on long-horizon benchmarks.

1. Introduction

  • As decoding lengthens, MLLMs can lose access to early visual evidence and intermediate reasoning states carried in a single growing context.The paper notes that visual-token attention decays as generation proceeds.Source: §1 · p. 1
  • DLMR separates reusable visual observations from evolving reasoning constraints and selectively revisits either source when needed.The router can verify image cues for grounding or reuse derived state during multi-step inference.Source: §1 · p. 2
  • DLMR improves task performance on challenging long-horizon multimodal reasoning benchmarks under controlled inference budgets.The authors also report improved long-horizon performance and better ability to revisit visual evidence.Source: §1 · p. 2
  • The three-stage training recipe progresses from latent memory construction to selective router learning while keeping the backbone frozen.The method adds a small parameter overhead.Source: §1 · p. 2
  • A lightweight injector appends selected latent memories as tokens, while a discrete router chooses the memory and token budget for each step.The base MLLM remains unchanged; Figure 1 reports stronger late-stage visual attention and fewer generated tokens on average than the vanilla model.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Prior vision-language reasoning work uses prompting, training, and retrieval, while memory-based approaches seek to preserve reusable information beyond one growing context. DLMR distinguishes itself with specialized visual and reasoning memories and state-dependent routing.

2.1. Vision Language Reasoning

2.2. Memory Systems

  • Memory-augmented methods include explicit retrieval and latent tokens, but existing transfers typically offer a single undifferentiated memory source and limited state-dependent control.DLMR instead uses separate buffers for visual evidence and reasoning constraints with a discrete router; its memories are injected as budgeted tokens during decoding.Source: §2.2 · pp. 2–3

3. Preliminary

The preliminary analysis links long-horizon visual forgetting to the growing prefix used by standard autoregressive MLLMs. DLMR is motivated as an explicit memory-and-control interface for reusing visual evidence and reasoning constraints.

3. Preliminary

  • The task is to generate an answer consistent with an image and tokenized textual instruction or dialogue history using an autoregressive MLLM.The backbone encodes the image as visual tokens, concatenates them with text tokens, and predicts each next token conditioned on the image, text, and prior output.Source: §3 · p. 3
  • Standard MLLMs concatenate image tokens and the prompt into one prefix, then append generated tokens, so the visible context grows at every decoding step.The next-token state uses causal self-attention over previous tokens.Source: §3 · p. 3
  • As context grows, attention mass on the fixed image-token set tends to shrink unless attention becomes increasingly concentrated on the early visual prefix.The paper connects this trend to visual grounding decay and says early evidence becomes progressively harder to reuse; DLMR therefore stores visual evidence and evolving constraints in dual latent memories and injects compact tokens on demand.Source: §3 · p. 3

4. Method

DLMR adds dual latent memories, a memory injector, and a discrete router to a frozen backbone for selective, budget-aware memory reuse. Its three-stage training recipe learns the memories, injector, and adaptive router while keeping the backbone frozen.

4. Method

  • DLMR adds dual latent memories, an injector, and a discrete router on top of the frozen backbone.The components determine what memory to inject and how much at each step.Source: §4 · pp. 3–4

4.1. Memory Generation

  • Shared visual and reasoning latent memories supply selected vectors that the injector contextualizes into memory tokens for appending during decoding.The memories are input-agnostic; the injector conditions on current multimodal embeddings and the selected latents.Source: §4.1 · p. 4
  • The injector uses LoRA in a replica of the frozen base MLLM to contextualize the selected latents.Source: §4.1 · p. 4

4.2. Memory Routing

  • Memory injection is limited to delimiter-eligible steps and a per-sample cap; the context-dependent router selects a memory type and budget, or decoding proceeds without injection.Injected tokens are appended before the frozen backbone predicts the next token.Source: §4.2 · p. 4

4.3. Training Recipe

  • Stage 1 aligns visual and reasoning memories to teacher representations using self-supervision, with cross-teacher and separation terms discouraging mismatched alignment and shared representations.Source: §4.3 · pp. 4–5
  • Stage 2 trains the injector and memories on downstream data with mixed memory types and budgets; Stage 3 freezes them and trains the router with GRPO using a cost-aware objective.The backbone remains frozen across all three stages, and the router objective encourages smaller average injection budgets only when the answer is correct.Source: §4.3 · pp. 4–5

5. Experiments

Experiments evaluate DLMR against prompting, post-training, and retrieval baselines on general and reasoning benchmarks. Ablations and analyses examine its memory design, routing, transfer, and token efficiency.

5.1. Experimental Setup

  • Evaluation spans general and reasoning vision-language benchmarks and compares DLMR with prompting, post-training, and retrieval-augmented baselines.Experiments use Qwen2.5-VL-7B and InternVL3-8B, with the same training dataset across the three training stages.Source: §5.1 · pp. 5–6

5.2. Main Results

  • DLMR outperforms prompting, post-training, and retrieval-based baselines, with DLMR (SFT) strongest on general benchmarks and DLMR (RL) strongest on reasoning benchmarks.The reported gains are most pronounced on long-horizon reasoning benchmarks, while general VQA also improves.Source: §5.2 · p. 6
  • The dual-latent design raises the six-benchmark average from 52.05 to 59.73 and the four-reasoning-benchmark average from 46.61 to 53.84.MathVision improves from 22.37 to 35.32 when dual memories replace a shared memory.Source: §5.2 · p. 6, Figure 3 · p. 6
  • Freezing the injector reduces reasoning performance because it cannot adapt latent memories into task-useful, context-aligned tokens.The ablation retains the dual memories and injection interface while disabling injector training.Source: §5.2 · pp. 6–7
  • Fixed budgets increase average generation length from 664 to 732 to 765 tokens for k = 4, 8, 16, while accuracy peaks at 52.71 for k = 8.General tasks peak at kmax = 16, whereas reasoning tasks peak at kmax = 32.Source: §5.2 · p. 7, Table 4 · p. 7, Figure 4 · p. 7

5.3. Detailed Analysis.

  • Across LLaVA-OV-1.5-4B, Qwen2.5-VL-3B, and InternVL-3-2B, DLMR improves general VQA averages by +6.8, +6.2, and +5.1, and reasoning-benchmark averages by +7.2, +7.9, and +6.9, respectively.Source: §5.3 · p. 8
  • Without vision features, DLMR improves over the vanilla backbone and GRPO on text-only benchmarks, including GSM8K and ARC-Challenge.The reported gains are +15.39 on GSM8K and +11.08 on ARC-Challenge.Source: §5.3 · p. 8, Figure 5 · p. 8
  • DLMR reduces average generated tokens by 9.7% versus Visual-RFT and 4.2% versus SFT while achieving the best overall average token efficiency.The calculation includes tokens used to generate both latent memories.Source: §5.3 · p. 8, Figure 6 · p. 8

6. Conclusion

DLMR augments frozen MLLMs with disentangled visual and reasoning memories and state-dependent routing. The paper reports gains across multimodal benchmarks and backbones, as well as text-only transfer and improved token efficiency.

6. Conclusion

  • DLMR combines dual latent memories, a lightweight injector, and a discrete router that selects what to reuse and how much to inject at each step.Its three-stage training recipe keeps the backbone frozen; reported gains are largest on tasks requiring visual revisiting and constraint maintenance.Source: §6 · p. 8
  • DLMR shows consistent gains on general VQA and reasoning benchmarks across multiple backbones, with the largest improvements on long-generation tasks.Source: §6 · p. 8
  • The method also transfers to text-only reasoning and improves token efficiency, while routing concentrates computation on harder states.Source: §6 · p. 8

Impact Statement

The impact statement presents DLMR as a research mechanism with potential benefits in visual-problem tutoring, accessibility, and scientific or engineering assistance, alongside deployment risks and safeguards.

Impact Statement

  • Selective reuse may reduce unnecessary generation, latency, and energy under fixed inference budgets in the proposed applications.The statement names education and tutoring on visual problems, accessibility tools, and scientific or engineering assistants as possible applications.Source: Impact Statement · p. 9
  • Stronger grounding and long-horizon reasoning could also increase capability in harmful settings and encourage over-trust despite hallucinations, biases, or distribution-shift brittleness.The statement gives surveillance enablement and more convincing misinformation as examples of harmful settings.Source: Impact Statement · p. 9
  • Training and deployment practices may raise data-provenance concerns involving privacy or copyright.The statement makes these concerns conditional on those practices.Source: Impact Statement · p. 9
  • The authors position DLMR as a research mechanism, not an autonomous high-stakes decision system, and recommend oversight, dataset documentation, bias auditing, privacy-preserving handling, and safety filtering.These are proposed deployment safeguards, not reported experimental findings.Source: Impact Statement · p. 9

A. More experiments

Additional experiments report benchmark-suite accuracy and inference-time comparisons, transfer to unseen datasets, and task- and position-dependent memory routing.

Interpretable routing behavior

  • Reasoning tasks invoke reasoning memory more often, while general tasks favor visual memory; both memories are injected more in early and middle decoding than near the end.The figure plots injection frequency against relative generation position for reasoning benchmarks (top) and general benchmarks (bottom).Source: A. More experiments · p. 12, Figure A1 · p. 12

Inference efficiency on the benchmark suite

  • Across both backbones, DLMR improves accuracy with comparable or lower latency; on Qwen2.5-VL-7B reasoning, accuracy is 47.5%→53.8% and time 14.0→11.5s (≈18%).The comparison is against SFT baselines; the reasoning accuracy and time pairs are reported for Qwen2.5-VL-7B.Source: A. More experiments · p. 12

Transfer to unseen benchmarks

  • LOGICVISTA reaches 55.0 vs. 50.2 for GRPO, OCR-BENCH 92.0 vs. 90.1 for SFT, and MMT-BENCH 62.5 vs. 58.0 for GRPO.DLMR achieves the best performance on all three unseen datasets, spanning reasoning and perception/OCR.Source: A. More experiments · p. 12

B. Methodology Details

This appendix supplies implementation and modeling details that complement Sections 4.1–4.3.

B.1. Injection Eligibility and Budget Cap

  • Routed injections are eligible only when the decoded prefix ends with a delimiter; ineligible steps receive no injection, and a per-sample cap stops further router queries.The delimiter set can include commas, periods, and newlines; the cap is Nmax injections per sample.Source: §B.1 · p. 13

B.2. Injection Placement

  • A fixed visual-memory injection is placed at the prompt end before the assistant marker, so memory tokens are available to generation while the preceding token remains prompt text.This placement applies to chat-style MLLMs.Source: §B.2 · p. 13

B.3. Router Action Space and Parameterization

  • At eligible steps, the router chooses no injection or a visual/reasoning memory and budget; a lightweight classifier maps the latest prefix hidden state to action logits.Router actions are sampled during training and selected greedily at inference.Source: §B.3 · p. 13

B.4. Memory Injector Architecture

  • The injector contextualizes selected latents into memory tokens, using projections when embedding dimensions differ; the key–value cache is reset after mid-generation injection.The injector is a backbone copy with LoRA adapters enabled.Source: §B.4 · p. 13

B.5. Training Details

  • Training moves from teacher–student alignment and mixed memory/budget injections, through an optional specialization regularizer, to GRPO router optimization using task and efficiency rewards.The efficiency reward favors smaller average injection budgets and is counted only when the answer is correct.Source: §B.5 · pp. 13–14

C. Training Details

  • The training-details section describes implementation choices and omits backbone identity and parameter count because the method is backbone-agnostic.The section covers training configurations, preprocessing, optimization, and implementation.Source: C. Training Details · p. 14

C.1. Configuration and Reproducibility

The appendix records configuration and reproducibility choices, comparison baselines, data handling, model and injection settings, and latent cold-start training.

C.1. Configuration and Reproducibility

  • Experiments use YAML configurations, a fixed default seed of 42, mixed-precision activations, and DeepSpeed ZeRO stage-2; the backbone remains frozen.Table A3 summarizes key architectural hyperparameters for the two backbones used in the main experiments.Source: §C.1 · p. 14, Table A3 · p. 15

C.2. Baselines

  • Seven baselines span vanilla, prompting, post-training, and RAG approaches, and are implemented on Qwen2.5-VL-7B and InternVL3-8B.Strategies originally implemented on other backbones are transferred to these two models for comparison.Source: §C.2 · p. 14

C.3. Data Format and Preprocessing

  • The evaluation suite covers general multimodal understanding, long-horizon reasoning, and three text-only benchmarks for cross-modality transfer; the training corpus combines selected benchmark splits with OpenMMReasoner.Table A4 summarizes the benchmarks; benchmark protocols are followed for evaluation.Source: §C.3 · pp. 14–15, Table A4 · p. 15
  • Official dataset splits are retained where available, evaluation-only benchmarks remain evaluation-only, and chat or plain-text inputs receive completion-only supervision with prompt, padding, and vision-placeholder tokens masked.Samples exceeding stage-specific prompt-length limits are filtered to avoid memory issues and unstable delimiter injection.Source: §C.3 · pp. 14–15

C.4. Model and Injection Hyperparameters

  • The model uses visual and reasoning latent banks, typically budgets {4, 8, 16}, at most five routed injections per sample, delimiter-gated queries, and LoRA adapters while backbone weights stay frozen.The default LoRA settings are rank 16, scaling α = 32, and dropout 0.1.Source: §C.1 · p. 14, §C.4 · p. 15

C.5. Stage 1: Latent Cold-Start

  • Latent cold-start aligns pooled teacher and student representations, alternates or samples memory type and budget at eligible steps, and uses AdamW with a cosine schedule and linear warmup.Visual latents are always injected at the prompt end during this stage; a typical run uses learning rate 10−3, warmup ratio 0.1, batch size 1 per device, and 3 epochs.Source: §C.5 · p. 16

C.6. Stage 2: Injector Training (SFT/GRPO)

Stage 2 trains the injector and latent banks while the router is disabled, using mixed memory injections during training and SFT or optional GRPO procedures.

C.6. Stage 2: Injector Training (SFT/GRPO)

  • Stage 2 jointly optimizes injector parameters and latent banks with injector LoRA adapters enabled, while disabling the router.The backbone is not specified in these passages.Source: §C.6 · p. 16
  • SFT optimizes next-token cross-entropy on completion tokens using AdamW and a cosine schedule.Typical settings include learning rate 10−5, warmup ratio 0.1, BF16, and maximum sequence length 1024–2048.Source: §C.6 · p. 16
  • Training injects memories at the prompt end and eligible delimiter points, mixing visual and reasoning injections and sampling budgets from K+.Eligible injections alternate or are chosen randomly to match inference-time behavior.Source: §C.6 · p. 16
  • An optional small-weight regularizer discourages the visual and reasoning banks from collapsing into the same representation.It combines a hinge-based cross-branch penalty with margin m and an inter-bank separation term scaled by ϵ, for example ϵ = 10−3.Source: §C.6 · p. 16
  • When injector learning uses GRPO, multiple rollouts per prompt support group-relative advantages, with an optional reference-policy KL penalty for stabilization.Training enables nonzero-temperature sampling to ensure reward variance within each prompt group.Source: §C.6 · p. 16

C.7. Stage 3: Router Training (GRPO)

Stage 3 freezes the injector and latent banks while training a router to select memory type and budget at eligible steps, using a cost-aware GRPO objective.

C.7. Stage 3: Router Training (GRPO)

  • Stage 3 freezes the injector and latent banks and optimizes only router parameters, including its classification head and optional router LoRA adapters.Table A5 covers latent cold-start, injector SFT, and router GRPO hyperparameters, with values shared across Qwen2.5-VL-7B and InternVL3-8B unless stated otherwise.Source: §C.7 · p. 16, Table A5 · p. 17
  • At eligible steps, the router predicts an action combining visual or reasoning memory with a budget from K+; non-eligible steps use a null action.The action space is A = {v, r} × K+.Source: §C.7 · p. 16
  • Router GRPO uses a cost-aware objective that rewards smaller budgets when the answer is correct.Training uses group-sampled rollouts per prompt and caps prompt and completion lengths to bound compute and stabilize training.Source: §C.7 · p. 16
  • After each routed injection, decoding resets the key–value cache and continues from the augmented sequence.For chat-formatted prompts, prompt-end memory tokens are recommended before the assistant marker to avoid degenerate early termination.Source: §C.7 · p. 16
  • Table A5 reports hyperparameters for all three training stages, with per-device batch sizes and gradient accumulation as needed.Its values apply across Qwen2.5-VL-7B and InternVL3-8B unless stated otherwise.Source: Table A5 · p. 17
Loading 2609.05539v1…