PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Back, Think Ahead: Visual Memory on Demand for Efficient Multimodal Reasoning

Yicheng Xue, Han Wu, Jufeng Yang, Minjing Dong, Xinghao Chen, Hanting Chen, Jianyuan Guo

arXiv:2610.12060v1cs.CV

TL;DR

Long visual sequences make MLLM inference costly, and fixed one-shot contexts may not retain evidence as reasoning needs shift. ViMoD combines recoverable Coarse memory with history-based routing to access Fine tokens on demand. At a 20% target budget, its average normalized score exceeds the strongest one-shot baseline by 39.0%.

  • Problem

    One-shot visual compression fixes the context before decoding, although visual evidence needs can shift across successive reasoning stages.

    Source: §1 · p. 1

  • Method

    ViMoD links compact Coarse representations to recoverable Fine-token groups and uses decoding history to select, retain, or replace active groups.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    At a 20% target budget, ViMoD’s average normalized score exceeds DivPrune, the strongest one-shot baseline, by 39.0%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ViMoD improves accuracy under tight visual budgets and reduces total inference time in the reported evaluation.

    Source: §6 · p. 10

    Short responses or sustained access to broad visual evidence may limit savings, and generalization across backbones and robustness to alternative decoding configurations remain unverified.

    Source: §5 · p. 9

Abstract

from arXiv · show

Processing long visual token sequences from high-resolution images makes multi-step reasoning computationally expensive for multimodal Large Language Models (MLLMs). Existing one-shot pruning and aggregation methods compress visual tokens into a fixed context before decoding. However, visual evidence needs can shift as reasoning unfolds, making it difficult for a fixed compressed context to retain all the details needed across stages. To address this challenge, we propose ViMoD, a lightweight framework that maintains a compact visual context while preserving access to original fine-grained evidence as reasoning needs evolve. Deformable Aggregation of Region-wise Tokens (DART) learns content-adaptive groups and aggregation capacities, constructing compact Coarse representations linked to recoverable original Fine tokens. Temporal Routing for Adaptive Contextual Evidence (TRACE) integrates decoding history to anticipate upcoming evidence needs and select, retain, or replace active Fine-token groups. Selected Fine tokens augment the persistent Coarse context in the frozen backbone, enabling stage-specific evidence access without continuously attending to all visual tokens. On Qwen3-VL-4B, ViMoD outperforms all evaluated baselines on all eight reasoning benchmarks at a 20% target visual token budget, improving the mean normalized score by 39.0% over the strongest evaluated one-shot baseline. These gains are achieved with only 0.0546% additional trainable parameters relative to the frozen backbone.

1 INTRODUCTION

The introduction motivates adaptive visual evidence access because reasoning needs can shift while one-shot compression fixes the context, and per-token savings may not reduce full-response cost. ViMoD pairs recoverable visual memory with adaptive routing, and outperforms the strongest one-shot baseline under a 20% target budget.

1 INTRODUCTION

  • Long visual token sequences make inference costly, while one-shot compression fixes a compact context before decoding despite changing evidence needs across reasoning stages.Larger shifts in highly attributed tokens are associated with lower answer retention at 50% visual-token retention.Source: §1 · pp. 1–2
  • At 70% of Full accuracy, stage-specific Dynamic access uses an interpolated 23.9% visual-token budget versus 36.0% for Fixed oracle.Fixed oracle receives the union of annotated regions at generation start; Dynamic access provides them at corresponding reasoning stages.Source: §1 · p. 2, Figure 2 · p. 2
  • For FastV on MathVerse, longer responses at several retention ratios offset lower FLOPs per generated token and increase estimated total decoder FLOPs.The introduction therefore argues that compression efficiency should be evaluated over complete responses.Source: §1 · p. 2, Figure 3 · p. 3
  • ViMoD combines recoverable Coarse-to-Fine visual memory with TRACE, which uses decoding history to anticipate evidence needs and select, retain, or replace Fine-token groups.Joint-KV exposes selected Fine tokens alongside persistent Coarse context in the frozen backbone.Source: §1 · pp. 2–3
  • At a 20% target budget, ViMoD’s average normalized score exceeds DivPrune, the strongest one-shot baseline, by 39.0%.The comparison is reported for a 9:1 memory setting.Source: §1 · p. 3

2 RELATED WORK

The related-work section distinguishes ViMoD from fixed one-shot compression, attention-based recovery, and approaches that interleave reasoning with visual interaction. It combines persistent compressed context with learned, dynamic access to image evidence.

2 RELATED WORK

  • One-shot methods prune or merge visual tokens early and keep the context fixed; recovery methods reintroduce tokens using attention changes, which can be complicated by attention sinks and head-specific behavior.The cited one-shot methods include Chen et al. (2024a), Yang et al. (2025a), Song et al. (2026), Lu et al. (2026), Alvar et al. (2025), and Zhang et al. (2026a); recovery is associated with Kim et al. (2026), and attention complications with Kang et al. (2025a;b).Source: §2 · p. 3
  • ViMoD combines persistent Coarse context with dynamic Fine access supervised by upcoming reasoning needs.Figure 4 depicts DART building Coarse context and a Coarse-to-Fine map, TRACE updating active evidence, and Joint-KV exposing selected Fine evidence with Coarse context and text history.Source: §2 · p. 3, Figure 4 · p. 4
  • ViMoD adapts compressed-context and on-demand token recovery to images, accounting for spatial structure and region-dependent granularity.DART uses region-wise capacities and deformable groups shared by Coarse aggregation and Coarse-to-Fine-guided Fine recall.Source: §2 · p. 3
  • Unlike tool-based cropping or explicit interaction tokens, ViMoD learns access decisions through lightweight auxiliary modules and integrates evidence with Joint-KV while keeping the backbone frozen.The cited alternatives are DeepEyes (Zheng et al., 2026) and Hu et al. (2026).Source: §2 · p. 3

3 METHOD

ViMoD keeps a persistent compact visual context while retaining original Fine-token groups for selective access during decoding. DART organizes recoverable groups, TRACE routes evidence using decoding history, and training supervises both compression and access.

3.1 VISUAL MEMORY AND EVIDENCE ACCESS

  • Prefill retains the original visual and text key–value banks; DART produces compact Coarse descriptors and a map to Fine-token groups, while all Coarse entries stay active and unselected Fine entries remain stored.Only the active Fine groups participate in attention alongside the persistent Coarse context.Source: §3.1 · p. 4

3.2 DART: DEFORMABLE AGGREGATION OF REGION-WISE TOKENS

  • DART learns group capacities and membership, allocating a fixed total so smaller groups provide finer representations and cheaper recall units.Groups adapt to image content and can cross initial grid boundaries; the figure illustrates stronger compression in blank regions and finer granularity in text-bearing regions.Source: §3.2 · pp. 4–5, Figure 5 · p. 5
  • DART uses shared group support for Coarse aggregation and Fine recall, pooling keys directly and passing Values through a residual encoder before pooling.Each group’s Coarse token has a corresponding Coarse-to-Fine map to its Fine members.Source: §3.2 · p. 5

3.3 TRACE: TEMPORAL ROUTING FOR ADAPTIVE CONTEXTUAL EVIDENCE

  • At fixed intervals, TRACE combines causally available backbone states with active-set history to propose upcoming evidence and choose whether to retain or replace the set.Its recurrent state represents decoding history; HOLD preserves evidence, while UPDATE can add, remove, or clear groups.Source: §3.3 · p. 5
  • Joint-KV concatenates Coarse context, active Fine evidence, and available text history at each decoder layer for the frozen backbone’s attention.The resulting attention computes the next-token distribution during decoding.Source: §3.3 · p. 6

3.4 LEARNING COMPRESSION AND EVIDENCE ACCESS

  • With the backbone frozen, training first fits DART, then fixes DART while training TRACE through supervised initialization and policy optimization.DART uses task-verified responses and full-visual teacher predictions; TRACE receives upcoming-evidence supervision and is optimized on student-generated trajectories with correctness and Fine-occupancy feedback.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate ViMoD on a frozen Qwen3-VL-4B backbone across reasoning and general visual-understanding benchmarks, with 20–40% target budgets in the main evaluation. ViMoD leads under tight budgets, while ablations and efficiency analyses examine adaptive Fine access, learned memory, and end-to-end costs.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses a frozen Qwen3-VL-4B-Instruct backbone, with trainable modules containing 0.0546% of its parameters and separate 4:1 and 9:1 DART compressors.Training and evaluation sets are disjoint; target budgets are 20%, 30%, and 40%, and decoding settings differ from the backbone’s recommended visual-task configuration.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • ViMoD leads the normalized reasoning average at every target budget and all eight benchmarks at 20%, with a 39.0% higher 20% average than DivPrune.At 20%, TRACE improves the normalized reasoning average over DSTP† by 7.37–9.73 percentage points with the same 9:1 DART memory and frozen backbone.Source: §4.2 · p. 7, Table 1 · p. 7
  • At 20%, ViMoD exceeds all one-shot results at 40% on the general-suite average; DocVQA and ChartQA gains are 19.59 and 28.16 percentage points over their strongest one-shot baselines.The authors suggest the document and chart gains may reflect finer DART groups in text-bearing regions and stronger aggregation of blank regions.Source: §4.2 · p. 7, Table 2 · p. 8
  • TRACE transfers from 9:1 to separately trained 4:1 memory without retraining, improving both suite averages at 30% and 40%, while DocVQA and MMMU-Pro favor different configurations.The authors interpret the preferences as a task-dependent balance between persistent Coarse context and remaining Fine access, accommodated by a shared policy.Source: §4.2 · pp. 7–8

4.3 ABLATION STUDIES

  • Repeated Fine recall beats no recall and single recall, improving DynaMath and MMMU-Pro over single recall by 3.35 and 8.12 percentage points, respectively.The table reports Fine-recall accuracy with 9:1 compression at the 30% reference budget.Source: §4.3 · p. 8, Table 3 · p. 8
  • At 30%, DART exceeds regular 3 × 3 mean pooling by 11.27 and 10.35 percentage points, while TRACE optimization adds 5.99 and 4.22 percentage points on DynaMath and MMMU-Pro, respectively.Fine recall remains enabled in both memory variants; policy gains are relative to supervised initialization.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 EFFICIENCY ANALYSIS

  • At 40%, ViMoD is more accurate than all three one-shot baselines with 15.2–20.4% less total time; at 50%, it matches Full accuracy with 39.5% less visual occupancy and 3.9% less total time.One-shot baselines generate 15.9–29.9% more tokens than Full, offsetting their per-token decoding savings; autoregressive decoding accounts for 94.4% of Full runtime in this evaluation.Source: §4.4 · pp. 8–9, Figure 6 · p. 9

DART TRACE

  • Retaining Fine and Coarse KV banks fixes peak memory at 42.67 GiB versus 38.40 GiB for Full, while DART and TRACE computation takes 0.694% of total time.Despite initialization, memory-processing, and dynamic-access overhead, selective Fine access reduces total time versus Full across evaluated budgets.Source: DART TRACE · p. 9, Table 5

5 LIMITATIONS

ViMoD’s efficiency is bounded by initialization and storage overhead, evidence-access constraints, and unverified generalization beyond the evaluated settings.

5 LIMITATIONS

  • Coarse-memory initialization and retaining both Coarse and Fine KV add work and storage requirements.Short responses or sustained access to broad visual evidence may limit savings that offset initialization and dynamic-access overhead.Source: §5 · p. 9
  • C2F granularity, fixed routing intervals, and teacher supervision constrain which evidence TRACE can access and when.Recall may activate unnecessary group members, while missed groups or delayed updates can leave useful evidence inaccessible.Source: §5 · p. 9
  • Generalization across backbones and robustness to alternative decoding configurations remain unverified.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents ViMoD as coordinating content-adaptive visual memory with selective evidence access for multi-step reasoning. In the evaluation, the design improves accuracy under tight visual budgets and reduces total inference time.

6 CONCLUSION

  • ViMoD enables efficient multi-step visual reasoning through content-adaptive memory and selective evidence access.Source: §6 · p. 10
  • DART organizes recoverable visual detail, while TRACE learns to anticipate evidence needs from decoding history.Source: §6 · p. 10
  • The design improves accuracy under tight visual budgets and reduces total inference time in the authors’ evaluation.Source: §6 · p. 10

A METHOD DETAILS AND DERIVATIONS

The appendix details ViMoD’s capacity-constrained grouping and pooling, temporal-memory parameterization, and evidence-routing procedure. It also derives group-capacity sensitivity under a fixed total allocation.

A METHOD DETAILS AND DERIVATIONS

  • The appendix derives capacity-allocation and temporal-memory properties, specifies discrete grouping surrogates and learning objectives, and summarizes evidence access.Source: A METHOD DETAILS AND DERIVATIONS · p. 16

A.1.1 CAPACITY-CONSTRAINED ASSIGNMENT AND POOLING

  • Fine tokens are assigned by descending confidence to their highest-scoring group with remaining capacity, filling all groups because total capacity equals the Fine-token count.Training uses a finite-iteration Sinkhorn approximation for relaxed marginals; the separate confidence-ordered discrete assignment is not an exact optimizer, and the pooling surrogate retains hard assignments while passing relaxation gradients.Source: §A.1.1 · p. 16

A.1.2 GROUP-CAPACITY SENSITIVITY

  • Under the stated interior-capacity assumptions, the allocation Jacobian is symmetric positive semidefinite with rank M − 1, and raising one group’s score alone raises its capacity while lowering every other group’s.The capacity predictor receives each region’s marginal loss relative to a weighted regional mean.Source: §A.1.2 · p. 17

A.2.1 PARAMETERIZATION AND HISTORY

  • TRACE’s temporal state combines learned, input-dependent decay and write maps across branches, so past writes contribute through retention factors and a learned mixture.Each state coordinate is a convex combination of historical write coordinates and its zero initial value.Source: §A.2.1 · pp. 17–18

A.2.2 EVIDENCE POLICY AND DECODING ALGORITHM

  • At routing steps, TRACE compares retained and proposed evidence in a shared region-key space and gates whether to HOLD or UPDATE.Before generation, ViMoD switches from full visual KV to Coarse KV plus initially selected Fine KV, while retaining the full Fine KV bank for later recall.Source: §A.2.2 · p. 18

A.3 TRAINING OBJECTIVES

The training objectives distill compact Coarse memory, supervise evidence sets and routing decisions, and optimize TRACE’s policy using correctness and Fine-token residency. Training draws on broad, evaluation-disjoint task collections.

A.3.1 COMPACT MEMORY DISTILLATION

  • DART is trained to match full-visual teacher predictions on task-verified responses under identical teacher-forced prefixes.The objective also supervises response tokens by cross-entropy and distills the full-visual distribution into the Coarse-memory model.Source: §A.3.1 · p. 18

A.3.2 SUPERVISED SET AND GATE LEARNING

  • TRACE set training labels Fine cells with positive-area overlap with teacher evidence boxes, and gate targets UPDATE when consecutive annotated sets differ, otherwise HOLD.The set loss is class-balanced over positive and negative regions; joint training adds class-balanced gate cross-entropy.Source: §A.3.2 · p. 19

A.3.3 POLICY OPTIMIZATION AND ONLINE SUPERVISION

  • Policy optimization weights gate and region actions by correctness, while success-conditioned Fine-residency cost directly weights region log-probabilities without imposing a hard accuracy or computation constraint.Online teacher boxes specify a working set for approximately the next 16 student tokens; sampled trajectories also use RLOO correctness advantages.Source: §A.3.3 · p. 19

B.1 TRAINING DATA SOURCES

  • DART’s 57,420 trajectories and TRACE’s 80,000 SFT examples span reasoning, natural images, documents, charts, tables, OCR, and localization; the 4,800 RL questions are a task-balanced SFT subset.All training data are disjoint from every evaluation set.Source: §B.1 · p. 20

B.2 ARCHITECTURE AND OPTIMIZATION SETUP

ViMoD’s training keeps the Qwen3-VL-4B-Instruct backbone frozen while training lightweight DART and TRACE modules in separate compression, supervised, and reinforcement-learning stages.

B.2 ARCHITECTURE AND OPTIMIZATION SETUP

  • DART and TRACE add 0.0546% of the frozen backbone’s parameters; DART is trained independently at 4:1 and 9:1, while all TRACE stages use 9:1 compression.The two DART configurations share data, as do the two TRACE SFT stages; the RL pool is a subset of TRACE SFT.Source: §B.2 · pp. 20, 22, Table 6 · p. 21, Figure 7 · p. 21

B.3.1 DART DISTILLATION

  • DART starts from regular-grid mean pooling and is distilled using teacher-forced token distributions from the full-context backbone, supervising at most the first 64 answer tokens.A capacity-constrained Sinkhorn surrogate supplies cross-group gradients; training uses token cross-entropy and KL distillation.Source: §B.3.1 · p. 22

B.3.2 TRACE SUPERVISED TRAINING

  • TRACE’s 80,000-example SFT set is annotated by Qwen3.8-27B, with routing targets aligned every 16 tokens and answer, format, and coordinate checks.The Set stage uses class-balanced set BCE; the Joint stage adds class-balanced gate cross-entropy.Source: §B.3.2 · p. 22, Table 8 · p. 22

B.3.3 TRACE REINFORCEMENT LEARNING

  • TRACE reinforcement learning uses a correctness-filtered, task-balanced pool of 4,800 questions and samples 16 trajectories per question without a critic or PPO clipping.The online teacher proposes complete working sets without reference answers or future student text; exploration samples a shared gate bias.Source: §B.3.3 · pp. 22–23

C.1 BENCHMARKS AND METRICS

The evaluation sections specify benchmark coverage and protocols, define visual-budget calibration and attention-cost accounting, and describe efficiency and diagnostic measurements.

C.1 BENCHMARKS AND METRICS

  • The suite covers eight reasoning benchmarks and eight general-understanding or document–chart QA benchmarks, with official scoring rules and greedy decoding shared across compared methods.The main MMMU-Pro benchmark uses a 4,096-token output limit; the efficiency evaluation uses 8,192.Source: §C.1 · p. 23
  • Greedy decoding with zero presence penalty differs from the model-card recommendations, limiting direct comparability; repetition and truncation effects on the reported MMMU-Pro gap were not isolated.The Full baseline is 30.92% on MMMU-Pro under this protocol, versus 53.2% in the backbone technical report; normalized results use the measured Full baseline.Source: §C.1 · p. 23

C.2 VISUAL-BUDGET CALIBRATION

  • A scalar offset to TRACE’s learned boundary is calibrated on a moderate sample to match target average total visual occupancy, then fixed for evaluation.The persistent Coarse fraction bounds feasible occupancy, and closed-loop occupancy need not vary monotonically with the offset.Source: §C.2 · pp. 23–24

C.3.1 VISUAL OCCUPANCY AND GENERATION COST

  • Visual occupancy averages persistent Coarse and active Fine tokens across generation passes; the arithmetic model equates visual-attention work for equal occupancy and generation length.The model excludes kernel utilization and memory traffic, and compares modeled savings against initialization and routing overhead.Source: §C.3.1 · p. 24

C.3.2 ATTENTION ACCOUNTING

  • Attention accounting includes query–key and attention–value products involving active visual KV entries, while excluding projections, MLPs, visual encoding, and non-attention router computation.Occupancy and attention work are averaged per example and then across examples; cumulative cost is normalized to the task’s uncompressed baseline.Source: §C.3.2 · p. 24

C.4 EFFICIENCY MEASUREMENT PROTOCOL

  • Efficiency measurements use MMMU-Pro on one H200 GPU with batch size 8 and an 8,192-token cap, reporting complete-run wall time and decode throughput.Analytical TFLOPs include vision encoding, prefill, decoding, and method computation, but exclude elementwise operations and data movement.Source: §C.4 · pp. 24–25

C.5 DIAGNOSTIC SETTINGS

  • Visual-demand variation is measured as turnover of highly attributed tokens across 16 response windows; MathVerse examples are grouped by this value for Fig. 2(a).Attribution is gradient-times-input rather than attention weight, and heatmap display scaling is not used to compute the variation measure.Source: §C.5 · p. 25
  • The FastV diagnostic uses 300 MathVerse examples, while the annotation-guided comparison plots accuracy relative to Full and interpolates budgets at 70% of Full accuracy.The FastV setup uses an 8,192-token cap; the annotation-guided comparison uses Qwen3.8-annotated stage-specific regions on MMMU-Pro.Source: §C.5 · p. 25

D QUALITATIVE ANALYSIS

The circuit example illustrates Fine-token access changing across reasoning stages while Coarse tokens persist; the accompanying prompts specify offline annotation and forward-looking online evidence prediction.

D QUALITATIVE ANALYSIS

  • In the circuit example, ViMoD accesses resistor labels, reasons with Coarse tokens, then retrieves voltage labels while 35 Coarse tokens remain active.The sequence illustrates access behavior but does not isolate the causal effect of individual accesses; figure annotations mark active Fine-token counts and routing decisions.Source: D QUALITATIVE ANALYSIS · p. 26, Figure 8 · p. 26

PROMPT 1: OFFLINE EVIDENCE ANNOTATION (SFT)

PROMPT 2: ONLINE EVIDENCE PREDICTION (RL)

Loading 2610.12060v1…