PaperlayerView exact revision on arXiv

Source-linked AI summary

FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping Wang, Ding Wang

arXiv:2608.22883v1cs.CV

TL;DR

Multimodal speculative decoding needs visual evidence that matches each decoding state, but fixed visual interfaces do not provide that adaptability. FOVEA retrieves bounded, state-conditioned evidence from reusable visual memory and applies it through cache-compatible hidden-state correction. Across three backbones and nine benchmarks, it improves acceptance and speed, reaching up to 2.13× speedup over autoregressive decoding.

  • Problem

    Visual demand varies across decoding states, while fixed visual interfaces assume similar amounts or forms of evidence and can create under- or over-supply.

  • Method

    FOVEA caches projected image evidence once, retrieves a bounded state-conditioned subset using cumulative mass, and gates its readout into the current scoring representation without changing the historical KV cache.

  • Results

    Across three backbones and nine benchmarks, FOVEA achieves the highest average acceptance lengths and average speedups of 1.68×, 1.66×, and 1.60×.

  • Takeaways & Limitations

    State-conditioned evidence retrieval provides a stronger overall trade-off than fixed-cardinality alternatives while preserving KV-cache reuse.

Abstract

from arXiv · show

Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically condition the drafter on a fixed visual interface, such as a predefined visual-token budget or a static compressed representation. However, our controlled visual-budget analysis shows that visual demand varies substantially across tasks and decoding stages, which means more visual input is not always beneficial. Actually, insufficient evidence may weaken visual grounding, while excessive context adds overhead and may disrupt drafting. We propose FOVEA (Focused On-demand Visual Evidence Adaptation), a cache-friendly approach that builds a reusable visual memory and dynamically retrieves a bounded subset for a draft state. A cumulative-mass rule determines both how many and which entries are selected. The selected entries are aggregated into a visual readout and fused with the current draft hidden state through a lightweight gated residual correction. Rather than inserting visual tokens into the autoregressive context, the correction modifies only the representation passed to the language-model head. Experiments across multiple vision-language backbones and multimodal benchmarks show that FOVEA improves draft acceptance and end-to-end decoding speed, achieving up to $2.13\times$ speedup over autoregressive decoding. These results demonstrate that state-conditioned evidence retrieval is an effective alternative to reusing a fixed visual representation throughout multimodal generation.

Introduction

Multimodal speculative decoding must balance inexpensive drafting with enough image evidence for visual grounding. FOVEA addresses state-varying visual demand by retrieving selected evidence from reusable memory while preserving cached context, improving acceptance and decoding speed.

  • Visual grounding may depend on small objects, OCR text, spatial relations, or chart elements among many projected visual tokens.
  • Existing multimodal draft methods expose many visual tokens or reuse compressed representations through a fixed visual interface.
  • Preferred visual budgets vary substantially within images and across decoding states, challenging the assumption that one visual budget suits all states.
  • FOVEA constructs reusable image-specific visual memory and retrieves a dynamic, state-conditioned subset without modifying the cached token context.
  • FOVEA applies gated hidden-state correction to adapt draft-node scoring while retaining the autoregressive context and historical KV cache.
  • Experiments across three vision-language backbones and nine multimodal benchmarks report improved draft acceptance and end-to-end speed, with up to 2.13× speedup over autoregressive decoding.

Related Work

Prior multimodal acceleration methods commonly fix or precompress visual information before decoding. FOVEA instead treats visual evidence as reusable memory that can support different decoding states while retaining KV-cache reuse.

  • Vision-language speculative decoding must approximate both language continuation and visual grounding, making acceptance sensitive to visual evidence.
  • Recent methods use vision-aware adaptation, target-feature injection, multimodal training, or visual compression to improve multimodal drafting.
  • Most visual-token acceleration methods select a compact representation before decoding or reuse one interface throughout generation.
  • FOVEA stores image evidence once, retrieves state-conditioned subsets, and corrects the current scoring representation instead of inserting new context tokens.

Method

FOVEA builds a reusable visual memory, selects a bounded cumulative-mass subset for each draft state, and applies the resulting evidence through cache-compatible hidden-state correction.

  • Visual-budget analysis: FOVEA addresses non-monotonic, state-varying visual-budget sensitivity by selecting evidence dynamically rather than using one fixed interface.
  • Visual memory: The target-side visual encoder and projector produce projected visual tokens that FOVEA stores once as an image-specific memory.
  • Visual memory: The same memory is shared across speculative iterations and draft-tree nodes, avoiding repeated full-sequence processing while retaining fine-grained evidence access.
  • State-conditioned retrieval: Each enabled draft node forms a query from its hidden state and depth, scores valid memory keys, and retrieves a bounded subset using cumulative probability mass.
  • Cache-friendly correction: The retrieved readout is fused with the current hidden state through a gated residual correction, and the language-model head produces the corrected logits.
  • Cache-friendly correction: Because only the current scoring representation changes, the draft sequence, attention structure, position identifiers, and historical KV cache remain unchanged during tree expansion and verification.

Experiment

Experiments across three vision-language backbones and nine multimodal benchmarks show that FOVEA improves decoding speed and acceptance through state-conditioned retrieval and cache-compatible correction. Ablations and analyses indicate that visual correction, adaptive retrieval cardinality, and cache reuse are central to its performance.

  • Main Results: FOVEA achieves the strongest average speedup for all three backbones: 1.68×, 1.66×, and 1.60×.These correspond to relative improvements of 11.3%, 13.7%, and 13.5% over the best competing averages.
  • Main Results: FOVEA ranks first in 21 of 27 backbone–benchmark combinations and remains above 1× average speedup in every evaluated combination.The wins span perception-heavy and reasoning-intensive workloads.
  • Main Results: FOVEA obtains the highest average acceptance length for every backbone, with τ values of 3.960, 3.882, and 3.925.It improves over the strongest competing acceptance lengths by 8.1%, 6.0%, and 5.6%, respectively.
  • Ablation and Analysis: Removing visual correction reduces macro-average speedup and acceptance length by 30.7% and 27.2%, respectively.Removing depth embedding or the correction gate causes smaller but consistent declines; without distribution distillation, speedup falls to 1.21× and acceptance length to 2.50.
  • State-Conditioned Visual Analysis: Across MMT-Bench, OCRBench, and ChartQA, five matched retrieval controls reduce macro-average speedup by 0.157–0.247 and acceptance length by 0.193–0.225.The controls are Random, Static Image, Query-agnostic, State-only, and Dense Retrieval.
  • State-Conditioned Visual Analysis: FOVEA improves speedup by approximately 5.1% on OCRBench and nearly 3% on MMT-Bench and ChartQA, while fixed budgets behave task-dependently and non-monotonically.Acceptance-length changes are modest, reaching 0.91% on MMT-Bench and 0.50% on ScienceQA.
  • State-Conditioned Visual Analysis: The retrieved focus shifts from clothing to the head and then the left-side hose as the draft state evolves during one decoding trajectory.This qualitative trajectory shows different predictions retrieving different spatial evidence.
  • Cache-Friendly Efficiency Analysis: Reusing the historical cache reduces draft time from 19.90 to 16.49 ms, a 17.1% reduction, and increases speedup from 1.764× to 1.855×.Acceptance length remains nearly unchanged, while dynamic insertion raises draft time to 20.86 ms and lowers speedup to 1.532×.

Conclusion

FOVEA adapts visual evidence to decoding states through dynamic retrieval and gated correction, preserving the historical KV cache. Across three backbones and nine benchmarks, it improves acceptance and speed over fixed-cardinality alternatives.

  • FOVEA retrieves different amounts and contents of evidence from reusable visual memory using a cumulative-mass rule, then updates only the current latent representation.The gated correction preserves the historical KV cache.
  • 1.68×, 1.66×, and 1.60× average speedups are achieved across three backbones, alongside the highest average acceptance lengths among evaluated baselines.
  • Dynamic retrieval provides a stronger overall trade-off than fixed-cardinality alternatives.

Visual-Budget Diagnostic Details

The diagnostic evaluates whether visual evidence needs remain stable across decoding states, using budget sweeps, state-level metrics, trajectory comparisons, and attention-trace analysis. Results show strongly varying and often increasing visual demand later in generation.

  • Visual-Budget Setup: The diagnostic varies visual input across five budget levels to test whether a globally optimal compression ratio remains stable across decoding states.The 75% condition is the default; 0% and 100% represent no visual tokens and full visual access.
  • State-Level Metrics: Preferred budget is the smallest tested ratio attaining maximum accept length, providing a deterministic tie-break and measuring least evidence needed for best observed drafting quality.
  • State-Level Metrics: 25% visual budget is preferred for 56.0% of states, compared with 4.0% for the default 75% budget and 2.9% for full access.
  • Trajectory Analysis: On OCRBench, mean preferred budget rises from 0.150 Early to 0.237 Late, while mean accept-length range increases from 0.699 to 0.915 tokens and sensitivity from 42.2% to 48.8%.
  • Trajectory Analysis: Across valid samples, 62.2% show larger Late mean preferred budgets and 55.1% show larger Late Top-16 Jaccard distance.The attention diagnostic measures changes in selected Top-16 membership between adjacent steps.
  • Diagnostic Caveat: The focus-movement diagnostic captures Top-16 token-set membership changes, not attention-weight, spatial-distance, or image-region-area changes.It is complementary evidence of evolving visual focus rather than causal attribution.

Training Setup

FOVEA training uses backbone-specific draft models with frozen target VLMs and precomputed target artifacts. Ablations show that broader visual correction adds draft-side cost for modest acceptance gains, while alternative retrieval interfaces reduce macro metrics.

  • Correction Ablation: Extending correction from depth ≤0 to ≤3 raises macro-average accept length from 2.423 to 2.507 but lowers speedup from 1.237× to 1.190× and faster-sample fraction from 75.5% to 68.2%.
  • Training Setup: FOVEA trains a separate draft model for each target backbone while keeping the target VLM frozen and using precomputed target artifacts as supervision.
  • Training Data: The LLaVA and Qwen2.5-VL draft models use 57,671 target-artifact samples each, including broad instruction data and six task-oriented sources.
  • Retrieval Setup: Visual-memory retrieval uses fixed cardinality K_r = 16 during training and inference, so state conditioning changes selected entries rather than the number scored or aggregated.
  • Retrieval Ablation: All five alternative retrieval interfaces reduce both macro-average speedup and acceptance length relative to Full FOVEA.State-only Top-16 is the closest matched selection control, while static and query-agnostic retrieval show less robust cross-task trade-offs.

Implementation and Training Details

The implementation defines backbone-specific supervision states and a lightweight visual-correction module, with standardized training and evaluation configurations. The target remains frozen, and ablation results are reported as repeated-inference changes from Full FOVEA.

  • Supervision States: Draft supervision uses the first draft-block output, while the pre-correction draft state comes from the final draft block and the target state from the last target layer.
  • Supervision States: Intermediate target features are selected token-adaptively from early target layers using the lowest attention entropy for hidden-state alignment.LLaVA searches layers 1–10, while Qwen2.5-VL-7B searches layers 1–9.
  • Visual Correction: The correction module concatenates the draft state and visual readout, producing a hidden correction and a scalar broadcast gate through two-layer networks.For every backbone, d_k = d_u = d_h; each network uses a 512-dimensional hidden layer and SiLU activation.
  • Training Configuration: All reported training configurations keep the target VLM frozen, with batch size represented as per-device size, gradient accumulation, and global size.
  • Evaluation Configuration: Table 7 reports absolute changes in speedup and average acceptance length from Full FOVEA over three repeated inference runs using the same checkpoint.Higher values are better, and variability reflects repeated inference rather than independent training seeds.
Loading 2608.22883v1…