PaperlayerView exact revision on arXiv

Source-linked AI summary

CONDUIT: A Unified Residual-Stream Restoration Framework for KV Cache Reuse in Vision-Language Models

Pengan Chen, Kaisheng Zheng, Liang Hong, Lixia Yi, Jiyue Jiang, Jiayang Chen, Yixuan Wang, Yimin Fan, Xinyuan Liu, Jiayi Li, Zhanqiu Zhang, Yiwen Guo, Yu Li

arXiv:2609.05821v1cs.CLcs.CV

TL;DR

Changed query prefixes make exact KV-cache reuse invalid even when visual content recurs, and choosing which stale tokens to refresh is difficult. CONDUIT combines norm-aware token ranking with image-level relevance amplification in a training-free policy. At a 10% refresh budget, it achieves 97.0–99.5% of the corresponding full-prefill five-dataset average and leads budgeted baselines on average.

  • Problem

    When visual content recurs under a changed query prefix, exact-prefix reuse fails, and selective recomputation depends on identifying which stale visual tokens to refresh.

    Source: §1 · p. 1

  • Method

    CONDUIT ranks cached visual tokens using cached-key query attention and a pre-output cached-value-norm proxy, then amplifies scores for query-relevant images before one global selection.

    Source: §3 · p. 4, §5 · p. 9

  • Results

    At a 10% refresh budget, CONDUIT achieves 97.0–99.5% of the corresponding full-prefill five-dataset average and leads budgeted baselines on average.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The same refresh principle covers single-image reuse, where the image coefficient is one, and multi-image reuse with cross-image relevance amplification.

    Source: §5 · p. 9

    Video, streaming observations, and long-horizon agentic workflows remain untested.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) often answer new questions about recurring visual content, where reusing the key-value (KV) cache can avoid re-encoding expensive visual prefixes. Exact-prefix reuse, however, fails when the same visual content appears under a changed prefix. Selective recomputation can recover quality under a small visual-token budget, but only when the right stale tokens are refreshed. Raw-attention selection can waste budget on high-attention tokens with small value-norm proxy scores and on query-irrelevant images. To address these failure modes, we propose CONDUIT, a training-free refresh policy that unifies single- and multi-image reuse as residual-stream restoration. Building on norm-weighted attention, CONDUIT ranks cached visual tokens using cached-key query attention and an accessible pre-output cached-value-norm proxy, then applies empirical image-level relevance amplification before one global selection. With one image, the coefficient is one and the rule reduces to intra-image token selection. The method preserves model architecture and weights, adding only a single query-conditioned scoring pass at inference. At a 10% refresh budget, CONDUIT achieves 97.0-99.5% of the corresponding full-prefill five-dataset average across three VLM backbones and leads budgeted methods on average; on the MMLongBench-Doc latency subset, it uses 13.5% of full-prefill FLOPs and achieves a 2.99x time-to-first-token speedup.

1 Introduction

CONDUIT treats shifted-prefix visual-cache reuse as residual-stream restoration, selecting stale tokens using query attention, a cached-value-norm proxy, and image-level relevance. At a 10% refresh budget, it retains 97.0–99.5% of the corresponding full-prefill five-dataset average and leads budgeted methods on average.

1 Introduction

  • Changed query prefixes make exact-prefix KV reuse invalid even when visual content recurs, leaving the challenge of choosing which stale tokens to refresh.Repeated prefill is costly for long visual prefixes, so selective recomputation keeps most cached entries and refreshes only a small subset under a fixed budget.Source: §1 · p. 1
  • Raw attention can favor high-attention, low-value-norm tokens and distractor-image patches, spending shared refresh budget away from answer-bearing evidence.CONDUIT combines norm-aware token ranking with empirical image-level relevance amplification before spending the shared budget.Source: §1 · pp. 1–2
  • CONDUIT frames refresh as residual-stream restoration, combining cached-key query attention with cached value norm as a pre-output write-scale proxy.An empirical per-image coefficient amplifies scores for images read more strongly by the query; with one image, the coefficient is one and selection is intra-image.Source: §1 · p. 2
  • The training-free policy preserves model architecture and weights, using one query-conditioned scoring pass to refresh the highest-scoring visual tokens.Its unified formulation combines within-image norm-aware ranking and cross-image budget reweighting without learned parameters or weight changes.Source: §1 · p. 2, Figure 1 · p. 2
  • At 10% refresh, CONDUIT achieves 97.0–99.5% of the corresponding full-prefill five-dataset average and leads budgeted baselines on average.On the MMLongBench-Doc latency subset, it uses 13.5% of full-prefill FLOPs and achieves a 2.99× time-to-first-token speedup.Source: §1 · p. 3

2 Related Work

Prior work addresses cache compression, query-driven reuse, multimodal caching, and attention’s contribution to residual streams. The passages identify an open combination: norm-aware visual-token refresh under one query-conditioned budget shared across images.

2.1 KV Cache Compression

  • KV-cache compression retains important entries or adapts budgets, but commonly uses attention magnitude and treats the cache as one uniform sequence.Visual prefixes instead segment tokens by image, and raw attention does not expose the scale of a token’s transformed contribution.Source: §2.1 · p. 3

2.2 Query-Driven Cache Reuse

  • Query-driven cache reuse uses selective recomputation and lightweight scoring, but prior text-context methods leave visual-token ranking and cross-image budget allocation open.Yao et al. (2025) show partial recomputation can recover answer quality at a fraction of full-prefill cost.Source: §2.2 · p. 3

2.3 Multimodal KV and Visual Tokens

  • Multimodal cache reuse, inter-image attention, and visual-token pruning are studied in separate lines, leaving their combination in stale-cache refresh unresolved.The open combination is norm-aware token selection under one query-conditioned budget shared across images.Source: §2.3 · p. 3

2.4 Attention and the Residual Stream

  • Norm-weighted attention treats contribution as attention times transformed-value norm, motivating separation of how strongly a query reads a token from its potential write scale.CONDUIT applies an accessible pre-output cached-value-norm proxy to stale-cache refresh and pairs it with cross-image relevance amplification.Source: §2.4 · pp. 3–4

3 Method

CONDUIT treats visual-cache reuse as residual-stream restoration, ranking stale visual tokens by query readout and a proxy for their write scale. It combines token scores with image-level relevance amplification before global selection, with single-image reuse as the one-image limit of the same rule.

3 Method

  • CONDUIT ranks cached visual tokens by cached-key query attention and a pre-output value-norm proxy, then applies image-level amplification before global top-k selection.The same refresh rule covers single- and multi-image inputs.Source: §3 · p. 4, Figure 2 · p. 4

3.1 Problem Setup

  • For K images, CONDUIT chooses cached visual tokens to refresh under a visual-token budget, while new or edited text is processed normally.Source: §3 · p. 4, §3.1 · p. 4

3.2 Local Residual-Staleness Model

  • The local model ranks stale-token residual error using query readout and value-write displacement, relying on cached-key attention because estimating attention drift requires a fresh execution.Its diagonal-risk surrogate drops cross-token terms under the assumption that stale displacements are not consistently aligned.Source: §3.2 · pp. 4–5

3.3 Token-Level Throughput Score

  • CONDUIT multiplies cached-key query attention by cached pre-output value norm, then aggregates query- and head-level signals across decoder layers, using all layers by default.Under the relative-ranking assumption, the proxy favors tokens that are strongly read and have larger pre-output write scale.Source: §3.3 · p. 5

3.4 Image-Level Reweighting

  • Image-level mean cached-key attention supplies empirical relevance amplification, interpolating between uniform and attention-proportional weighting without fixed per-image quotas.Images with above-average query readout receive greater pressure in the shared ranking.Source: §3.4 · p. 5

3.5 Composite Refresh Policy

  • A single global top-k selector recomputes the highest-scoring tokens; when K=1, the image coefficient is one and the policy reduces to intra-image selection.The realized per-image split adapts to token-score distributions while the image coefficient controls each image’s pressure on the shared budget.Source: §3.5 · pp. 5–6

4 Experiments

At a 10% refresh budget, CONDUIT preserves nearly all of full-prefill quality across the main shifted-prefix evaluation and reduces serving cost. Experiments also test its single-image limit, score components, and ranking diagnostics.

4.1 Setup

  • The cache-reuse protocol holds cached-state materialization fixed and refreshes exactly ⌊rNvis⌋ visual entries, isolating the refresh-selection rule in reported gaps.The evaluation uses three open VLM backbones, five main shifted-prefix datasets, and four separate K=1 benchmarks.Source: §4.1 · pp. 6–7

4.2 Main Results

  • At a 10% refresh budget, CONDUIT leads budgeted methods on every dataset–backbone pair, reaches 97.0–99.5% of full-prefill average, and improves over ProphetKV by about 1.7 points on average.At 5%, it has the best five-dataset average on all three backbones and ranks first in 13 of 15 dataset–backbone cells.Source: §4.2 · p. 7, Figure 3 · p. 7
  • In the K=1 setting, c1 = 1 reduces CONDUIT to intra-image throughput selection, and a 5% refresh remains close to the full-prefill average.Table 2 uses official benchmark metrics, with no visual tokens recomputed at r = 0 and the top-throughput 5% refreshed at r = 0.05.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Serving Latency

  • At r = 0.10, CONDUIT reduces prefill cost from 90.36 to 12.24 TFLOPs and TTFT from 0.519 to 0.174 s on the latency subset.The evaluation averages three backbones and 4k/8k/16k contexts; its FLOP reduction exceeds its wall-clock speedup because TTFT includes other costs.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 Ablations

  • Across Qwen2.5-VL-3B benchmarks at r = 0.10, the average rises from 48.53 at λ = 0 to 49.33 at λ = 1; intermediate-strength variation is at most 0.38 points.The paper uses λ = 1 across models and datasets without per-benchmark tuning.Source: §4.4 · p. 8, Table 4 · p. 8
  • Removing value-norm weighting causes the larger drop on two of three backbones and on average, while removing image-level amplification consistently lowers quality.The ablation compares query-attention-only ranking and intra-image throughput-only ranking against the composite score at r = 0.10.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 Analyses

  • The controlled projected-error ranking reaches Spearman 0.79–0.89, while an audit of 23 divergent cases finds limited page coverage in 15 and possible unmodeled attention drift in four.The other four audited cases had near-tied evidence and distractor page scores.Source: §4.5 · pp. 8–9

5 Conclusion

CONDUIT frames shifted-prefix visual KV-cache reuse as residual-stream restoration, unifying within-image token ranking and cross-image budget reweighting. Across three backbones, it retains most full-prefill quality at a 10% refresh budget while adding one query-conditioned scoring pass and no architectural modifications.

5 Conclusion

  • CONDUIT treats shifted-prefix visual KV-cache reuse as residual-stream restoration, combining within-image norm-aware token ranking with cross-image relevance amplification.With one image, the coefficient is one and the same intra-image rule applies.Source: §5 · p. 9
  • At a 10% refresh budget, CONDUIT achieves 97.0–99.5% of full-prefill average and a 2.99× TTFT speedup on the MMLongBench-Doc latency subset.The evaluation spans three open VLM backbones and nine visual benchmarks; the latency result uses a context-stratified subset.Source: §5 · p. 9
  • The training-free policy leaves model architecture unchanged and adds one query-conditioned scoring pass.It uses cached value norm as a pre-output proxy for write scale.Source: §5 · p. 9

Limitations

The evaluation and practical benefits of CONDUIT are bounded by untested settings, serving conditions, and the cached-value-norm proxy.

Limitations

  • Video, streaming observations, and long-horizon agentic workflows remain untested; evaluation covers static-image and document shifted-prefix reuse.Source: Limitations · p. 9
  • Latency measurements use a compact nano-vllm-derived backend at batch size one and do not cover continuous-batching production servers.Source: Limitations · p. 9
  • Cached value norm is a practical ranking proxy, and the selector does not explicitly model key- or query-state-induced attention drift.Source: Limitations · p. 9
  • Benefits are smaller when the reusable visual prefix is short relative to surrounding text, and cached visual K/V storage costs remain.Source: Limitations · p. 9

Ethics Statement

The paper reports its research and dual-use scope, describes how the refresh mask selects visual positions, and specifies the scoring pass and shared hyperparameters.

Ethics Statement

  • The work uses public VLM checkpoints and benchmarks, trains no new model, involves no human subjects, and collects no new data; reduced serving costs also lower the cost of generating multimodal content at scale.Source: Ethics Statement · p. 9

A.1 Refresh-Mask Algorithm

  • Algorithm 1 builds a binary visual-token refresh mask: selected positions are recomputed for the serving request, while other visual K/V entries are reused.Source: §A.1 · p. 14

A.2 Scoring Pass and Hyperparameters

  • Scoring uses a query-conditioned pass, with cached-key attention averaged over the query span and heads; main results average scores over all decoder layers and use λ = 1 without per-benchmark tuning.Source: §A.2 · p. 14

A.3 Single-Image Evaluation Protocol

The single-image protocol tests CONDUIT’s intra-image throughput rule while holding the cached image state fixed and varying the refresh budget.

A.3 Single-Image Evaluation Protocol

  • With one image, c1 = 1; the protocol caches each image under a generic instruction, then serves benchmark questions with zero or 5% of visual tokens refreshed.Source: §A.3 · p. 14

B Complete Experimental Results

The section describes the complete shifted-prefix results, reports budget–quality and latency patterns, and outlines the analytical surrogate analysis.

B Complete Experimental Results

B.1 Budget–Quality Curves

  • At r = 0.10, CONDUIT recovers 97.0–99.5% of full-prefill quality and exceeds every budgeted baseline in average score at r = 0.05 and 0.10 across all three backbones.Source: §B.1 · p. 15

B.2 Serving Latency by Backbone and Context

  • Across nine backbone/context settings, CONDUIT stays within 0–11 ms of ProphetKV; at 16k tokens, scoring takes 68 ms, or 26.2% of profiled TTFT and 7.3% of the full-prefill profile.Source: §B.2 · p. 15

C Surrogate Analysis

  • The surrogate analysis motivates throughput scoring from a local staleness model, connects global top-k selection to an additive surrogate, and examines image-coefficient normalization, limiting cases, and empirical support.Source: C Surrogate Analysis · p. 16

C.1 From Local Error to a Ranking Surrogate

The section turns local residual error into a query-conditioned ranking surrogate, then extends selection across images with empirical relevance amplification. It also describes diagnostics for the surrogate and its staleness proxy.

C.1 From Local Error to a Ranking Surrogate

  • CONDUIT ranks cached-key attention times value-write scale, omitting attention drift because estimating it requires a fresh execution.The local decomposition separates value-write error from attention drift; the deployed score uses the first term.Source: §C.1 · p. 16
  • Under a diagonal approximation, the top-k tokens by the surrogate score minimize the stated ranking objective.The exact staleness displacement is unavailable at scoring time, so cached pre-output value norm serves as its ranking proxy.Source: §C.1 · pp. 16–17

C.2 Empirical Image-Level Relevance Amplification

  • The image coefficient reweights token scores using each image’s mean cached-key attention; ablations show a consistent gain, with limited variation across λ = 0.5, 0.75, and 1.0.Global top-k still enforces the total budget, and when K = 1 the coefficient is one, reducing the rule to intra-image selection.Source: §C.2 · pp. 17–18

D.1 Monte Carlo Staleness Estimator

  • The Monte Carlo estimator is unbiased for the isotropic residual-error functional, has variance O(M^-1), and uses M = 5 paired direction samples per token.The two-query proxy measures prompt-specific pre-output value drift while holding visual content fixed.Source: §D.1 · p. 19

D.2 Analysis Protocol

The diagnostics use captured attention and value vectors from multi-image prompts, with larger samples for several extensions. A qualitative example examines retrieving a table entry from a multi-page document.

D.2 Analysis Protocol

  • The shared diagnostic protocol samples 16 LongDocURL multi-image prompts averaging 3.7 images, captures Qwen2.5-VL-3B at layers 7, 14, and 24, and uses M = 5 samples per token for Table 13.Tables 11 and 12 use the same captures.Source: §D.2 · p. 19
  • Extended diagnostics use 64 prompts for surrogate correlation, 64 pages, slides, and natural images for scale similarity, and 64 shifted-prefix pairs for cached-norm stability.The scale-similarity diagnostic compares within-image squared-throughput quantiles before and after normalization by g_j.Source: §D.2 · pp. 19–20
  • The qualitative LongDocURL example asks for a ratio tied to a specific period and marriages row, requiring the target table tuple to be found among four document pages.The accompanying table reports generated answers and their relation to the target entry.Source: Figure 4 · p. 20, Table 15 · p. 20

D.3 Diagnostic Results

Diagnostics support query-attention ranking and the cached value-norm proxy in the evaluated setting, while also identifying evidence-allocation ambiguities and answer discrepancies. The evaluation covers static-image and document-page reuse, not dynamic settings.

D.3 Diagnostic Results

  • Query-attention rules reduce projected residual error roughly 5× to over 500× versus random selection, and surrogate-ordering correlation ranges from 0.79 to 0.89 across budgets.Value-norm-only selection remains close to random in the controlled setting.Source: §D.3 · p. 20
  • Across 64 LongDocURL shifted-prefix pairs, cached value-norm rankings have layer-averaged Spearman correlations of 0.996 and 0.990 for Qwen2.5-VL-3B and Qwen2.5-VL-7B, respectively.Removing the value-norm factor drops performance by 0.89, 1.35, and 0.53 points across the three backbones.Source: §D.3 · p. 20
  • Attention, key-norm, and bottom-value-norm criteria induce distinct token rankings, with no systematic positive enrichment among their overlaps.Attention–low-value overlap is near the independent-decile baseline in middle and deep layers.Source: §D.3 · p. 20
  • Actual and explicit coefficient-based reweighting remain within a few percent in the cross-image diagnostic; task-level evidence for the coefficient comes from the end-to-end ablation.The comparison is against Global TopK without image reweighting.Source: §D.3 · pp. 20–21
  • Normalizing squared-throughput profiles by g_j reduces mean cross-type Kolmogorov–Smirnov distance from 0.66 to 0.24 across document pages, slides, and natural images.Normalized distances range from 0.18 to 0.26 by type.Source: §D.3 · p. 21

D.4 Evidence Allocation, Failure Audit, and Case Study

  • In the closest attention-gap quartile, CONDUIT stays within 0.5 points of full prefill and assigns 26.1% of budget to evidence pages, versus 28.5% for raw-attention top-k.From Q2 through Q4, CONDUIT allocates progressively more evidence-page budget than raw-attention top-k as the page-level gap widens.Source: §D.4 · p. 21
  • Among 23 differing-answer cases, 15 gave the answer-bearing page nontrivial budget but refreshed at most 20% of its tokens; five recovered at a 20% budget.Four cases had near-tied evidence and distractor page scores, while four were qualitatively consistent with unmodeled attention drift.Source: §D.4 · p. 21
  • In one four-page LongDocURL example at a 10% refresh ratio, CONDUIT returned 1.97, matching full prefill, while the other budgeted methods returned distractor answers.Cache reuse, CacheBlend, MPIC, and KVShare returned 2.62; ProphetKV returned 0.95.Source: §D.4 · p. 21
Loading 2609.05821v1…