PaperlayerView exact revision on arXiv

Source-linked AI summary

From Patching to Pruning Visual Computation in Vision Language Models

Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Shaoyi Huang, Pu Zhao, Yanzhi Wang

arXiv:2610.03389v1cs.CVcs.LG

TL;DR

Visual tokens make VLM inference costly, and token-pruning methods can miss tokens whose importance changes across layers. P2P uses validation-guided activation patching to bypass selected visual computations while preserving the sequence. Across the evaluated models and datasets, it reduces computation while retaining most dense accuracy, though full-test accuracy drops can sometimes exceed the specified tolerance.

  • Problem

    Visual tokens are repeatedly processed through decoder layers, creating substantial inference cost, while token pruning based on early importance can discard tokens that become important downstream.

    Source: §1 · p. 1

  • Method

    P2P uses modality-conditioned activation prototypes and disjoint validation-set forward and backward sweeps to select visual-token projections for inference-time bypass.

    Source: §1 · p. 2

  • Results

    Across four models and eight datasets, P2P yields a 55% FLOPs reduction and 22% latency reduction while retaining 94% of dense accuracy.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    P2P’s layer-wise analysis indicates that visual processing is concentrated in task-sensitive intermediate layers, while substantial early- and late-layer visual computation is redundant.

    Source: §6 · p. 10

    On some datasets, full-test accuracy drops can exceed the specified tolerance, and task sensitivity varies.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-specific visual computation is unnecessary at many depths. We introduce Patch-to-Prune (P2P), inspired by Mechanistic Interpretability, a training-free framework that converts activation patching from a diagnostic tool into an inference-time computation bypass. P2P performs validation-guided forward and backward layer sweeps to identify decoder regions whose visual-token projection outputs can be replaced by fixed neutral proxy activation vectors within a user-specified accuracy tolerance. Unlike conventional token-pruning methods, P2P preserves the sequence length, token order, positional information, attention mask, and residual pathways, thereby pruning computation without removing tokens or modifying the pretrained model weights. We evaluate P2P on four VLMs from the Qwen2.5-VL and LLaVA families across seven multi-modal benchmarks using mutually disjoint calibration, validation, and test partitions. P2P at a 3% tolerance retains around 94% of dense accuracy while reducing FLOPs by 55%. Beyond these efficiency gains, our layer-wise analysis suggests that visual processing in VLMs is non-uniformly distributed across decoder depth: early and late layers often require little token-specific visual computation, whereas intermediate layers appear to perform most task-relevant visual integration, enabling later reasoning to rely largely on visual information already embedded in shared residual and textual representations. This makes P2P both an efficient inference framework and a causal lens into visual information processing in VLMs.

1 INTRODUCTION

Visual tokens make VLM inference costly, while early token-importance pruning can miss tokens that become important later. P2P uses activation patching to locate safe visual-computation bypasses, reporting consistent efficiency gains while retaining tokens and sequence structure.

1 INTRODUCTION

  • Earlier training-free methods such as FastV and PyramidDrop discard tokens using early attention or similarity, risking removal of tokens that matter in later layers.This motivates asking which layers actually process visual tokens.Source: §1 · p. 1
  • Visual-token projection outputs can be replaced in many early and late layers, but intermediate layers are markedly more sensitive to substitution.This suggests decoder redundancy is structured across depth rather than uniform.Source: §1 · p. 2
  • P2P uses calibration-set modality-conditioned prototypes and disjoint validation-set forward and backward sweeps to select visual computations replaceable within an accuracy tolerance.The selected safe layers define where inference can bypass token-specific visual computation.Source: §1 · p. 2
  • At inference, P2P bypasses selected visual-token projections while preserving token count, order, attention mask, and residual pathways, without changing pretrained weights or architecture.Visual embeddings remain in the sequence throughout the network.Source: §1 · p. 2
  • P2P reports consistent FLOP and latency reductions across four VLMs and seven multimodal benchmarks, with decoder redundancy concentrated near early and late layers.It also reports that P2P-aware fine-tuning can substantially recover accuracy lost under aggressive pressing.Source: §1 · p. 2

2 RELATED WORK

Prior VLM acceleration work largely reduces computation by reducing visual tokens. Activation patching offers a way to study internal computation, but ordinarily does not itself accelerate inference.

2 RELATED WORK

  • Figure 2 frames layer-wise sensitivity as cumulative forward and backward mean patching of image, text, and joint token streams on POPE.Source: Figure 2 · p. 3
  • Token-reduction methods include merging redundant tokens, progressively dropping tokens across decoder stages, and text-guided ranking with layer-dependent sparsification.PruMerge, PyramidDrop, and SparseVLM exemplify these approaches.Source: §2 · p. 3
  • Activation patching localizes component importance by replacing activations and measuring behavioral change, but the original computation is normally run before its output is overwritten.Thus, patching has primarily served interpretability rather than inference acceleration.Source: §2 · p. 3

3 LAYER SENSITIVITY ANALYSIS

The analysis measures how replacing layer activations affects model accuracy, then uses disjoint statistics and validation splits to locate depth regions sensitive to token-specific computation. Visual-stream patching is tolerated more at the decoder edges than in intermediate layers.

3.1 NOTATIONS

  • The analysis represents a frozen instruction-tuned VLM as a decoder with N layers and tracks outputs for four attention and three gated-MLP projections at each layer.Each module output activation is indexed by token position.Source: §3.1 · p. 3

3.2 STREAM-CONDITIONAL ACTIVATION STATISTICS

  • For each layer, module, and image or text stream, the method records example-level mean activations from a statistics split disjoint from evaluation data.Each example contributes equally, regardless of its token count.Source: §3.2 · pp. 3–4

3.3 CUMULATIVE LAYER INTERVENTION

  • Held-out validation sweeps patch cumulative layer prefixes from the input and suffixes from the output, testing image-only, text-only, and dual-stream interventions.Accuracy drops at each sweep point indicate how much token-specific information each modality contributes at that depth.Source: §3.3 · p. 4

3.4 OBSERVATIONS AND INSIGHTS

  • The sensitivity evaluation uses POPE, WhatsUp, and ScienceQA-IMG, with 100-example statistics and validation splits disjoint from each other and the final test set.The localization grid crosses model, dataset, sweep direction, and stream setup.Source: §3.4 · p. 4
  • Visual-token projection replacement is often tolerated in early and late decoder regions, whereas extending it into intermediate layers markedly degrades accuracy.Text-token computation patching remains broadly damaging, suggesting the redundancy is primarily associated with the visual stream.Source: §3.4 · p. 4

4 PATCH-TO-PRUNE FRAMEWORK

P2P profiles layer sensitivity, selects safe decoder regions under a validation accuracy tolerance, then bypasses visual-token projections there using cached prototypes. Its tolerance sweep exposes an accuracy–computation trade-off, and optional fine-tuning adapts the model to patched execution.

4.1 OVERVIEW

  • P2P has three stages: layer-sensitivity profiling, validation-guided safe-layer selection, and computation-bypassed inference.Source: §4.1 · p. 4

4.3.1 TOLERANCE SENSITIVITY.

  • At 1%, 7%, and 10% tolerance, average accuracy is 74.6%, 70.6%, and 67.1%, while computation falls from 2.1 to 1.3 TFLOPs versus a 3.1 TFLOP dense reference; the authors adopt 3%.Figure 3 presents the tolerance sensitivity analysis; the 3% setting is selected as a better trade-off between accuracy and computation.Source: Figure 3 · p. 5, §4.3.1 · p. 5

4.2 LAYER SENSITIVITY ANALYSIS

  • P2P first profiles how accuracy changes when visual-token computations are patched across decoder layers, identifying layers where replacement causes little accuracy loss..Source: §4.2 · p. 5

4.3 SAFE-LAYER SELECTION

  • P2P selects safe layers by requiring cumulative patching to preserve validation accuracy within the specified tolerance, then unions the forward-sweep prefix and backward-sweep suffix.The selected safe set is fixed after selection and applied unchanged at test time.Source: §4.3 · p. 5

4.4 COMPUTATION-BYPASSED INFERENCE

  • At selected safe layers, P2P skips visual-token attention and MLP projection computations and fills their outputs with cached prototypes.Textual tokens in those layers continue through the original attention computations.Source: §4.4 · p. 5

4.5 P2P-AWARE FINE-TUNING

  • Optional P2P-aware fine-tuning inserts LoRA adapters in safe-layer projections, trains on the patched computation graph, and computes cross-entropy only on ground-truth assistant-answer tokens.The adapters are optimized through remaining active computations, including unpatched textual tokens and modules outside patched layers.Source: §4.5 · p. 6

5 EVALUATION

P2P is evaluated on four VLMs against dense inference and token-pruning baselines, with tolerance settings controlling the accuracy–computation trade-off. Across the reported evaluation, it reduces FLOPs and latency while retaining most dense accuracy, and ablations examine proxy choice, proxy-set size, and fine-tuning.

5.1 EXPERIMENT SETUP

  • Evaluation covers seven benchmarks, four Qwen2.5-VL and LLaVA models, dense inference, and FastV and PyramidDrop baselines, with separate 100-example proxy and layer-selection sets.Main results use frozen models; the passage also describes an ablation fine-tuning protocol using LoRA.Source: §5.1 · p. 6

5.2 MAIN RESULT FOR P2P

  • At 3%, LLaVA-7B reduces average FLOPs from 30.8 to 12.6 T (59.1%) while retaining 93.8% of Vanilla average accuracy.LLaVA-13B reduces average FLOPs from 59.2 to 22.4 T (62.2%) while retaining 94.8% of Vanilla accuracy.Source: §5.2 · pp. 7–8, Table 1 · p. 7
  • At 3%, P2P uses fewer average FLOPs than FastV and PyramidDrop on both LLaVA sizes, but its average accuracy is also lower.On LLaVA-7B, P2P uses 12.6 T FLOPs versus 16.2 T for FastV and 14.5 T for PyramidDrop; its average accuracy is 0.601 versus 0.635 and 0.639, respectively.Source: Table 1 · p. 7, §5.2 · p. 8
  • At 3%, average latency falls by 15.4% on LLaVA-7B and 25.4% on LLaVA-13B, while FastV and PyramidDrop are slower than Vanilla on LLaVA-7B.The reported latency changes are 0.351 to 0.297 seconds per sample for LLaVA-7B and 0.619 to 0.462 seconds for LLaVA-13B.Source: Table 1 · p. 7, §5.2 · p. 8
  • Pressable computation varies by task: 3% P2P nearly preserves LLaVA-7B AI2D accuracy, while TextVQA and CV-Bench are more sensitive to aggressive pressing.On AI2D, accuracy is 0.641 compared with 0.642 for the dense model; some full-test accuracy drops can exceed the specified tolerance.Source: §5.2 · p. 8
  • 55% lower FLOPs and 22% lower latency accompany 94% of dense accuracy on average across four models and eight datasets.The passage describes these results as consistent savings without collapse.Source: §5.2 · p. 8

5.3 ABLATION STUDY

  • Mean activation replacement performs best, and 100 statistics examples provide a stable proxy efficiently.The proxy-set-size study varies the statistics set over 1, 25, 50, 100, 250, and 500 examples while keeping validation at 100 examples.Source: §5.3 · pp. 8–9, Figure 4 · p. 9, Figure 5 · p. 9
  • P2P-aware LoRA improves over frozen P2P across ScienceQA, TextVQA, and CV-Bench, recovering approximately 100% of the average accuracy gap at 3%.The patched inference graph is preserved; the reported recovery is approximately 75% at 1% and 59% at 5%.Source: §5.3 · pp. 9–10, Table 2 · p. 9

6 CONCLUSION

P2P turns activation patching into an inference-time computation bypass by replacing selected visual-token projections with fixed proxies, without removing tokens or disrupting sequence structure. Its layer-wise findings point to a task-sensitive intermediate region amid redundant early- and late-layer visual computation.

6 CONCLUSION

  • P2P bypasses selected visual-token projection computation with fixed proxies while preserving tokens and the original sequence structure.The conclusion frames this as a practical inference mechanism and identifies the intermediate region as task-sensitive, with substantial early- and late-layer computation redundant.Source: §6 · p. 10

A.1 ADVANTAGES

The paper presents P2P as a depth-wise complement to token pruning, with findings about modality-specific sensitivity and a small static activation cache. Its fine-tuning adapts only safe layers while retaining patches during training.

A.1 ADVANTAGES

  • Visual-token projection replacement is often tolerated across substantial early and late layers, while replacing text-token computations remains damaging.The authors describe the observation as a source of insight into visual information flow in VLMs.Source: §A.1 · p. 13
  • P2P prunes computation along decoder depth rather than tokens along the sequence, making it orthogonal and complementary to token-pruning methods.Source: §A.1 · p. 13
  • P2P-aware LoRA adapters are placed only in safe layers, with patches active during training so adaptation uses the inference computation graph.The adapters learn to compensate for information removed by the intervention.Source: §A.1 · p. 13
  • The cache stores one activation vector per projection module per decoder layer, reused across visual tokens and independent of sequence length and batch size.The paper characterizes this as a small static-memory cost traded for reduced visual-token computation.Source: §A.1 · p. 13

A.2 SWEEP RESULTS

Figures 6 and 7 report layer-wise sensitivity under cumulative forward and backward mean patching, covering different token-stream configurations on WhatsUp COCOQA and ScienceQA.

A.2 SWEEP RESULTS

  • Figure 6 shows WhatsUp COCOQA sensitivity under cumulative forward and backward mean patching of image, text, and joint image–text streams.Source: Figure 6 · p. 14
  • Across both figures, compare sensitivity across image-only, text-only, and joint image–text token streams in forward versus backward sweeps.Source: §A.2 · p. 13, Figure 6 · p. 14, Figure 7 · p. 14
  • Figure 7 shows ScienceQA sensitivity under cumulative forward and backward mean patching of image, text, and joint image–text streams.Source: Figure 7 · p. 14

A.3 MAIN RESULTS FOR QWEN MODELS

The section directs readers to Table 3 for the Qwen results.

A.3 MAIN RESULTS FOR QWEN MODELS

A.4 PATCH-TYPE ABLATION

The ablations compare proxy types and statistics-set sizes, while the benchmark section describes the evaluation tasks.

A.4 PATCH-TYPE ABLATION

  • Mean replacement best preserved dense behavior in early layers; zero or noise degraded more, while random visual-token replacement was sometimes competitive but less stable.The results support the modality-conditioned mean as a deterministic, distributionally grounded proxy that preserves typical module contributions while removing example-specific visual computation.Source: §A.4 · p. 14

A.5 PROXY-ESTIMATION SET SIZE ABLATION

  • Across Nstat values from 1 to 500, depth sensitivity remained similar; intermediate-layer interventions sharply reduced accuracy, and larger sets yielded no consistent improvement.The authors use 100 statistics examples for a stable layer- and module-specific proxy while keeping mechanistic discovery inexpensive.Source: §A.5 · pp. 14–15

A.6 BENCHMARKS

  • Seven test sets cover spatial relations, object hallucination, image-grounded science, compositional reasoning, scene text, vision-centric understanding, and scientific-diagram reasoning.The benchmarks are WhatsUp, POPE, ScienceQA-IMG, GQA, TextVQA, CV-Bench, and AI2D.Source: §A.6 · p. 15
Loading 2610.03389v1…