PaperlayerView exact revision on arXiv

Source-linked AI summary

PixelPrune: Pixel-Level Adaptive Visual Token Reduction via Predictive Coding

Nan Wang, Zhiwei Jin, Chen Chen, Haonan Lu

arXiv:2604.00886v1cs.CVcs.AIcs.CL

TL;DR

High-resolution document and GUI VLM workloads create heavy computational costs, despite substantial pixel-level patch redundancy. PixelPrune uses parameter-free predictive coding before the ViT to remove redundant patches and accelerate the full pipeline. Across document and GUI benchmarks, it maintains competitive accuracy while achieving up to 4.2× inference speedup and 1.9× training acceleration.

  • Problem

    Document and GUI VLM tasks require high-resolution inputs that generate many visual tokens, while 22–71% of image patches are pixel-unique within each image.

  • Method

    PixelPrune applies parameter-free patch-level predictive coding before the ViT, supporting pixel-lossless τ=0 and controlled lossy τ>0 compression.

  • Results

    Across document and GUI benchmarks, PixelPrune maintains competitive accuracy while delivering up to 4.2× inference speedup and 1.9× training acceleration.

  • Takeaways & Limitations

    Pre-ViT pixel-level pruning reduces computation across the ViT encoder, patch merger, and LLM decoder while preserving fine-grained task performance.

  • Takeaways & Limitations

    Pixel-level guarantees do not preserve the ViT’s full-input internal representations because the ViT receives a shorter sequence.

Abstract

from arXiv · show

Document understanding and GUI interaction are among the highest-value applications of Vision-Language Models (VLMs), yet they impose exceptionally heavy computational burden: fine-grained text and small UI elements demand high-resolution inputs that produce tens of thousands of visual tokens. We observe that this cost is largely wasteful -- across document and GUI benchmarks, only 22--71\% of image patches are pixel-unique, the rest being exact duplicates of another patch in the same image. We propose \textbf{PixelPrune}, which exploits this pixel-level redundancy through predictive-coding-based compression, pruning redundant patches \emph{before} the Vision Transformer (ViT) encoder. Because it operates in pixel space prior to any neural computation, PixelPrune accelerates both the ViT encoder and the downstream LLM, covering the full inference pipeline. The method is training-free, requires no learnable parameters, and supports pixel-lossless compression ($τ{=}0$) as well as controlled lossy compression ($τ{>}0$). Experiments across three model scales and document and GUI benchmarks show that PixelPrune maintains competitive task accuracy while delivering up to 4.2$\times$ inference speedup and 1.9$\times$ training acceleration. Code is available at https://github.com/OPPO-Mente-Lab/PixelPrune.

1 Introduction

Document and GUI VLM workloads require high-resolution inputs, creating substantial visual-token and computation costs. PixelPrune exploits pixel-level redundancy before vision encoding to reduce tokens across the full pipeline while preserving competitive accuracy and accelerating inference and training.

  • Motivation: High-resolution document and GUI inputs produce tens of thousands of visual patches, making VLM computation especially costly.The burden arises from fine-grained text rendering, small UI elements, and native-resolution vision encoders.
  • Motivation: 22–71% of patches are pixel-unique within document and GUI images, exposing substantial redundancy for token reduction.The remaining patches are exact pixel duplicates within each image.
  • Motivation: Existing acceleration methods often reduce tokens after or around the LLM while leaving the ViT encoder’s cost untouched.At 4096^2 resolution, the vision encoder accounts for up to 86% of Qwen3-VL-2B prefill latency.
  • PixelPrune: PixelPrune applies patch-level predictive coding before vision encoding, omitting patches whose predictions match within threshold τ.It is parameter-free, training-free, and supports pixel-lossless τ=0 or controlled lossy τ>0 compression.
  • Results: 4.2× inference speedup and 1.9× training acceleration are achieved while maintaining competitive accuracy across document and GUI benchmarks.Because pruning occurs before the ViT, the method accelerates the ViT encoder, patch merger, and LLM decoder.

2 Related Work

Prior vision and VLM token-reduction methods mainly prune or merge learned representations at intermediate or downstream stages. PixelPrune instead compresses redundant image patches directly in the 2D pixel domain before ViT feature extraction, targeting document and GUI redundancy.

  • Token Reduction in Vision Transformers: ViT token-pruning methods remove or merge tokens using learned importance scores, attention, or feature averaging after computation begins.DynamicViT, EViT, and ToMe operate on intermediate representations rather than raw pixels.
  • Token Reduction in Vision Transformers: RLT uses run-length encoding for 1D temporal video sequences, whereas PixelPrune applies predictive coding to 2D spatial patch redundancy.PixelPrune enables exact patch compression before ViT feature extraction.
  • Efficient Vision-Language Models: VLM efficiency methods compress tokens at projector, post-ViT, or inside-LLM stages, leaving pre-ViT computation outside their reduction point.These categories include Q-Former and TokenPacker, LLaVA-PruMerge and DivPrune, and FastV, ZipVL, and PyramidDrop.
  • Document and GUI Understanding: Document and GUI models commonly address high-resolution inputs with cropping, tiling, or dual encoders, while few methods exploit their inherent pixel-level redundancy.These domains contain substantial visual redundancy alongside fine-grained text and interface elements.

3 Method

PixelPrune compresses redundant image patches before vision encoding by predicting each patch from causal spatial neighbors and retaining only mismatches. Its coordinate-preserving, predictive-coding design supports exact reconstruction, controlled approximation, and acceleration across the ViT-to-LLM pipeline.

  • Patch compression: PixelPrune identifies a compact subset of patches to retain while preserving each patch’s original 2D grid coordinate.The retained patch–position pairs continue through the ViT, Patch Merger, and LLM, where positional encodings preserve spatial layout.
  • Patch compression: Predictive coding adapts lossless image-compression principles from pixels to patch blocks, omitting a block when its residual from the prediction is zero.For Qwen3-VL, 32×32 blocks correspond to one independently removable LLM token after patch merging.
  • Prediction strategies: Pred-2D predicts each raster-scanned patch from its left, upper, and upper-left neighbors, selecting a likely match through neighbor-agreement rules.It uses A=left, B=upper, and C=upper-left; agreement between upper neighbors directs prediction from the left, while agreement between left neighbors directs prediction from above.
  • Prediction strategies: Raster and Serpentine use single-neighbor predictors, whereas Pred-2D exploits both horizontal and vertical redundancy and is adopted as the default.All three strategies run in O(N) time and guarantee exact recoverability; Pred-2D consistently achieves the lowest retain ratio at comparable accuracy.
  • Matching and reconstruction: At τ=0, PixelPrune omits only pixel-identical predictions and already achieves 30–80% token reduction on document and GUI images.At τ>0, the acceptance criterion becomes bounded distance, enabling omission of near-exact patches while retaining original values for non-omitted patches.
  • Matching and reconstruction: For τ=0, a deterministic decoder exactly reconstructs the original patch sequence from retained patches and positions using the same prediction rule and scan order.For τ>0, every omitted patch has reconstruction error at most τ, with no error accumulation across patches.

4 Experiments

Experiments evaluate PixelPrune across document and GUI benchmarks, model scales, training regimes, compression strategies, and ViT pruning depths. The results show competitive accuracy alongside substantial inference and training efficiency gains, with task- and threshold-dependent limitations.

  • Experimental Setup: Experiments cover document and GUI benchmarks using Qwen3-VL models at 2B, 4B, and 8B scales, with GUI studies mainly using 2B.Document experiments use all three model scales, while GUI experiments and ablations use 2B unless otherwise noted.
  • Experimental Setup: PixelPrune is compared with Full, Random, ConnComp, and Resize under matched per-image token budgets.Full retains all visual tokens; the other baselines match PixelPrune’s token budget.
  • Document Understanding: 73.7 vs. 72.8 Avg at 4B shows PixelPrune matching or exceeding Resize on document understanding while preserving fine-grained details.At 8B, PixelPrune reaches 75.3 Avg versus 75.5 for Full, a 0.2-point gap, while retention varies across document benchmarks.
  • GUI Understanding: 47.6 vs. 50.2 on ScreenSpot Pro and 90.6 vs. 90.6 on ScreenSpot V2 show PixelPrune (+KD) nearly matching or matching Full on GUI benchmarks.Retention is especially low for Web, Mobile, Desktop, and Scientific interfaces, but CAD falls from 40.2 to 30.3 during KD recovery.
  • Training from Scratch: 85.1 vs. 81.8 SSv2 Avg shows PixelPrune→Full slightly outperforming Full→Full, supporting cross-mode deployment from one checkpoint.PixelPrune→PixelPrune also achieves competitive accuracy, indicating that training with token reduction does not significantly degrade performance.
  • Efficiency: 3.0–4.2× TTFT speedup, 3.5–6.6× FLOPs reduction, and 45–63% KV cache savings are achieved at inference, while 49.1 h to 25.3 h yields 1.9× training acceleration.From-scratch training also reports 2.0× forward and backward speedups and a 33.6% peak GPU memory reduction.
  • Ablations: Pred-2D lowers retention beyond Raster and Serpentine while maintaining comparable accuracy, and pruning before the ViT saves 100% of FLOPs for removed tokens.At τ=0, accuracy is competitive on most benchmarks; aggressive τ>0 pruning can reduce retention substantially but makes ChartQA particularly sensitive.

5 Conclusion

PixelPrune removes redundant visual tokens before the ViT using 2D predictive coding, preserving document accuracy while accelerating document and GUI processing.

  • PixelPrune removes redundant visual tokens before the ViT encoder via 2D predictive coding.
  • 23–50% token reduction matches full-token accuracy across document benchmarks and three model scales without training.
  • 4.2× inference speedup, 1.9× training acceleration, and 33.6% memory reduction are achieved across document and GUI benchmarks.
  • GUI accuracy gaps are recovered through light post-training and training-from-scratch settings while retaining significant training acceleration.

A Pixel-Level Redundancy and Resolution Statistics

The appendix characterizes benchmark redundancy using exact 32×32 patch matching and reports resolution after Qwen3-VL preprocessing, showing greater duplication in GUI images than documents.

  • Table 9 counts pixel-unique and duplicate non-overlapping 32×32 blocks within each image.
  • PixelPrune’s actual retain ratio is higher than global deduplication because causal prediction exploits redundancy only along scan order.
  • Mean Resolution reports benchmark width×height after Qwen3-VL preprocessing, with higher resolution creating more patch opportunities but content density affecting redundancy.
  • 51.4% mean duplication in GUI images exceeds 39.7% in documents across the evaluated benchmarks.
  • Duplicate Ratio is the dataset-level fraction of non-unique patches, equal to 1 − the unique-patch fraction.
  • Table 10 presents training-free Qwen3.5 document results with dataset-level PixelPrune retain ratios in parentheses.

B Training Details

Training configurations vary GPU count, batch size, and steps across post-training, training-from-scratch, and ablation experiments.

  • Post-training knowledge distillation uses 32 GPUs with 36 samples per GPU for 2,000 steps.
  • Training from scratch uses 18 samples per GPU for 4,000 steps to avoid out-of-memory errors and match the total sample budget.
  • Prediction-strategy and ViT-depth ablations use 8 GPUs with 36 samples per GPU for 1,000 steps.

C Cross-Architecture Validation: Qwen3.5

PixelPrune is evaluated on Qwen3.5, which shares a NaViT-style ViT with Qwen3-VL, and maintains small accuracy gaps across model scales.

  • Qwen3.5 shares Qwen3-VL’s NaViT-style ViT and separate spatial positional encodings, enabling direct PixelPrune compatibility.
  • Qwen3.5’s hybrid LLM interleaves full-attention and linear-attention layers, with linear layers relying on naturally decaying temporal information.
  • Accuracy gaps remain ≤0.8% Avg across all Qwen3.5 scales, confirming generalization across NaViT-based architectures.

D Training from Scratch: Detailed Convergence

Training with compressed visual tokens converges comparably to full-token training, reaching similar accuracy by roughly step 2,400 and plateauing around steps 2,400–3,200.

  • Both Full and PixelPrune configurations use matching visual-token settings during training and inference.Full uses all tokens, whereas PixelPrune uses compressed tokens in both phases.
  • By step 2,400, Full and PixelPrune converge to similar accuracy and plateau around steps 2,400–3,200.
  • 58.1 vs. 56.6 mean accuracy at step 4,000 favors PixelPrune slightly over Full.The passage notes that reduced token count may provide implicit regularization.

E Detailed FLOPs Analysis

The FLOPs analysis decomposes Qwen3-VL computation across the ViT, Patch Merger, and LLM, then shows how retaining fewer patches reduces costs throughout the pipeline.

  • FLOPs decomposition: Qwen3-VL FLOPs comprise ViT encoding, Patch Merger and DeepStack, and LLM processing for N image patches and T text tokens.Here N = (H/p) × (W/p), where p is the ViT patch size.
  • ViT Encoder: The ViT’s attention maps contribute 4N^2D_v FLOPs per layer, while its linear projections and FFN contribute 8D_v^2 + 4D_vD_f,v FLOPs per token.Attention is computed per image, so N denotes the patch count of a single image.
  • Patch Merger & DeepStack: After M×M spatial merging, Patch Merger input dimension is D_in = M^2D_v, and the main and DeepStack mergers are counted through L_m.
  • LLM: The LLM’s per-token linear cost combines GQA projections with a gated MLP, while attention maps add 4N_l^2D_l FLOPs per decoder layer.The cited architecture uses n_q query heads, n_kv key-value heads, and d_h = D_l/n_q.
  • Architecture parameters: Qwen3-VL-2B uses a 24-layer ViT, 2B-scale LLM, and M=2 spatial merging in the reported architecture parameters.
  • FLOPs savings: Replacing N with N_s reduces ViT attention quadratically, Patch Merger costs linearly, and LLM sequence length from N_l to N′_l.When N_s ≪ N, the passage reports substantial end-to-end reduction across all three stages.

F General-Domain Evaluation

PixelPrune generalizes beyond document and GUI images: exact matching preserves accuracy with little compression, while near-exact matching provides moderate compression with modest degradation.

  • Exact matching (τ=0) retains 80–100% of tokens with ≤1.1% accuracy change across general-domain benchmarks.
  • At τ=0.05, retention drops to 72–89% while accuracy remains within 0.6% on most benchmarks.
  • At τ=0.10, retention reaches 68–83% and accuracy degradation remains ≤2%.
  • Content-adaptive compression handles general-domain images by combining negligible-loss exact matching with moderate near-exact compression.
Loading 2604.00886v1…