PaperlayerView exact revision on arXiv

Source-linked AI summary

PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference

Junjie Liu, Shengyuan Ye, Xu Chen

arXiv:2608.27206v1cs.CVcs.AI

TL;DR

High-resolution VLM inference is costly because visual encoding and LLM processing handle many visual tokens, while existing pruning leaves encoder cost and detail preservation insufficiently addressed. PACE combines pre-encoder adaptive pixel condensation with post-encoder dual-attention extraction to address both bottlenecks. On Qwen2.5-VL-7B, it retains 93.8% of uncompressed performance at a 90% token reduction and provides a 3.1× TTFT speedup.

  • Problem

    Existing visual token pruning mainly operates after vision encoding and struggles to preserve holistic contexts and fine-grained details under strict token budgets.

  • Method

    PACE is a training-free Condense-and-Extract framework that uses APC for pre-encoder adaptive pixel condensation and DDAE for post-encoder dual-attention token extraction.

  • Results

    93.8% of Qwen2.5-VL-7B’s uncompressed performance is retained with a 90% token reduction and a 3.1× TTFT speedup.

  • Takeaways & Limitations

    PACE provides a performance–efficiency trade-off for high-resolution VLM inference by addressing both vision-encoder and LLM bottlenecks.

  • Takeaways & Limitations

    APC can miss faint or tiny characters, small chart labels, thin lines, and low-contrast objects because its one-shot condensation is query-agnostic.

Abstract

from arXiv · show

Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.

1 Introduction

PACE addresses two bottlenecks in high-resolution VLM inference: vision encoding and LLM prefill, while targeting the loss of holistic context and fine-grained detail under aggressive token compression. Its Condense-and-Extract framework combines pre-encoder pixel condensation with post-encoder token extraction, retaining over 93% of performance while discarding 90% of visual tokens and achieving 3.1× TTFT acceleration on Qwen2.5-VL-7B.

  • Motivation: High-resolution inputs create substantial visual-token and inference costs, with Qwen2.5-VL producing over 42,000 patches for a 4K image and over 10,500 visual tokens after pooling.These sequences extend the LLM context and increase processing demands.
  • Limitations of Existing Methods: Existing pruning methods mainly reduce LLM-side computation because they operate after vision encoding, leaving the ViT encoding bottleneck unaddressed.At high resolutions, ViT encoding and LLM prefill both impose severe latency bottlenecks.
  • Limitations of Existing Methods: Aggressive token pruning can discard layout structure and visual cues, causing severe degradation on detail-sensitive tasks such as ChartQA and DocVQA.The cited cues include text strokes and alignment anchors.
  • PACE: PACE uses APC before the vision encoder to estimate information density and condense pixels, then uses DDAE to fuse ViT self-attention with LLM cross-modal attention.The two stages target encoder efficiency and preservation of task-relevant fine-grained details, respectively.
  • Results: PACE preserves over 93% of original performance while discarding 90% of visual tokens and delivers a 3.1× TTFT acceleration on Qwen2.5-VL-7B.This is reported as the framework’s performance–efficiency trade-off.

2 Related Work

Related work reduces visual-token costs through pruning and adaptive-resolution processing. These approaches generally target LLM-side prefill after visual encoding, while adaptive-resolution methods often add learned routing, compression, or reinforcement-learning components.

  • Visual Token Pruning: Visual token pruning selects or merges tokens before they enter the LLM using attention, vision–language relevance, similarity, diversity, or coverage signals.Examples include FastV, SparseVLM, LLaVA-PruMerge, VisionZip, DART, DivPrune, and MMTok.
  • Visual Token Pruning: Most existing pruning methods operate after visual encoding, reducing LLM-side prefill cost while leaving ViT encoding cost unchanged.This limitation is stated across the surveyed pruning approaches.
  • Adaptive-Resolution VLMs: Adaptive-resolution VLMs adjust image scale, compression rate, or token allocation according to visual complexity.Examples include ViCO, HyperVL, AdaptVision, and VisionThink.
  • Adaptive-Resolution VLMs: Existing adaptive-resolution methods often require additional modules, learned routing, or reinforcement-learning procedures.These requirements distinguish them from the training-free framework introduced by PACE.

3 Method

PACE uses a unified Condense-and-Extract pipeline to reduce computation before both vision encoding and LLM prefill while preserving global layouts and fine-grained visual evidence. APC adapts input resolution using visual-density signals, and DDAE combines LLM and ViT attention to retain task-relevant tokens.

  • Unified Condense-and-Extract: PACE addresses dual latency bottlenecks by condensing inputs before ViT encoding and extracting residual redundancy before LLM prefill.APC reduces pixel volume processed by the vision encoder, while the Extract stage removes redundant encoded tokens.
  • Condense: Adaptive Pixel Compressor: APC dynamically modulates global input resolution, allocating higher pixel budgets to information-dense images and lower budgets to redundant ones.This preserves continuous visual layouts while adapting computation to image complexity.
  • Condense: Adaptive Pixel Compressor: APC estimates global redundancy and local detail from a lightweight shallow feature preview before adaptively resizing the raw image.The preview uses the initial ViT block and is reported as more reliable than RGB, entropy, edge-density, and Laplacian statistics.
  • Condense: Adaptive Pixel Compressor: APC combines global and local retention scores into a target ratio, then resizes the input while enforcing any lower maximum scaling ratio imposed by the token budget.The unified ratio weights global and local scores through α and operates independently before the encoder.
  • Extract: Dynamic Dual-Attention Extractor: DDAE fuses LLM semantic attention with ViT self-attention to counter the narrow focus of either signal alone.LLM attention can prune unreferenced visual anchors, whereas ViT attention can retain query-irrelevant background clutter.
  • Extract: Dynamic Dual-Attention Extractor: DDAE uses attention-distribution sharpness as confidence, dynamically weights both maps, and preserves the top-K tokens ranked by the resulting saliency score.Semantic and visual maps are normalized before confidence-weighted fusion.

4 Experiments

PACE is evaluated across Qwen2.5-VL variants, nine benchmarks, multiple retention budgets, and latency settings. It consistently outperforms pruning baselines while retaining performance and reducing TTFT under aggressive compression.

  • Evaluation setup: PACE is evaluated on Qwen2.5-VL-3B and 7B, with additional cross-model testing on InternVL3.5-4B.The evaluation uses lmms-eval and includes fixed-resolution and dynamic-resolution configurations.
  • Evaluation setup: The benchmark suite covers nine tasks spanning perception, multimodal reasoning, visual question answering, document understanding, chart understanding, and OCR.The datasets include MME, POPE, MMBench, MMStar, RealWorldQA, TextVQA, DocVQA, ChartQA, and OCRBench.
  • Main results: PACE consistently surpasses baselines across configurations, especially on detail-sensitive DocVQA and OCRBench under 5% and 10% retention.Post-encoder pruning methods show marked degradation on these tasks because high-frequency visual anchors are lost.
  • Main results: At 10% retention, PACE maintains 93.8% of full-model performance and gains 12.7 and 13.9 percentage points over VisionZip and FastV.On DocVQA, PACE reaches 69.55 versus 59.56 for FastV and 49.55 for VisionZip.
  • Robustness: At 10% retention, PACE scores 84.99% on POPE, 59.08% on MMStar, and 70.90% on OCRBench, exceeding the strongest baseline by 1.44, 4.18, and 17.90 points.At 5% retention, the corresponding advantages are 2.00, 7.39, and 14.70 points.
  • Efficiency: TTFT falls from 365.89 ms to 116.79 ms at 10% retention, a 3.13× speedup, despite 34.62 ms of APC preview-and-resizing overhead.Vision encoding speeds up 3.01× and LLM prefill 6.64× in the isolated stage measurements.

5 Analysis and Discussion

The analysis tests APC’s compatibility, adaptive resolution, DDAE attention fusion, and extraction depth. Results show improved detail preservation and balanced performance, alongside an explicit quality–latency trade-off.

  • Latency accounting: Table 2 separates encoder and prefill latency from TTFT, with TTFT including APC preview and resizing overhead.Measurements use Qwen2.5-VL-7B at 10% retention in the fixed-resolution setting on one RTX 4090.
  • Adaptive resolution: Adaptive resolution outperforms the best static counterpart on RealWorldQA at both 10% and 5% budgets.Fixed Res.-50 gains 0.36 points on ChartQA at 10% but loses 2.74 points on RealWorldQA.
  • Attention fusion: Dynamic DDAE provides the most resilient performance profile by combining vision-side and LLM-side attention.LLM-only attention loses over 10 points on ChartQA, while vision-only attention compromises task alignment.
  • Extraction depth: Deeper DDAE extraction improves visual reasoning but reduces prefill speedup from 2.11× at layer 2 to 1.09× at layer 24.The paper therefore uses Lext = 2 as the efficiency-oriented default.

6 Conclusion

PACE accelerates high-resolution VLM inference through a unified Condense-and-Extract framework. It combines pre-encoder pixel condensation with dual-attention token extraction to preserve layouts and fine-grained details while reducing computation.

  • Conclusion: PACE retains 93.8% of Qwen2.5-VL-7B’s uncompressed performance at a 90% token reduction and delivers a 3.1× TTFT speedup.APC removes pixel-level redundancy before encoding, while DDAE fuses ViT visual priors with LLM semantic relevance.

7 Limitations

PACE’s limitations concern architecture-dependent encoder gains and preview overhead, as well as one-shot condensation that can lose faint or tiny visual evidence. These constraints bound portability and safe aggressiveness of compression.

  • Scope and portability: APC accelerates the encoder only when reducing pixel or tile budgets also reduces encoder-token counts.On fixed-grid VLMs, DDAE can reduce LLM prefill cost, but APC provides no encoder-side gain.
  • Scope and portability: Preview overhead varies with architecture and resolution, so Qwen2.5-VL speedups may not transfer unchanged to other backbones.This limits direct extrapolation of the reported latency results.
  • Information loss: APC’s query-agnostic, one-shot condensation can miss faint or tiny characters, small chart labels, thin lines, and low-contrast objects.The paper recommends higher retention, less aggressive budgets, or lower α for high-stakes applications.

A.2 Detailed Benchmark Descriptions

The section describes the evaluation benchmarks, implementation settings, and sensitivity analyses used to assess PACE across visual-processing demands.

  • Evaluation benchmarks: PACE is evaluated across nine benchmarks spanning general visual understanding, object hallucination, multimodal reasoning, OCR, charts, documents, and text embedded in images.The benchmark suite includes RealWorldQA, POPE, MME, MMBench, MMStar, OCRBench, TextVQA, ChartQA, and DocVQA.
  • Evaluation benchmarks: ChartQA measures numerical and structural reasoning over charts, while DocVQA measures document reading and layout understanding on scanned documents.
  • Implementation settings: The APC configuration uses K = 1, α = 0.6, γ = 1.5, and scales image width and height by √r to achieve the target pixel-retention ratio.Bicubic interpolation is used for adaptive resizing.
  • Implementation settings: DDAE obtains visual self-attention from the final ViT block and semantic relevance from the second LLM block, with τ = 0.5 and greedy decoding.Semantic attention weights are averaged uniformly across valid tokens.
  • Sensitivity analyses: α = 0.6 provides the most balanced MMStar behavior, whereas α = 1.0 reduces instance reasoning by 3.61 points and logical reasoning by 4.35 points versus the default.Larger α values favor global density, while smaller values emphasize local detail contrast.
  • Sensitivity analyses: α = 0.4 and α = 0.2 reduce the MMStar average by 3.32 and 2.90 points, respectively, by over-emphasizing isolated high-frequency patterns.
  • Sensitivity analyses: The 5%, 10%, and 20% top-detail fractions perform similarly, so the study fixes 10% as a midpoint across models, datasets, and token budgets.

B.2 Shallow Feature Preview Ablation

The ablations examine APC’s preview signal, model transfer, retention behavior, latency, and DDAE attention aggregation. Results favor semantic previews, adaptive pre-encoding compression, and complete-sequence semantic aggregation.

  • Shallow Feature Preview Ablation: Pixel statistics cannot reliably distinguish semantic information density from texture, noise, decorative patterns, or irrelevant edges.Edge density is comparatively effective on ChartQA, while Laplacian responses are more sensitive to fine-scale noise.
  • Cross-model evaluation: PACE surpasses the best InternVL3.5-4B baseline average by 5.1, 4.7, and 4.0 points at 25%, 20%, and 10% retention, respectively.It is not uniformly best on every task but preserves substantially more OCRBench and DocVQA performance.
  • Retention and resolution: PACE degrades more gradually than post-encoder baselines under fixed-resolution 10% and 5% retention, particularly on detail-sensitive benchmarks.The result is consistent with adapting input resolution before full visual encoding.
  • Latency profiling: 3.13× is the average TTFT speedup at 10% retention, compared with 2.64× at 20%, 1.67× at 50%, and 1.10× at 80%.Preview overhead is nearly fixed, so it is increasingly amortized at stricter budgets.
  • Latency profiling: 3.34× is the average TTFT speedup at 10% retention, with encoder latency falling from 152.41 ms to 46.10 ms and prefill latency from 222.16 ms to 31.63 ms.Including APC and resizing, average TTFT decreases from 374.58 ms to 112.08 ms.
  • DDAE attention ablation: Vision + Text aggregation performs best on both MME and DocVQA, while restricting aggregation to linguistic or terminal tokens reduces performance.DocVQA drops by nearly 10 points when aggregation is restricted.
  • DDAE attention ablation: Moving DDAE extraction from layer 2 to layer 24 reduces prefill speedup from 2.11× to 1.09× at 2048 patch units and from 2.32× to 1.10× at 4096.Decode and time per output token remain approximately unchanged at ≈1.00×.

C Detailed Computational Complexity Analysis

The complexity analysis separates PACE’s effects on vision encoding and LLM prefill. APC reduces the encoded patch count, while earlier DDAE extraction reduces the number of later full-sequence LLM layers.

  • Baseline complexity: An H × W image produces N = HW/P^2 encoder patches, and each of the L_v ViT blocks costs O(ND_v^2 + N^2D_v).
  • Baseline complexity: With T prompt tokens and N_vis visual tokens, Vanilla prefill length is S_0 = T + N_vis, with per-layer cost O(S_0D_l^2).Autoregressive decoding determines TPOT and is unchanged by PACE.
  • PACE complexity: APC retains an image-area ratio p_1, causing the full encoder to process approximately p_1N patches and the merger to output approximately p_1N_vis tokens.The preview itself uses K = 1 block at the original resolution.
  • PACE complexity: DDAE retains a further ratio p_2, giving a final visual-token ratio B = p_1p_2 relative to Vanilla.The post-extraction sequence has length S_post = T + N_keep.
  • PACE complexity: APC reduces full-encoder computation, whereas earlier DDAE extraction leaves fewer LLM layers operating on the longer pre-extraction sequence.

D Qualitative Results

Qualitative analyses show that information density varies substantially across benchmarks, motivating image-adaptive pixel allocation rather than a single fixed-resolution policy. The paper identifies further directions for extending this approach.

  • Information density: Information density distributions are relatively concentrated for RealWorldQA, POPE, MME, and MMBench, but broader or heavier high-density regions appear in ChartQA, OCRBench, TextVQA, and DocVQA.The latter datasets contain fine-grained textual, structural, or layout information.
  • Information density: Low-density samples can be aggressively condensed, whereas high-density samples require higher pixel budgets to preserve local details.This variation makes a single fixed-resolution policy unsuitable across diverse visual characteristics.
  • Adaptive allocation: APC estimates information density before full visual encoding and adjusts input resolution according to each image’s intrinsic complexity.The stated goal is to preserve holistic structures while retaining fine-grained evidence under strict token budgets.
  • Future directions: Future work proposes autoregressive high-resolution crop recovery, query-aware pixel allocation, localized patching, and integration with model quantization.These directions are presented as extensions for more aggressive or broader end-to-end acceleration.
Loading 2608.27206v1…