PaperlayerView exact revision on arXiv

Source-linked AI summary

PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding

Selim Kuzucu, Alessio Tonioni, Vasile Lup, Bernt Schiele, Federico Tombari, Muhammad Ferjad Naeem

arXiv:2605.30126v1cs.CVcs.AIcs.CLcs.LG

TL;DR

LVLM visual-token sequences make inference costly, and existing elastic methods trade spectral detail against spatial grounding under aggressive compression. PARCEL combines pooled low-frequency anchors with anchor-conditioned elastic queries, improving the performance-efficiency Pareto frontier across 27 benchmarks and budgets while preserving train-once, deploy-anywhere deployment.

  • Problem

    Existing elastic compression methods struggle under aggressive budgets because spatial pooling induces spectral aliasing while query resampling weakens explicit spatial grounding.

  • Method

    PARCEL uses spatial pool tokens as low-frequency layout anchors and conditions elastic query tokens on them through Pool-Conditioned Query Resampling.

  • Results

    Across 27 diverse vision-language benchmarks and visual-token budgets from 16 to 256 tokens, PARCEL improves the performance-efficiency Pareto frontier over existing matryoshka baselines.

  • Takeaways & Limitations

    PARCEL’s dynamic division of labor supports efficient visual understanding across varying inference budgets while retaining spatial layout and complementary visual detail.

  • Takeaways & Limitations

    The method may reduce computational barriers to biased predictions, hallucinated visual interpretations, privacy risks, and misuse in automated decision-making.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) map visual inputs into dense token sequences, imposing a quadratic computational bottleneck for inference. Elastic visual-token compression addresses this by training a single model that can run at multiple visual-token budgets. However, existing approaches struggle under aggressive compression. Spatial-only compression, as in nested pooling, behaves as an imperfect low-pass filter and induces spectral aliasing that obscures fine-grained detail. Query-only compression, as in nested query resampling, replaces explicit grid-aligned tokens with non-local summaries and substantially degrades spatial grounding. To resolve this representational conflict, we introduce PARCEL (Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understanding), a visual tokenization architecture that dynamically partitions the labor of feature extraction. PARCEL establishes spatial pool tokens as low-frequency layout anchors and conditions elastic query tokens on these anchors through Pool-Conditioned Query Resampling. This encourages query tokens to focus on complementary visual features rather than redundant spatial mapping. Extensive evaluations across 27 benchmarks show that PARCEL improves the performance-efficiency Pareto frontier, consistently outperforming existing matryoshka baselines across visual-token budgets while preserving the "train once, deploy anywhere" paradigm.

1. Introduction

LVLMs face costly long visual-token sequences, while elastic compression exposes a conflict between spatial detail and grounding. PARCEL addresses this with spatial anchors and conditioned elastic queries, improving efficiency–performance trade-offs across budgets.

  • LVLM inference becomes expensive because visual inputs may contain hundreds or thousands of tokens, increasing Transformer sequence length and attention cost.
  • Fixed-length compression forces a deployment choice between sacrificing fine-grained detail and retaining computationally expensive high resolution.
  • M3 uses nested spatial pooling, whereas MQT uses nested-dropout query resampling to support elastic visual-token budgets.
  • Under aggressive compression, M3 risks spectral aliasing and MQT reduces spatial grounding by replacing grid-aligned tokens with non-local summaries.
  • PARCEL combines low-frequency spatial pool tokens with query tokens conditioned on those anchors, assigning complementary feature-extraction roles.

2. Background and Related Work

Prior LVLM compression methods reduce visual-token cost through dropping, merging, reshaping, projection, or query resampling, but elastic approaches chiefly rely on pooling or queries. These paradigms remain vulnerable to complementary spatial and spectral bottlenecks under compression.

  • Visual-token compression research includes dropping, merging, spatial reshaping, resolution adaptation, projecting, and query-based resampling.
  • Many methods addressing positional bias and spatial distortion are training-free post-hoc optimizations rather than jointly trained elastic connectors.
  • Hybrid and distillation-based approaches often operate inside the language model and do not provide native connector-level inference elasticity.
  • Matryoshka Representation Learning embeds multiple granularities in one nested structure, enabling train-once, deploy-anywhere operation.
  • M3 obtains elasticity through successive spatial average pooling, while MQT uses nested-dropout query resampling.
  • M3 is prone to spectral aliasing and MQT is suboptimal for spatial understanding, motivating PARCEL’s explicit division of feature-extraction labor.

3. PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries

PARCEL separates coarse spatial layout from fine-grained visual detail by combining pooled anchors with pool-conditioned query resampling. Budget-aware routing preserves an explicit spatial representation while allocating remaining tokens to complementary queries.

  • Motivation: PARCEL targets the complementary degradation caused by spatial pooling and query-only resampling under aggressive compression.
  • Spectral Bottlenecks: Spatial-frequency analysis treats lower frequencies as global structure and higher frequencies as localized detail-sensitive features.
  • Spectral Bottlenecks: PARCEL’s pooled spatial tokens concentrate spectral power at low frequencies faster than M3, providing stronger low-frequency anchoring.
  • Spectral Bottlenecks: MQT lacks clear separation between layout and detail, and PARCEL achieves up to a +6.1 point retention advantage across RefCOCO at 64 tokens.
  • Pool-Conditioned Query Resampling: PCQR conditions learnable queries on pooled anchors before they cross-attend to full-resolution visual features, encouraging complementary detail extraction.
  • Pool-Conditioned Query Resampling: The final representation concatenates structural anchors, semantic-explorer query tokens, and text tokens before language decoding.
  • Budget-Aware Routing: Budget-aware routing dynamically selects anchor resolution and query count, using 16 anchors below 64 tokens and 64 anchors from 64 to 256 tokens.
  • Budget-Aware Routing: This routing preserves an explicit spatial anchor at every budget, while larger budgets add query tokens for source-grid detail.

4. Results and Discussions

PARCEL is evaluated against M3 and MQT across 27 vision-centric benchmarks, targeted compression-sensitive tasks, and architectural ablations. It achieves the strongest retention and accuracy–efficiency trade-offs across image, video, spatial-grounding, and resolution-sensitive settings.

  • Experimental Setup: PARCEL is evaluated across 27 benchmarks spanning video understanding, dense spatial grounding, resolution-sensitive reasoning, and general multimodal comprehension.The study measures aggregate retention, compression-sensitive task groups, and ablations of budget routing and Pool-Conditioned Query Resampling.
  • Aggregate Results: Across all visual-token budgets, PARCEL achieves the highest mean retention among compressed models and improves the accuracy–efficiency trade-off relative to Vanilla PG2.Visual-token count controls decoder prefill and KV-cache cost, linking retention gains to efficiency.
  • Video Understanding: At 16 tokens, PARCEL retains 92.8% of Vanilla PG2 performance on the top-3 compression-sensitive video tasks, versus 91.1% for MQT and 88.8% for M3.The comparison evaluates multi-frame inputs under severe compression.
  • Image Segmentation: At 256 tokens, PARCEL reaches 90.6% mean retention on RefCOCO, outperforming MQT by +4.2 points and M3 by +8.9 points.At 16 tokens, PARCEL retains 80.5% mean retention across the RefCOCO suite, compared with 79.0% for MQT.
  • Resolution-Sensitive Benchmarks: At 16 tokens, PARCEL achieves 77.1% mean retention on the top-3 resolution-sensitive tasks, exceeding M3 at 75.5% and MQT at 74.2%.The block covers DocVQA and ChartQA, which stress fine-grained visual evidence.
  • Ablations: Budget-aware routing and the division of labor are necessary: anchor-only baselines reach 83.1% and 92.8% retention at 16 and 64 tokens, while fixed 16-token anchors stagnate at 90.2%.Upgraded MQT and M3 still trail PARCEL at 256 tokens, supporting the role of dynamic routing rather than added capacity alone.

5. Conclusion

PARCEL addresses the computational bottleneck of LVLM inference by combining spatial anchors with pool-conditioned semantic queries. Across 27 benchmarks, this spectral partitioning establishes a stronger performance–efficiency Pareto frontier while preserving elastic deployment.

  • Conclusion: PARCEL combines spatial anchors with pool-conditioned semantic queries to partition visual feature extraction under elastic token budgets.The architecture separates low-frequency geometric layouts from high-frequency visual details.
  • Conclusion: Across 27 vision-centric benchmarks, PARCEL sustains dense recognition, temporal reasoning, and resolution-sensitive performance under 16-token constraints.The conclusion frames these results as establishing a new performance–efficiency Pareto frontier.
  • Conclusion: PARCEL preserves the “train once, deploy anywhere” paradigm while targeting efficient LVLM deployment.The stated scope is deployment efficiency across varying visual-token budgets.

A. Spectral Analysis Protocol

The spectral analysis protocol characterizes compressed visual representations on their native grids using normalized Fourier power and radial summaries. It compares low-frequency concentration and query-attended source-feature spectra without upsampling pooled grids.

  • Spectral Diagnostics: The protocol analyzes how visual-token compression mechanisms allocate spatial-frequency power, especially the effects of downsampling on representable frequencies.Components above a reduced Nyquist limit may fold or leak into lower frequencies without prior attenuation.
  • Feature Grids: Spectra are computed on native feature grids before language-model projection, and pooled grids are not upsampled before Fourier analysis.This preserves the native resolution associated with each compressed representation.
  • Fourier Analysis: The protocol removes spatially constant components, applies a normalized 2D discrete Fourier transform, and uses Parseval normalization for scale-consistent spectral power.The normalization prevents larger grids from producing larger Fourier magnitudes solely because they contain more samples.
  • Radial Binning: Radial frequency is measured as Euclidean distance from the Fourier-plane origin, with comparisons restricted to the largest centered circle fitting inside the grid.For example, the inscribed radial Nyquist limits are 8 for a 16 × 16 grid and 4 for an 8 × 8 grid.
  • Radial Binning: Radial mean power averages PSD values within unit-width frequency rings and measures per-coefficient spectral concentration without extra weighting for larger high-frequency rings.Dataset-level curves average per-sample radial profiles after this computation.
  • Diagnostics: Cumulative spectral concentration measures how quickly normalized power enters the low-frequency baseband, whereas normalized radial mean power shows relative per-mode concentration.Query-based compressors are analyzed through attention-weighted footprints on the original ViT feature grid rather than as native 2D query sequences.
  • Comparative Analysis: The two diagnostics compare pool tokens and query-attended ViT features to assess whether spatial tokens anchor low frequencies while queries retain higher-frequency support.The protocol also includes high-resolution pretraining evaluations as an additional stress test for settings where visual detail matters.

B.1. Detailed Results Across All PaliGemma-2 Benchmarks

Across default-resolution PaliGemma-2 evaluations, PARCEL achieves its strongest compression advantages on video, segmentation, and resolution-sensitive tasks, while remaining competitive on general vision-language benchmarks.

  • Video understanding: 98.0%, 97.9%, and 95.0% retention at 256, 64, and 16 tokens, respectively, gives PARCEL the highest video-understanding retention at every budget.These results indicate stronger preservation of temporal visual evidence than M3 and MQT under compression.
  • Image segmentation: 90.6% and 90.8% retention at 256 and 64 tokens surpass MQT's 86.4% and 84.9%, while PARCEL remains strongest at 16 tokens with 80.5%.The results support explicit 2D spatial anchors for preserving layout-sensitive information.
  • Resolution-sensitive benchmarks: 96.7% mean retention at 256 tokens exceeds MQT (96.3%) and M3 (94.8%), while PARCEL reaches 95.7% versus 94.1% and 92.3% at 64 tokens.At 16 tokens, M3 has the highest aggregate retention, while PARCEL remains competitive and outperforms MQT.
  • General vision-language benchmarks: 99.4% and 99.2% retention at 256 and 64 tokens is best among compressed models on general vision-language benchmarks, while M3 leads at 16 tokens.All compressed models retain a large fraction of Vanilla PG2 performance in this less compression-sensitive task group.
  • Evaluation caveat: VATEX is excluded from aggregate retention calculations because its validation split shows high variance and its official test set is unavailable.At this validation-only benchmark, method differences are comparable to seed-level variation.

B.2. Detailed Results for Benchmarks at High Resolution

High-resolution evaluation treats compression as a stress test and finds that PARCEL retains the strongest aggregate performance across 1024, 256, and 64 visual-token budgets.

  • Evaluation setting: High-resolution results use 448 × 448 PaliGemma-2 evaluations at 1024, 256, and 64 visual-token budgets as a compression stress test.The analysis is not intended to replace the three-seed default-resolution evaluation.
  • Evaluation caveat: High-resolution pretraining and evaluation results use a single seed and exclude video benchmarks because of their substantially higher cost.Accordingly, they serve as a stress test rather than a replacement for the default-resolution evaluation.
  • Aggregate results: 98.2% mean retention at both 1024 and 256 tokens exceeds M3's 96.0% and 95.4% and MQT's 95.4% and 95.8%, respectively.At 64 tokens, PARCEL remains strongest with 95.4% versus 93.5% for MQT and 93.4% for M3.
  • Image segmentation: PARCEL consistently improves over both baselines across RefCOCO, RefCOCO+, and RefCOCO-g image-segmentation splits.The results support explicit spatial anchors for preserving layout-sensitive evidence.
  • Resolution-sensitive and general benchmarks: PARCEL achieves strong retention on ChartQA, DocVQA, InfoVQA, and TextCaps, while providing the best aggregate trade-off on general multimodal benchmarks.The reported benefits continue under high-resolution visual encoding, although general-benchmark gaps are smaller.

C. FLOP and KV-Cache Calculations

The efficiency analysis estimates theoretical prefill FLOPs and decoder KV-cache memory across the visual encoder, connector, projection, language decoder, and output head.

  • Token accounting: PARCEL's compressed budget B is decomposed into N_p spatial anchor tokens and N_q query tokens before compressed visual tokens enter the language decoder.The analysis also specifies the full prefill sequence length after adding text-prefix tokens.
  • Connector costs: At matched visual-token budgets, M3, MQT, and PARCEL share dominant ViT and LLM costs, so their FLOP differences come from connector terms.PARCEL's connector uses Query ↔Pool self-attention followed by Query →ViT cross-attention when query tokens are allocated.
  • Connector routing: When N_q = 0, PARCEL reduces to a spatial-anchor-only representation and deactivates the query pathway.For N_q > 0, the query pathway contributes self-attention, cross-attention, and query-token MLP costs.
  • FLOP accounting: Total theoretical prefill compute is the sum of vision-encoder, connector, projection, language-decoder, and output-head costs.The reported TFLOP values are obtained by substituting constants and rounding to one decimal place.
  • KV-cache accounting: KV-cache memory is computed for Gemma-2's grouped-query decoder using four key-value heads and bfloat16 storage, with image and 16-frame video token totals defined separately.For images, N_tot = B + 129; for 16-frame videos, N_tot = 16B + 65.

D. Benchmark Details

The evaluation suite spans video understanding, dense spatial grounding, resolution-sensitive document and text tasks, captioning, and general multimodal reasoning across diverse benchmarks.

  • Video understanding: Video benchmarks test dense captioning, temporal understanding, action reasoning, open-ended video question answering, and multilingual video captioning.The suite includes ActivityNet, MSRVTT, MSVD, and VATEX tasks, with VATEX reported only in Section B because its official test set is unavailable and validation variance is high.
  • Dense spatial grounding and segmentation: RefCOCO, RefCOCO+, and RefCOCO-g evaluate referring-expression segmentation and increasingly demanding visual grounding.RefCOCO+ reduces reliance on absolute location words, while RefCOCO-g uses longer, more descriptive expressions.
  • Resolution-sensitive document, chart, OCR, and screen tasks: ChartQA, DocVQA, InfoVQA, ST-VQA, TextVQA, OCR-VQA, and TextCaps probe chart reasoning, document layout, infographic understanding, scene-text reading, OCR, and text-aware captioning.These tasks require preserving fine-grained evidence and text-sensitive visual information.
  • General visual question answering and reasoning: VQAv2, GQA, xGQA, OKVQA, AOKVQA, NLVR2, MARVL-5, VizWizVQA, and RSVQA cover general, compositional, multilingual, knowledge-based, multi-image, accessibility-focused, and remote-sensing reasoning.The benchmarks vary visual quality, language, required outside knowledge, and input structure.
  • Captioning and multilingual image understanding: COCO-CAP, NoCaps, COCO-35L, and XM3600 evaluate standard, open-vocabulary, multilingual, and cross-lingual image captioning.NoCaps specifically tests generalization to novel objects beyond standard COCO categories.

E. Implementation Details

Implementation follows the corresponding elastic-compression baselines while routing PARCEL budgets between spatial anchor grids and nested query tokens. Training uses PaliGemma-2 recipes, method-specific connector pretraining, and benchmark transfer tuning.

  • Budget sampling and routing: PARCEL samples total budgets from 16 through 256 tokens, using 16- or 64-token spatial anchors and queries to fill the remaining budget.For budgets below 64, the 16-token anchor is used; from 64 through 256, the 64-token anchor is used.
  • Connector architecture: PARCEL uses one Query ↔Pool self-attention block followed by one Query →ViT cross-attention block, with a shared ordered query bank supporting nested truncation.All connector attention operates at visual width D_v=1152 with 12 heads, and shared query prefixes span routing regimes.
  • Pretraining setup: Pretraining retains the PaliGemma-2 configuration, including its data mixture, task mixture, learning-rate multipliers, and Gemma 2 logit soft-capping.The mixture includes captioning, grounded captioning, OCR, VQA, detection, and instance segmentation.
  • Intermediary pretraining of compressed models: Compressed models receive initialized shared weights and randomly initialized method-specific connectors, followed by an additional 100M-sample intermediary pretraining stage with nested dropout for MQT and PARCEL.The learning-rate schedule is restarted because newly added connector components are learned from scratch.
  • Transfer tuning and evaluation: Transfer tuning keeps budget sampling active, applies video budgets per frame, and follows the PaliGemma/PaliGemma-2 evaluation setup.High-resolution experiments use 448 × 448 inputs and PARCEL anchor sizes of 16, 64, and 256 tokens.

F. Limitations and Social Impact

The paper identifies backbone, data, compute, budget-selection, and responsible-deployment boundaries while presenting token compression as a route toward more practical multimodal models.

  • Limitations: PARCEL inherits biases and demographic, cultural, geographic, and linguistic imbalances from the PaliGemma-2 backbone and its web-scale pretraining data.
  • Limitations: Training and evaluating large multimodal models still requires substantial compute, limiting accessibility despite reduced inference cost.The authors motivate more compute-efficient training, lightweight ablations, and lower-cost evaluation protocols.
  • Limitations: Budget lists are specified by practitioners rather than predicted from inputs, leaving input-adaptive budget allocation as future work.
  • Social impact: Efficiency improvements may ease deployment at scale while retaining risks associated with large vision-language models, including bias, hallucination, privacy, and misuse.The method introduces no new data sources or novel domain-specific capabilities beyond the underlying model.
  • Conclusion: The authors position efficient visual-token compression as a step toward more practical and sustainable multimodal models, with future work on adaptive budgets, broader backbones, and bias-aware evaluation.
Loading 2605.30126v1…