PaperlayerView exact revision on arXiv

Source-linked AI summary

Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning

Yihong Huang, Fei Ma, Yihua Shao, Jingcai Guo, Zitong Yu, Laizhong Cui, Qi Tian

arXiv:2602.02951v1cs.CVcs.AIcs.CL

TL;DR

Existing token pruning preserves VQA performance but substantially degrades visual grounding because pruning can disrupt the global spatial reference frame. Nüwa combines spatial-aware aggregation that preserves global anchors with text-guided LLM refinement, achieving reported gains across VQA and visual grounding while reducing computation.

  • Problem

    Existing token pruning methods can preserve VQA performance yet substantially degrade visual grounding by disrupting global spatial reference frames.

  • Method

    Nüwa performs spatial-aware pruning through separation, alignment, and aggregation before applying text-guided refinement in the LLM.

  • Results

    95% performance retention on VQA and 47.2% performance retention on visual grounding are reported alongside 89% TFLOPs and 62% prefill reductions.

  • Takeaways & Limitations

    Nüwa preserves global spatial anchors while reducing visual-token computation across 13 datasets and multiple VLMs.

Abstract

from arXiv · show

Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM). However, existing pruning methods demonstrate excellent performance preservation in visual question answering (VQA) and suffer substantial degradation on visual grounding (VG) tasks. Our analysis of the VLM's processing pipeline reveals that strategies utilizing global semantic similarity and attention scores lose the global spatial reference frame, which is derived from the interactions of tokens' positional information. Motivated by these findings, we propose $\text{Nüwa}$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity. In the first stage, after the vision encoder, we apply three operations, namely separation, alignment, and aggregation, which are inspired by swarm intelligence algorithms to retain information-rich global spatial anchors. In the second stage, within the LLM, we perform text-guided pruning to retain task-relevant visual tokens. Extensive experiments demonstrate that $\text{Nüwa}$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%).

1 INTRODUCTION

Nüwa addresses task-dependent failures in visual token pruning by preserving global spatial anchors before applying text-guided refinement. It reports improved VQA and visual grounding performance alongside substantial computational reductions.

  • Motivation: Existing pruning methods reduce inference overhead but can degrade visual grounding substantially more than visual question answering.The paper frames this as a task-dependent limitation of current pruning strategies.
  • Nüwa framework: Nüwa uses two stages: spatial-aware pruning with separation, alignment, and aggregation, followed by text-guided refinement in the LLM.The first stage preserves global anchors while reducing redundancy; the second uses textual information for multimodal alignment-based pruning.
  • Task-specific analysis: Grounding tasks depend on global spatial reference frames that pruning can disrupt by overlooking spatial structure.Position reconstruction experiments support the link between spatial perception and global reference-frame integrity.
  • Performance validation: 95% performance retention on VQA and 47.2% performance retention on visual grounding establish Nüwa’s reported state-of-the-art results.These results span 13 datasets and multiple VLMs.
  • Performance validation: 89% lower TFLOPs, 62% lower prefill time, and 88.9% token reduction accompany the reported task performance.The reductions are reported as part of Nüwa’s validation across multiple VLMs.

2 DISSECTING THE VISUAL PROCESSING PIPELINE: FROM SEMANTIC FLOW TO SPATIAL INTEGRITY

The analysis finds that VLM visual processing progresses from global semantic integration to fine-grained, task-dependent processing, and that pruning harms grounding by disrupting the global spatial reference frame. Position reconstruction experiments support preserving continuous positional structure as a remedy.

  • 2.1 Evaluating Competitive Advantages: Cross-task evaluation finds that advanced pruning offers limited VQA benefits, while all methods degrade systematically on grounding tasks.Average pooling achieves the best performance among the compared methods.
  • 2.2 Unveiling Task-Dependent Visual Processing Pipeline: Attention and gradient-weighted analyses reveal task-independent early multimodal interactions but task-dependent mid-stage processing, with grounding relying more heavily on vision tokens.The visual processing pipeline therefore changes its integration pattern according to task requirements.
  • 2.2 Unveiling Task-Dependent Visual Processing Pipeline: VAE decreases through the ViT middle stage while OCC peaks in the middle stage of both ViT and LLM, indicating a shift toward fine-grained object-level representations.VAE captures diffuse versus concentrated visual attention, whereas OCC measures localization around relevant objects.
  • 2.3 Spatial Integrity: Grounding depends on a global spatial reference frame constructed from token positions, which pruning can disrupt during spatial reasoning.The analysis links grounding degradation to the loss of spatial continuity rather than only to reduced token count.
  • 2.3.1 A Taxonomy of Position Embedding Strategies: PERC compresses positional embeddings into a tiny range, PESP preserves sparse original positions, and RPME linearly extends relative positions across the original range.RPME is designed to preserve spatial distances while restoring a complete spatial frame.
  • 2.3.1 A Taxonomy of Position Embedding Strategies: 5.6% and 13.4% gains for VisionZip and 1.8% and 3.2% gains for FastV show that RPME improves visual grounding across two settings.The improvements are reported after replacing the original positional-embedding strategies with RPME.

3 METHODOLOGY

Nüwa compresses visual tokens in two stages while preserving spatial integrity: region-based vision processing retains global anchors, then text-guided LLM pruning selects task-relevant tokens.

  • Nüwa applies spatial uniformity, vision-centric aggregation, and text-modulated filtering to preserve spatial integrity during token compression.
  • Stage 1: Stage 1 partitions the token grid into non-overlapping regions, creating a complete global coordinate system before selection and aggregation.
  • Alignment via Salience Identification: Representative tokens are selected using global attention and key-vector L2-norm information capacity, with the highest-salience tokens forming the benchmark set.
  • Role Assignment: Benchmark tokens are divided into Pillars and Collectors by the top-quartile key-vector norm; Pillars remain unmodified while Collectors aggregate information.
  • Aggregation via Spatial Proximity: Weighted aggregation combines positive semantic similarity with spatial proximity, limiting long-range merging while enabling extended local-neighborhood interaction.
  • Stage 2: Stage 2 averages text-token embeddings into a query, scores visual-token relevance in the shared space, and retains the top-Kfinal tokens for later LLM layers.

4 EXPERIMENT

Experiments across multiple VLMs and benchmarks show that Nüwa preserves or improves VQA performance, substantially improves visual grounding, and adds little computational overhead.

  • Experimental Setup: Experiments evaluate LLaVA-1.5 and LLaVA-NeXT across 10 VQA and 3 visual-grounding benchmarks on NVIDIA A100-40G GPUs.
  • VQA Performance: Nüwa achieves optimal performance across nearly all reported VQA benchmarks and continues to show gains on LLaVA-NeXT-7B.
  • Visual Grounding Performance: Approximately 35% performance improvement over previous methods is achieved on RefCOCO benchmarks under the 64-average-token configuration.
  • Efficiency Analysis: TFLOPs increase by only 0.01 and prefill latency by 1 ms compared with the previous SOTA method.
  • Ablation Study: The spatial-proximity threshold peaks at 26% of maximum distance; region partitioning is essential for grounding, while key-norm selection improves performance across tasks.

5 CONCLUSION

The paper concludes that pruning can preserve efficiency and task performance when it maintains global spatial reference frames and combines spatially aware aggregation with text-guided refinement.

  • Grounding relies on global spatial reference frames that existing pruning can disrupt, motivating Nüwa’s spatial-integrity design.
  • Across 13 datasets and multiple VLMs, Nüwa reports 95% VQA performance retention and 47.2% visual-grounding retention.
  • The framework reports 89% TFLOPs reduction, 62% prefill reduction, and 88.9% token pruning.

REPRODUCIBILITY STATEMENT

The reproducibility materials describe comparative baselines, analytical and position-reconstruction experiments, implementations, environments, models, and datasets used in the study.

  • Comparative baselines are configured for the Section 2.1 experiment, while analytical experiments require no additional configurations.
  • The study includes position-reestimation experiments, attention-blocking experiments, and reproducible explanations for key experiments.
  • The related-work coverage describes vision-encoder pruning, LLM pruning, multi-stage optimization, and training-based methods.
  • The paper states that model weights come from the Hugging Face community and datasets originate from the lmm-lab datasets.

B.2 MAIN EXPERIMENT

The experiments evaluate Nüwa across VLM configurations and compare its two-stage pruning with baseline token-reduction methods. Results cover visual grounding, VQA, positional reconstruction, and implementation settings.

  • Experimental setup: Nüwa uses a two-stage pruning process configured for different VLMs, with stage-1 processing after the vision encoder and later pruning in the LLM.The supplied implementation passages identify the framework as a two-stage process and provide model-specific configuration context.
  • Stage-1 pruning: Nüwa’s stage-1 algorithm selects and aggregates visual features using vision-tower hidden states, attention-derived metric maps, grid separation, and configured token targets.The algorithm takes images as input, obtains hidden states and attentions, reshapes the metric map to a 2D grid, and returns selected aggregated tokens and their indices.
  • Baseline comparison: Adaptive token pooling reshapes visual tokens into a 2D feature map, pools to a target grid while preserving the original aspect ratio, and flattens the result.The target token count is computed from the retention ratio before constructing the output grid.
  • Position reconstruction: Position reconstruction experiments show that regenerating positional embeddings for a pruned sequence cannot naturally produce bounding boxes in the original image coordinate system.The passage describes this limitation for PERC-style methods under ideal pruning, where the retained target is treated as a cropped image.

B.5 ATTENTION BLOCK EXPERIMENT

The attention-blocking experiment partitions the LLAVA-1.5 7B decoder into four phases and tests how selectively blocking attention affects multimodal processing.

  • Experimental setup: The experiment evaluates attention blocking on three datasets using system, visual, text, and last-token categories across four equal decoder-layer phases.The setup is reported in Table 15 and uses LLAVA-1.5 7B.

B.5.1 BLOCKING ATTENTION FROM VISION TO VISION

Attention blocking reveals a strong task difference: VQA tolerates some disrupted visual interactions, whereas grounding depends heavily on spatial and contextual relationships across visual and multimodal stages.

  • Vision-to-vision blocking: Blocking vision-to-vision attention reduces GQA from 61.9 to 55.2 in layers 8-15 but does not cause catastrophic VQA degradation.Intermediate layers are described as important for integrating low-level features into object-level representations.
  • Vision-to-vision blocking: Blocking vision-to-vision attention drops RefCOCO performance from 58.30 to 2.64 in layers 8-15, showing much greater sensitivity than VQA.The supplied analysis attributes grounding dependence to spatial and contextual relationships between visual features.
  • Last-token-to-vision blocking: Blocking last-token-to-vision cross-attention has minimal VQA impact, with GQA changing from 61.9 to 62.55 and MMBench from 64.7 to 64.26.The GQA change is reported as within error margins, with a marginal improvement.
  • Last-token-to-vision blocking: The same last-token-to-vision blocking causes RefCOCO performance to fall to 2.01 in layers 8-15, while blocking later stages can improve accuracy.The results are described as a catastrophic grounding failure under the mid-layer block.
  • Task-specific dependence: The analysis concludes that VQA mainly uses early-to-mid abstract semantic extraction, whereas grounding requires sustained later visual extraction and precise mid-stage positional information.The reported grounding dependencies combine abstract semantic information with positional information from the model’s mid-stage.

C.4 COMPUTATIONAL OVERHEAD OF PRUNING METRICS

The computational-overhead analysis accounts for both the main Transformer computation and the extra cost of deciding which visual tokens to prune. It provides a common basis for comparing pruning methods.

  • Metric overheads: Attention-score overhead is 2 · Sq · Sv · H, where Sq is the number of query tokens, Sv the number of vision tokens, and H the vector dimension.The metric computes dot products between query and vision-key vectors.
  • Metric overheads: The analysis separately defines cosine-similarity and L2-norm costs for visual-token metrics before summarizing method-level overhead in Table 17.This accounts for pruning-decision computation in addition to the main model graph.
  • Cost formulation: The total FLOPs of a pruning method combine main computation with the overhead of its pruning metric, while treating LayerNorm, activations, and softmax as negligible.The framework is intended to compare methods regardless of where or how often pruning occurs.
  • Notation: The initial number of vision tokens is 576, and Sv,i denotes the remaining vision tokens after pruning stage i.These quantities are used in the computational-overhead accounting.

D.1 LLAVA

The LLAVA analysis examines attention flow, vision-text alignment, token contributions, and pruning visualizations across VQA and visual grounding. It also compares feature extraction patterns in CLIP and SIGLIP2.

  • Attention Flow and Alignment: Attention-flow and vision-text-similarity analyses characterize how multimodal information is processed across layers and tasks.The analysis includes complete attention flow, two-dimensional similarity visualization, and layer-wise similarity heatmaps.
  • Pillar Token: 72% VQA and 66% VG top-3 overlap rates link high-L2-norm “register” tokens with the highest weighted-attention tokens.The overlap is computed between the three highest-norm register tokens and the three highest weighted-attention tokens.
  • Pruning Results: The pruning visualizations report that Nüwa preserves the integrity of the global space.The reported visual results are presented as cropped examples of pruning outcomes.
  • Vision Encoder Analysis: CLIP visualizations include attention maps and object-centric similarity maps, while SIGLIP2 uses token-received attention values because it lacks a CLS token.SIGLIP2’s VAE and OCC metrics accompany its sample visualizations.
  • Vision Encoder Analysis: Compared with CLIP, SIGLIP exhibits a later phase of fine-grained feature extraction and a less pronounced overall trend.This comparison is based on visual results and metrics including OOC.

E CASE STUDY

The case study compares token selection without regional partitioning, RPME, and the full Nüwa configuration. Regional partitioning preserves spatial coverage, RPME reduces positional bias but can distort box shape, and some failures arise from target misinterpretation.

  • Configurations: The case study evaluates token selection without regional partitioning, RPME, and Nüwa’s full configuration, alongside a model-comprehension failure case.These settings are used to examine pruning and localization under several configurations.
  • Regional Partitioning: Regional partitioning preserves all four image corners, whereas insufficient spatial information biases predictions toward the top-left corner.The comparison attributes the difference to preservation of global positioning during token pruning.
  • RPME: RPME removes the consistent top-left localization bias, but stretching operations can produce problematic predicted-box aspect ratios.The overall position becomes more accurate while the predicted shape remains problematic.
  • Model Comprehension: Localization can fail when the VLM misidentifies the target, assigning extremely high attention to the wrong region and producing an incorrect bounding box.The successful case shows correlation between predicted boxes and highly attended vision-token regions; the failure case does not identify the intended target.
Loading 2602.02951v1…