PaperlayerView exact revision on arXiv

Source-linked AI summary

Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation

Yuhan Liu, Yixiong Zou, Yuhua Li, Ruixuan Li

arXiv:2608.26142v1cs.CLcs.AI

TL;DR

MLLM-based referring expression segmentation faces high inference costs, while existing token compression methods can severely degrade its performance. PAYN is a training-free, position-only compression method that preserves spatial structure and original positional indices, and experiments show improved efficiency while maintaining performance.

  • Problem

    Acceleration for MLLM-based referring expression segmentation is rarely studied, despite its high inference cost from quadratic attention scaling.

  • Method

    PAYN is a plug-and-play, training-free token compression method that selects spatially distributed tokens and preserves their original positional indices.

  • Results

    Existing token compression methods maintain over 90% of original performance on regular vision-language tasks but can retain only about 20% on RES, whereas PAYN outperforms existing approaches while maintaining performance.

  • Takeaways & Limitations

    Visual-token positional information is substantially more critical for RES than for other tasks, supporting position-based compression for this setting.

  • Takeaways & Limitations

    PAYN may become suboptimal when background regions are uninformative and the retained-token budget is extremely small, because local spatial structures may not be preserved.

Abstract

from arXiv · show

Referring Expression Segmentation (RES) aims to generate pixel-wise segmentation masks from complex and implicit textual queries. While recent advances in Multimodal Large Language Models (MLLMs) have substantially boosted RES performance, their prohibitive computational overhead remains a critical bottleneck, which, however, is rarely explored. To fill this gap, we first evaluate typical token compression methods on this task and observe a surprising performance degradation. In this paper, we aim to understand this phenomenon for a solution. By extensive experiments, we find that token compression for RES requires preserving the original position embeddings and local neighboring spatial structures, indicating that visual token position information is far more critical than in other tasks. Building on this insight, we ask: Can we design the token compression method purely based on the position information? Therefore, we propose PAYN, a plug-and-play, training-free token compression method that relies solely on position information. PAYN retains tokens that are adequately distributed in every local neighboring region while strictly preserving original positional indices, thereby maintaining spatial relational consistency. Experiments on multiple RES benchmarks demonstrate that our method outperforms existing token compression methods, verifying that position is indeed all you need for token compression in the MLLM-based RES task. Codes are avaliable at https://github.com/YuhanLiu231/PAYN.

1. Introduction

RES uses complex textual descriptions to generate pixel-level segmentation masks, but MLLM-based approaches incur high inference costs because attention scales quadratically with sequence length. The paper finds that existing token compression methods severely degrade RES performance and proposes PAYN, a training-free method that preserves positional structure through position-only token selection.

  • RES predicts pixel-level masks from complex or implicit textual descriptions, unlike traditional category-based segmentation.
  • MLLM-based RES has high inference costs because attention computation scales quadratically with input sequence length, limiting practical deployment.
  • Existing token compression methods cause severe performance degradation when applied to MLLM-based RES, an underexplored acceleration problem.
  • Experiments show that visual-token positional information is substantially more critical for RES than for other tasks.
  • PAYN is a plug-and-play, training-free compression method that relies solely on positional information, preserves original indices, and retains spatially distributed tokens.It has checkerboard-style spatial sampling and farthest point sampling instantiations.
  • Extensive experiments show that PAYN outperforms existing state-of-the-art token-pruning approaches for MLLM-based RES.

2. Related Work

Related work develops MLLM-based approaches for referring expression segmentation through embedding, text-generation, and polygon-coordinate paradigms, while visual token compression targets efficiency by exploiting spatial redundancy. Compression methods differ in whether they require finetuning, calibration data, or neither.

  • MLLM-based RES: RES segments target-object pixels from natural-language referring expressions, with MLLMs enabling unified language-instructed segmentation across images and videos.InstructSeg is cited as an example of a unified framework for language-instructed segmentation.
  • MLLM-based RES: Embedding-as-mask methods are complemented by Text4Seg’s text-as-mask paradigm, which reformulates segmentation as text generation using semantic textual descriptors.Text4Seg encodes images into sequences of semantic textual descriptors for mask prediction.
  • MLLM-based RES: Polygon-coordinate methods use MLLMs to generate mask coordinates but often struggle to achieve satisfactory performance.
  • Visual token compression: Visual token compression improves MLLM efficiency by exploiting the greater spatial redundancy of visual signals compared with text.
  • Visual token compression: Compression methods range from finetuning-based and calibration-dataset-based strategies to training-free approaches.The passage distinguishes methods by optimization requirements and pruning-strategy selection.

3. Rethinking Token Compression for Referring Expression Segmentation

The section shows that token compression in MLLM-based RES is unusually dependent on positional information: preserving original position indices and local spatial structures prevents the degradation seen with common methods. Controlled experiments validate these two requirements and motivate position-only compression.

  • Compression setup: RES compresses 576 visual tokens to 192 in a unified Text4Seg baseline with LLaVA-1.5-7B, but existing methods show significant performance degradation.The compression operator produces compact visual representations that are combined with language tokens before decoding the segmentation mask.
  • Position indices: Attention- and diversity-based methods outperform similarity-based methods, consistent with pruning that preserves original position indices rather than merging or re-indexing tokens.Similarity-based methods reorganize tokens into contiguous sequences, whereas diversity-based methods retain original positional indices.
  • Position indices: Preserving original position indices gives attention- and similarity-based methods a clear advantage and performance comparable to diversity-based methods.This controlled comparison keeps the number of retained tokens unchanged while varying whether original indices are preserved.
  • Position indices: Replacing original indices with contiguous ones causes a sharp performance drop in diversity-based methods, establishing that original positional embeddings are essential for RES compression.This result directly supports Insight 1.
  • Motivation: Together, the findings motivate asking whether position alone is sufficient for token compression in MLLM-based RES.The proposed direction follows from RES dependence on original positional information and local spatial structures.
  • Local spatial structures: Dense prediction tasks experience substantially larger performance drops than sparse prediction tasks under equivalent local spatial perturbations, validating RES sensitivity to local spatial consistency.Sparse tasks can rely on spatially distant semantic patches, whereas RES requires consistent local neighborhoods for pixel-wise prediction.

4. Method

PAYN is a plug-and-play, training-free token compression method for RES that uses only positional information, without attention maps or feature information. It preserves local spatial coverage through position-guided token selection and retains original image-token positional indices before decoding the LLM output into a segmentation mask.

  • Position-guided token compression: PAYN uses positional information alone for plug-and-play, training-free token compression without attention maps or feature information.The method is presented as a free-lunch solution for RES.
  • Position-guided token selection: Fcomp maintains enough tokens in every local neighboring region to provide broad and uniform spatial coverage after compression.The design responds to RES sensitivity to local spatial variations and local spatial structures.
  • Checkerboard-Style Spatial Sampling: Checkerboard-style sampling selects patch tokens on a regular 2D grid using a fixed offset and stride, preserving their original positional indices.The retained tokens form a spatially uniform checkerboard-like distribution without semantic information.
  • Farthest Point Sampling (FPS): Farthest point sampling selects tokens directly in spatial coordinate space by iteratively maximizing distance from the already selected set.Starting from an initially random token, the procedure continues until K tokens are selected and retains their original positional indices.
  • Position index preserving: Compressed image tokens retain their original positional indices while subsequent instruction and output tokens shift to preserve sequence order.The projected compressed visual tokens are concatenated with text tokens before entering the LLM.
  • Position index preserving: Position index preserving retains image-token spatial structure in the LLM, whose output is combined with the image and fed to a decoder for mask prediction.This positional strategy is intended to enable more precise dense predictions.

5. Experiments

Experiments across standard RES benchmarks show that PAYN consistently outperforms alternative token-compression strategies while preserving spatial information and achieving improved computational efficiency. Ablation and extension studies further validate original position-index preservation, position-guided selection, backbone generalizability, and post-encoder compression.

  • Baselines and Models: PAYN is evaluated with Text4Seg and InstructSeg on RefCOCO, RefCOCO+, and RefCOCOg using cumulative Intersection-over-Union.Text4Seg uses LLaVA-7B, whereas InstructSeg is built on Mipha-3B.
  • Validating the Position Is All You Need Perspective: PAYN consistently outperforms attention-, similarity-, diversity-, L2-norm-, edge-, and other position-based token-selection alternatives.Alternative selections often concentrate tokens in high-activation or boundary regions, producing imbalanced spatial coverage; PAYN better covers local neighboring regions.
  • Effectiveness of Position Index Preserving: Removing PAYN’s preserved original positional indices and replacing them with continuous indices causes severe performance degradation.The same trend appears across other compression methods, supporting the position-index preserving module.
  • Effectiveness of Position-Guided Token Selection: Position-guided token selection outperforms center-biased, fixed-stride row/column, and group-wise random sampling by providing broad spatial coverage and preserving local structures.These comparisons validate retaining sufficient tokens within each local neighboring region.
  • Efficiency Analysis: PAYN improves the performance–efficiency trade-off on Text4Seg and InstructSeg while selecting tokens solely from positional information, with masks that can be precomputed and reused.Efficiency is assessed using total runtime, inference speedup, and TFLOPs on RefCOCO|TestA with one A6000 GPU.
  • Extension to Other MLLM Backbone: PAYN remains effective on a DeepSeekVL-based Text4Seg baseline, but compressing patches before the vision encoder causes complete performance collapse.Early vision-encoder self-attention requires neighboring-token interactions to aggregate local features, whereas PAYN compresses after the vision encoder and before the LLM.

6. Limitation and Future Direction

PAYN can become suboptimal in rare extreme cases when very few tokens are retained and background regions contain little or no informative content, because local spatial structures may not be preserved.

  • 6. Limitation and Future Direction: PAYN may fail to preserve local spatial structures when background regions contain little or no informative content and the retained-token count is extremely small.These limitations are illustrated in the right panel of Fig. 8(b).
  • 6. Limitation and Future Direction: Prioritizing regions with higher information content could be necessary in such extreme cases, which are described as extremely uncommon.

7. Conclusion

Token compression for MLLM-based referring expression segmentation is underexplored, and existing methods often degrade severely on RES. The paper finds visual-token positional information especially important and introduces PAYN, a plug-and-play, training-free method based solely on position information.

  • PAYN may fail in extreme cases where objects are very small and only a few tokens are retained.
  • Existing token compression methods developed for other vision-language tasks often suffer severe performance degradation when applied to RES.
  • Positional information of visual tokens plays a substantially more critical role in RES than in other tasks.
  • PAYN is a plug-and-play, training-free token compression method that relies solely on positional information.

A. Dataset Description

The method is evaluated on the RefCOCO, RefCOCO+, and RefCOCOg referring expression segmentation benchmarks, which provide image, expression, and annotated-object data with differing description characteristics.

  • Dataset scope: Evaluation uses the RefCOCO, RefCOCO+, and RefCOCOg benchmarks.Figure 8 presents samples from all three datasets.
  • RefCOCO: RefCOCO contains 19,994 images, 142,209 referring expressions, and 50,000 annotated objects.Its expressions use both spatial location and visual appearance attributes.
  • RefCOCO+: RefCOCO+ includes 19,992 images, 141,564 expressions, and 49,856 annotated objects.Unlike RefCOCO, it restricts location-related descriptions and mainly focuses on appearance.

B. Details of Baseline Methods

The study uses Text4Seg and InstructSeg as representative RES baselines with distinct mask-generation paradigms and architectures. Their compression ratios are selected separately to balance segmentation performance and inference efficiency.

  • The evaluation adopts Text4Seg and InstructSeg as representative RES baselines.
  • Text4Seg: Text4Seg divides images into a 16×16 patch grid and assigns semantic textual labels row-wise under a text-as-mask paradigm.SAM further refines the resulting pixel-level semantic masks.
  • InstructSeg: InstructSeg uses an embedding-as-mask paradigm for images and videos, combining an object-aware video perceiver, MLLM, visual encoder, VMTF, and segmentation decoder.Its OVP compresses temporal and object-aware information from video frames into compact tokens processed jointly with text tokens by the MLLM.
  • Different compression ratios are adopted for the two baselines because their architectures and structural complexity differ.The ratios aim to balance segmentation performance and inference efficiency.
  • InstructSeg: Compared with Text4Seg, InstructSeg adds a vision encoder branch that helps alleviate information loss from token compression.

C. Hyperparameters of the Compared Methods

The compared methods use well-tuned configurations for fair comparison, with hyperparameters summarized under the Text4Seg baseline’s 192-token setting and identical settings adopted for InstructSeg.

  • C. Hyperparameters of the Compared Methods: All compared methods use well-tuned configurations to ensure a fair comparison.The comparison is conducted under the 192-token setting of the Text4Seg baseline.
  • C. Hyperparameters of the Compared Methods: InstructSeg adopts the same hyperparameter settings as the comparison methods.The hyperparameters are summarized in Table 10 under the Text4Seg baseline’s 192-token setting.
Loading 2608.26142v1…