PaperlayerView exact revision on arXiv

Source-linked AI summary

SCMerge: Spatial-Consistency-Aware Token Pair Merging for Efficient Semantic Segmentation

Junfeng Ren, Yi Gong, Yuan Zeng

arXiv:2610.21642v1cs.CV

TL;DR

ViT self-attention is computationally costly at high resolution, and feature-similarity-only token merging can overlook spatial consistency needed for segmentation. SCMerge pairs tokens using both feature similarity and spatial consistency, then reconstructs the dense grid after decoding. Across benchmark datasets, it achieves a better accuracy–latency trade-off than existing training-free token-merging methods.

  • Problem

    ViT self-attention has quadratic computational complexity and long latency at high resolution, while feature-similarity-only merging can overlook spatial consistency important to segmentation.

    Source: §1 · p. 1

  • Method

    SCMerge is a training-free method that pairs tokens using feature similarity and a spatial-consistency penalty, then decodes the reduced sequence and reconstructs predictions onto the original grid.

    Source: §1 · p. 1, §3 · p. 3, §3.3 · p. 5

  • Results

    Across six model–dataset settings, SCMerge improves over MPM by 0.18–0.76 mIoU while reducing full-token latency by approximately 26%–36%.

    Source: §4.2 · pp. 6–7

  • Takeaways & Limitations

    SCMerge plugs into existing ViT-based segmentation pipelines without additional supervision, extra input modalities, or modifications to the segmentation head.

    Source: §1 · p. 2

    Very small objects and extremely thin structures may still be smoothed, and fixed hand-designed merging criteria may limit adaptation to diverse or unseen visual patterns.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision Transformer architectures have showcased impressive performance on semantic segmentation, yet their quadratic attention complexity hinders practical applications. Recent works have focused on merging tokens to reduce sequence length and thus improve transformer efficiency. However, existing token merging methods primarily rely on feature similarity to identify similar tokens, overlooking spatial consistency, which may introduce segmentation errors in semantic segmentation. In this work, we propose SCMerge, a training-free token merging method that jointly considers feature similarity and spatial consistency in pairing similar tokens. The pairwise tokens are then merged within the encoder, and the resulting token sequences are reconstructed onto the original patch grid after decoding. The proposed method is flexible and can be plugged into existing vision Transformer-based semantic segmentation pipelines. Extensive experiments on the benchmark datasets ADE20K, Pascal Context, and Cityscapes show that SCMerge consistently achieves a better trade-off between segmentation accuracy and efficiency than baseline methods.

1. Introduction

ViT self-attention is costly at high resolution, while token reduction for segmentation must preserve spatial and semantic detail. SCMerge pairs tokens using feature similarity and spatial consistency to improve the accuracy–efficiency trade-off without extra supervision or changes to the segmentation head.

1. Introduction

  • ViT self-attention has quadratic computational complexity and long latency, limiting its application to high-resolution images.Token reduction must preserve the spatial and semantic details required by the segmentation decoder.Source: §1 · p. 1
  • Feature-similarity-only merging can pair boundary tokens with neighboring-region interiors or merge similar-feature tokens from different objects, blurring boundaries.Prior approaches that add spatial information or learned policies can require extra inputs or training.Source: §1 · p. 1
  • SCMerge combines feature similarity with a spatial-consistency penalty to guide token pairing and improve the accuracy–efficiency trade-off in ViT-based segmentation.The method is training-free and requires no additional supervision, extra input modalities, or segmentation-head modifications.Source: §1 · pp. 1–2
  • SCMerge estimates local feature variation to identify boundary-sensitive tokens and penalize pairs that are locally similar but spatially incompatible.Paired tokens are averaged, and the reduced sequence is decoded before being mapped back to the original token grid.Source: §1 · pp. 1–2
  • Experiments on ADE20K, Pascal Context, and Cityscapes show a better accuracy–efficiency trade-off than baseline methods relying only on feature similarity.The method is designed to integrate into existing ViT-based semantic-segmentation pipelines.Source: §1 · p. 2

2. Related Work

Prior ViT efficiency work reduces computational cost through architectural changes or token reduction, while dense prediction requires preserving fine boundaries and spatial structure. SCMerge focuses on training-free, spatial-aware token merging for segmentation.

2. Related Work

  • Efficient segmentation architectures reduce costs through designs such as hierarchical encoders, local attention, lower spatial resolution, or tiled attention computation.Examples include SegFormer, Swin Transformer, PVT, and FlashAttention.Source: §2 · p. 2
  • Token-reduction approaches include learned pruning, token selection, halting, pooling, sampling, and compression-rate methods.Many target image classification or require learned policies, retraining, or architecture-specific changes, while segmentation must preserve thin structures and fine boundaries.Source: §2 · p. 2
  • Training-free token-merging methods such as ToMe and MPM combine feature-similar tokens, but their grouping strategies largely neglect spatial consistency.Other merging approaches use learned or adaptive mechanisms, fusion, or saliency-driven criteria.Source: §2 · p. 2
  • Spatial-aware segmentation methods incorporate spatial or task-specific information, but may rely on learned policies, specialized schedules, extra spatial information, or method-specific implementations.SCMerge shares their spatial-awareness motivation while remaining training-free and plug-and-play for ViT segmentation pipelines.Source: §2 · p. 3

3. Method

SCMerge reduces ViT token sequences using feature similarity and local spatial consistency, retains a merge map, and reconstructs dense predictions after decoding. Its token-pairing score penalizes boundary-sensitive or locally inconsistent pairs.

3. Method

  • SCMerge can be inserted into multiple ViT encoder blocks, reducing N tokens to M while maintaining a merge map for reconstruction after decoding.Tokens are matched using feature similarity and a spatial-consistency penalty, then passed through the decoder in compact form.Source: §3 · p. 3, Figure 1 · p. 4

3.1. Preliminaries

  • The ViT processes patch embeddings through Transformer blocks, while the decoder combines encoder tokens with learnable class embeddings to produce patch-level class logits.The original full-token model reshapes logits to the patch grid before bilinear upsampling.Source: §3.1 · p. 3

3.2. Spatial-Consistency-Aware Token Merging

  • SCMerge estimates scalar local variation v_i by averaging channel-wise variance within each 3 × 3 neighborhood, using a deterministic d_s-channel sketch formed by averaging contiguous groups of normalized feature channels on the current sparse token layout.Source: §3.2 · p. 3
  • The spatial-consistency penalty combines boundary sensitivity and the difference in local variation between candidate tokens; subtracting it from feature similarity gives the merge score.The score favors feature-similar pairs that are less likely to involve boundary-sensitive or locally inconsistent regions.Source: §3.2 · p. 4
  • Mutual nearest-neighbor pairs under the merge score are averaged into new tokens, while unmatched tokens remain singletons and merged tokens retain representative coordinates.The merge map tracks assignments to original patch locations across merge stages.Source: §3.2 · p. 5

3.3. Decoding and Reconstruction

  • The reduced sequence passes through the decoder without spatial expansion, and the merge map restores compact logits to the original patch grid for dense prediction.For the Mask Transformer decoder, reduced tokens remain jointly processed with class embeddings; the decoder architecture and parameters remain unchanged.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate SCMerge in ViT-based segmentation pipelines on three benchmark datasets, using quantitative, qualitative, ablation, and component analyses. Across six model–dataset settings, SCMerge improves the accuracy–latency trade-off over MPM without retraining or dataset-specific fine-tuning.

4.1. Experimental Setup

  • Experiments use Segmenter checkpoints on ADE20K, Pascal Context, and Cityscapes, inserting merging after encoder blocks two and five and comparing ToMe, MPM, and MPM-top.Latency is measured for the full forward pass on an NVIDIA RTX 3090 at batch size 1 in FP32 with TF32 disabled.Source: §4.1 · pp. 5–6

4.2. Quantitative Comparison

  • Across all six model–dataset settings, SCMerge improves over MPM by 0.18–0.76 mIoU while reducing full-token latency by approximately 26%–36%.Table 1 reports comparisons across ADE20K, Pascal Context, and Cityscapes; SCMerge achieves the best accuracy–latency trade-off among training-free token-merging methods.Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.3. Qualitative Comparison

  • 7.96% pixel error rate for SCMerge versus 9.52% for ToMe, 9.66% for MPM, and 8.50% for MPM-top on ADE20K.Figure 3 compares predictions and pixel-wise error maps.Source: Figure 3 · p. 7, §4.3 · p. 8

4.4. Ablation

  • The spatial-consistency penalty improves accuracy at a slight latency cost, while post-decoder reconstruction significantly reduces latency with a slight accuracy loss; together they improve the overall trade-off.Table 2 ablates the penalty and reconstruction choices on ADE20K with Segmenter-B/16.Source: Table 2 · p. 8, §4.4 · p. 8

4.5. Analyses

  • Earlier merging saves more computation but risks incorrect pairing; later merging preserves finer spatial structure and accuracy with smaller latency gains, motivating early-to-middle encoder placement.Figure 4 compares MPM and SCMerge across zero-based encoder-block insertion indices.Source: Figure 4 · p. 8, §4.4 · p. 8
  • SCMerge improves B-mIoU@5 and sharply reduces B-Merge, consistent with the boundary-sensitivity term Bij discouraging boundary-token merges.Table 3 reports these boundary measures on ADE20K.Source: Table 3 · p. 8, §4.5 · p. 8
  • SCMerge produces fewer cross-class pairs and higher N-Agree, indicating that Dij promotes selection of semantically consistent token pairs.Table 4 reports neighborhood consistency on ADE20K.Source: Table 4 · p. 8, §4.5 · p. 8

5. Conclusion

SCMerge combines feature similarity and spatial consistency in a training-free token-merging method for ViT-based semantic segmentation. Across three benchmark datasets, it achieves a better accuracy–latency trade-off than state-of-the-art training-free merging methods, though small objects and thin structures may still be smoothed.

5. Conclusion

  • SCMerge is a training-free token-merging method for semantic segmentation that combines feature similarity with spatial consistency.It can be plugged into existing ViT-based segmentation models to improve computation efficiency with less quality drop.Source: §5 · p. 8
  • Across ADE20K, Pascal Context, and Cityscapes, SCMerge achieves a better accuracy-latency trade-off than state-of-the-art training-free token-merging methods.Very small objects and extremely thin structures may still be smoothed by information aggregation.Source: §5 · p. 9
  • SCMerge uses fixed, hand-designed merging criteria, which may limit its adaptability to diverse or unseen visual patterns.Addressing this limitation while preserving its plug-and-play and training-free advantages is left for future work.Source: §5 · p. 9

Related papers

Loading 2610.21642v1…