Source-linked AI summary
MARS-CLIP: Multi-Resolution and Attention Refined Zero-Shot Image Segmentation
Nagito Saito, Shintaro Ito, Koichi Ito, Takafumi Aoki
TL;DR
CLIP-based zero-shot segmentation is limited by low spatial resolution and loss of structural information needed for dense prediction. MARS-CLIP combines multi-resolution global-local feature fusion with spatial and color bias injection into final attention, and experiments on six public datasets show significant outperformance of state-of-the-art methods.
Problem
CLIP-based zero-shot segmentation struggles with low spatial resolution and lost structural information, limiting tiny-object detection and complex-boundary delineation.
Method
MARS-CLIP fuses local multi-resolution features with global context and injects intermediate spatial and color biases into the final attention block.
Results
MARS-CLIP significantly outperforms state-of-the-art methods across experiments on six public datasets.
Takeaways & Limitations
Integrating local and global features with spatial and color bias refinement restores object boundaries while preserving semantic consistency.
Takeaways & Limitations
Color affinity bias may degrade in low-light or low-contrast scenes, leaving full illumination robustness for future work.
Abstract
from arXiv · showhide
Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in zero-shot transfer but often struggles with dense prediction tasks due to low spatial resolution and the loss of structural information. To address these limitations, we propose MARS-CLIP (Multi-resolution and Attention Refined Segmentation for CLIP), a novel framework for zero-shot semantic segmentation. Our approach introduces two key strategies: (i) a multi-resolution feature extraction module that fuses local fine-grained features with global context to overcome input resolution constraints, and (ii) an attention refinement mechanism that injects spatial and color biases from intermediate layers into the final self-attention block to accurately restore object boundaries. A set of experiments on six public datasets demonstrates that MARS-CLIP significantly outperforms state-of-the-art methods.
1. INTRODUCTION
Zero-shot segmentation aims to recognize unseen classes without dense retraining, but CLIP-based methods struggle with low spatial resolution and lost structural details. MARS-CLIP addresses these limitations through multi-resolution feature fusion and attention refinement using low-level information.
- Dense segmentation methods require large pixel-level datasets, while recognizing unseen classes otherwise demands additional annotation and model retraining.
- CLIP-based zero-shot segmentation exploits shared image-text representations but inherits limitations from CLIP’s global, low-resolution feature extraction.
- Fixed low-resolution feature maps make tiny objects and complex boundaries difficult to detect accurately.
- Deep-layer semantic abstraction dilutes shape information, while edges and colors remain underused and can produce ambiguous masks.
- MARS-CLIP fuses local grid-patch features with global context and injects spatial and color biases into attention to refine boundaries.The method also removes residual connections and feed-forward networks in the final layer to reduce noise.
2. RELATED WORK
Prior work adapted CLIP for zero-shot segmentation by modifying attention to improve pixel-level correspondence, but noise and spatial entanglement remained concerns. MARS-CLIP builds on these efforts with multi-resolution features and boundary-oriented attention refinement.
- CLIP maps images and text into a shared feature space, enabling strong generalization but making direct pixel-level segmentation difficult.
- MaskCLIP modified the last attention layer for pixel-wise features, but its prediction masks contained significant noise.
- MARS-CLIP combines multi-resolution global-local features with final-layer spatial and color bias injection for boundary refinement.
- CLIP Surgery introduced Value-Value attention to reduce spatial redundancy and improve feature distinctiveness.
- SCLIP used Query-Query and Key-Key similarities to address spatial feature entanglement, while ClearCLIP identified final-layer residual connections as a noise source.
3. MARS-CLIP
MARS-CLIP combines multi-resolution feature extraction with attention refinement to improve dense prediction while preserving local details, global context, and object boundaries. It then produces zero-shot masks by matching refined image features with text features.
- 3.1. Feature Extraction with Multi-resolution Images: Multi-resolution inputs combine local fine-grained features with global context, producing high-resolution feature maps despite CLIP’s fixed input size.Local patches are recombined spatially, while the resized full image supplies global features; bilinear upsampling aligns their resolutions before fusion.
- 3.2. Structure-Aware Attention Refinement: The refined attention mechanism injects intermediate spatial correlations and input-image color affinities into final KK⊤ attention to improve boundary consistency.The resulting attention map applies Softmax after adding the combined bias to the KK⊤ similarity matrix.
- 3.2. Structure-Aware Attention Refinement: Internal bias uses intermediate attention maps that preserve spatial layout, while external color bias uses CIELAB patch features and excludes the class token.The color affinity is computed with a Gaussian kernel, using σ = 30.0; removing residual connections and the feed-forward network further emphasizes refined attention.
- 3.3. Zero-Shot Semantic Segmentation: MARS-CLIP generates class text features from ImageNet templates and assigns each pixel the class with maximum cosine similarity to its refined image feature.The similarity map is upsampled to the original image size before the final prediction mask is formed.
4. EXPERIMENTS AND DISCUSSION
Experiments validate MARS-CLIP’s multi-resolution and attention-refinement components through ablations and comparisons across six benchmarks. The method improves segmentation, especially for high-resolution images and fine-grained objects, while adding only negligible inference latency.
- Experimental Setup: Evaluations cover six standard segmentation benchmarks and compare MARS-CLIP with state-of-the-art methods under settings with and without PAMR post-processing.The experiments follow protocols used by prior studies and include implementation and ablation analyses.
- Ablation Study: mIoU improves when global and local features are fused, with performance maximizing around α = 0.8 and crop size 112.Using only local or global features performs worse, while smaller crops provide insufficient context.
- Ablation Study: Attention-map biases combined with color information achieve the highest accuracy, while scores peak around the 8th layer before declining at deeper layers.The results support using intermediate spatial information and color cues to recover object boundaries.
- Ablation Study: Adding multi-resolution features improves over standard CLIP, and adding attention refinement produces further gains, particularly on City and ADE.The two modules provide complementary improvements by enhancing resolution and restoring boundary information.
- Comparison with State-of-the-Art Methods: 38.2% mIoU on City without PAMR exceeds NACLIP’s 35.5% by 2.7 points, while 40.1% with PAMR exceeds 38.3% by 1.8 points.MARS-CLIP surpasses existing methods across all evaluated datasets and settings, with strong gains on high-resolution and fine-grained scenes.
- Efficiency: MARS-CLIP uses 1.1 GB peak memory and 0.0403 seconds per image versus NACLIP’s 1.1 GB and 0.0390 seconds on V21.The reported latency overhead is 0.0013 seconds per image, while peak memory is unchanged.
5. LIMITATIONS
The color affinity bias can degrade in low-light or low-contrast scenes, while intermediate-layer spatial bias only partially mitigates this limitation.
- Color affinity bias may degrade in low-light or low-contrast scenes.Full illumination robustness remains future work, despite partial mitigation from intermediate-layer spatial bias.
6. CONCLUSION
MARS-CLIP is a zero-shot segmentation framework that combines multi-resolution inputs with attention refinement to capture fine image details. Experiments on six public datasets demonstrate that it significantly outperforms state-of-the-art methods.
- MARS-CLIP captures fine image details through multi-resolution inputs and attention mechanism refinement.The framework integrates local and global features while injecting spatial and color information as biases into the final layer.
- MARS-CLIP restores object boundaries while preserving semantic consistency.This is achieved through high-resolution feature representation and refined attention biases.
- Six public-dataset experiments show that MARS-CLIP significantly outperforms state-of-the-art methods.