Source-linked AI summary
CamoFormer: Masked Separable Attention for Camouflaged Object Detection
Bowen Yin, Xuying Zhang, Qibin Hou, Bo-Yuan Sun, Deng-Ping Fan, Luc Van Gool
TL;DR
Camouflaged object detection must distinguish objects that closely resemble their surroundings, a challenge for existing methods in complex scenes. CamoFormer introduces masked separable attention with progressive top-down refinement, and reports superior performance across three COD benchmarks and recent SOTA methods.
Problem
Camouflaged objects resemble their surroundings or are small, while existing methods struggle with fine shapes in complex scenes.
Method
CamoFormer separates attention heads to process foreground and background regions and uses progressive refinement to improve segmentation at increasing feature resolutions.
Results
CamoFormer surpasses recent SOTA methods on three COD benchmarks and reports clear improvements over 18 SOTA models.
Takeaways & Limitations
The proposed MSA and progressive refinement framework provides a high-performing approach for camouflaged object segmentation.
Abstract
from arXiv · showhide
How to identify and segment camouflaged objects from the background is challenging. Inspired by the multi-head self-attention in Transformers, we present a simple masked separable attention (MSA) for camouflaged object detection. We first separate the multi-head self-attention into three parts, which are responsible for distinguishing the camouflaged objects from the background using different mask strategies. Furthermore, we propose to capture high-resolution semantic representations progressively based on a simple top-down decoder with the proposed MSA to attain precise segmentation results. These structures plus a backbone encoder form a new model, dubbed CamoFormer. Extensive experiments show that CamoFormer surpasses all existing state-of-the-art methods on three widely-used camouflaged object detection benchmarks. There are on average around 5% relative improvements over previous methods in terms of S-measure and weighted F-measure.
1. Introduction
Camouflaged object detection is difficult because objects resemble their surroundings, while existing methods struggle with fine shapes in complex scenes. CamoFormer addresses this with masked separable attention and progressive refinement, achieving strong benchmark performance.
- Camouflaged objects can closely resemble their surroundings or be extremely small, making COD more challenging than traditional object detection.
- Existing SOTA methods still struggle to segment camouflaged targets with fine shapes because they process foreground and background cues indiscriminately.
- MSA divides attention heads into three groups to separately process foreground and background regions while retaining global interactions for segmentation.
- A progressive refinement decoder applies MSA at each feature level, improving predictions as feature resolution increases and producing finer borders.
- CamoFormer achieves new SOTA records on NC4K, COD10K, and CAMO; on COD10K-test it reaches 0.786 weighted F-measure and 0.023 MAE.
2. Related Work
COD research spans handcrafted, CNN-based, and Transformer-based approaches. CamoFormer remains within the Transformer framework but differentiates itself by assigning separate attention-head functions to foreground and background processing.
- Handcrafted COD features can handle simple scenes but show drastic performance degradation in complex conditions.
- CNN-based COD methods use strategies including multi-scale feature aggregation and multi-stage processing to address camouflage.
- Transformers are used in vision because they efficiently encode global contextual information across tasks such as segmentation and detection.
- Transformer-based COD methods include uncertainty modeling, dual-task border interaction, and progressive refinement for camouflaged-target segmentation.
- CamoFormer differs from other Transformer-based COD methods by assigning different attention-head functionalities to foreground and background regions.
- Its architecture uses a pretrained Transformer backbone, feature aggregation for coarse prediction, and an MSA-equipped decoder for progressive refinement.
3. Proposed CamoFormer
CamoFormer combines masked separable attention with a progressive encoder-decoder to distinguish camouflaged foreground and background regions and refine segmentation across feature levels. Its feature fusion and multistage supervision support progressively improved predictions, with reported average gains on three COD benchmarks.
- Overall Architecture: CamoFormer uses an encoder-decoder architecture whose decoder progressively refines features from the top encoder level using MSA at each feature level.The decoder applies bilinear upsampling for shape matching and uses MSA to distinguish camouflaged objects from background.
- Feature Fusion: CamoFormer fuses feature levels with element-wise products followed by summation, rather than only addition or concatenation, yielding over 0.2% average relative improvement in S-measure and weighted F-measure.The improvement is reported across NC4K, COD10K-test, and CAMO-test.
- Attention Formulation: MSA is built on Multi-Dconv Head Transposed Attention, which forms query, key, and value matrices using pointwise and depthwise convolutions with a learnable scaling parameter.The paper presents MSA as a modified self-attention mechanism intended to reduce computation.
- Masked Separable Attention: MSA divides attention heads into foreground, background, and normal groups to separately process predicted regions while preserving global foreground-background interactions.Foreground and background queries and keys are masked, while the normal heads remain unchanged for cross-region relationships.
4. Experimental Results
CamoFormer is evaluated on three COD benchmarks using standard metrics, qualitative comparisons, border-region analysis, and ablations. Results show strong performance, with MSA and progressive refinement improving segmentation quality and border precision.
- Experiment Setup: CamoFormer is evaluated on CAMO, COD10K, and NC4K using Structure-measure, mean absolute error, weighted F-measure, and adaptive E-measure.The implementation uses a pretrained PVTv2 encoder, with additional backbone variants also reported.
- Performance on Border Regions: CamoFormer produces prediction borders closer to ground-truth borders than competing methods on irregularly shaped camouflaged objects.The visual comparison marks ground-truth borders in white and predicted borders in orange.
- Performance on Object Regions: CamoFormer consistently surpasses previous CNN- and Transformer-based methods across all three benchmarks without post-processing tricks or extra training data.The comparison includes 12 CNN-based and 6 Transformer-based COD models.
- Performance on Border Regions: Border-region evaluation measures BR-wF and BR-M after dilating ground-truth boundaries with 15 × 15 and 30 × 30 kernels.The dilation kernel size determines the evaluated border-region area.
- Ablation Studies: Adding MSA, progressive decoder fusion, and MSA within the decoder progressively improves performance over the baseline.The ablation identifies MSA as important both in the encoder-side setup and in the progressive decoder.
- Ablation Studies: All three MSA branches contribute to performance, while combining foreground, background, and normal attention yields the best results across all three datasets.Separately processing foreground and background is reported as useful for camouflaged-object segmentation.
- Ablation Studies: The performance gap between models with and without MSA becomes more significant from feature level 5 toward feature level 1.This supports the role of MSA in progressive refinement as feature resolution increases.
5. Conclusions
CamoFormer combines masked separable attention with a progressive top-down refinement decoder for camouflaged object segmentation. The paper reports clear improvements over existing state-of-the-art models and visual gains in segmentation completeness.
- 5. Conclusions: CamoFormer uses masked separable attention to process foreground and background regions with different attention heads.A progressive refinement decoder improves segmentation quality across feature levels in a top-down manner.
- 5. Conclusions: CamoFormer surpasses 18 existing state-of-the-art models with clear improvements on camouflaged object segmentation.The paper also suggests applicability to other binary segmentation tasks.
- 5. Conclusions: Visual examples show improved completeness of CamoFormer’s segmentation results compared with previous methods.The supplementary visualization section includes examples spanning varied camouflaged scenes.
A. Performance on All Benchmark Datasets
Beyond the three main benchmarks, CamoFormer is evaluated on CHAMELEON, which contains 76 image-annotation pairs. The reported results show significant improvements over other methods.
- A. Performance on All Benchmark Datasets: CHAMELEON contains 76 image-annotation pairs and is used as an additional evaluation dataset.The main paper evaluates NC4K, COD10K-Test, and CAMO-Test, while this additional result extends evaluation to CHAMELEON.
- A. Performance on All Benchmark Datasets: CamoFormer significantly improves over all other methods on CHAMELEON.The comparison is reported in Table 7 alongside results on the other benchmark datasets.
B. More Visual Comparisons
Additional visual comparisons cover diverse camouflaged scenes and compare CamoFormer with recent state-of-the-art methods. The reported examples show stronger performance for CamoFormer.
- B. More Visual Comparisons: The additional visual examples include large and small objects, occlusion, and indefinable boundaries.These examples broaden qualitative coverage across varied camouflaged scenes.
- B. More Visual Comparisons: CamoFormer performs much better than the other methods in the additional visual comparisons.Table 7 provides corresponding quantitative comparisons across four datasets.
C. More Ablation Results of Decoder Channels
The decoder-channel ablation identifies Cd = 128 as a trade-off between performance and computation, while CamoFormer remains competitive with Cd = 32.
- C. More Ablation Results of Decoder Channels: Cd = 128 is selected as the trade-off between performance and computational cost.The ablation evaluates decoder channels across three benchmark datasets.
- C. More Ablation Results of Decoder Channels: Even with Cd = 32, CamoFormer outperforms previous methods.
D. PR & Fβ curves of COD methods
The PR and Fβ curves compare CamoFormer with previous COD methods across NC4K, CAMO, and COD10K, with higher curves indicating better performance. CamoFormer surpasses the other methods.
- D. PR & Fβ curves of COD methods: CamoFormer’s red PR and Fβ curves surpass those of previous methods across NC4K, CAMO, and COD10K.Higher curves indicate better model performance.