Source-linked AI summary
Feature Shrinkage Pyramid for Camouflaged Object Detection with Transformers
Zhou Huang, Hang Dai, Tian-Zhu Xiang, Shuo Wang, Huai-Xin Chen, Jie Qin, Huan Xiong
TL;DR
Camouflaged object detection requires subtle cues because objects can be indistinguishable from their backgrounds, while existing transformer methods have locality and decoder aggregation limitations. FSPNet uses non-local token enhancement and progressive adjacent-feature shrinking, and it reports superior performance over 24 competitors across three COD benchmark datasets.
Problem
Camouflaged object detection must identify objects with high appearance similarity to their backgrounds, while transformer-based methods have limited locality modeling and decoder feature aggregation.
Method
FSPNet combines a non-local token enhancement module with a feature shrinkage decoder using adjacent interaction modules to progressively aggregate locality-enhanced transformer features.
Results
FSPNet achieves superior performance over 24 existing methods on three widely used COD benchmark datasets.
Takeaways & Limitations
The proposed design provides a transformer-based approach for integrating local cues and global context in camouflaged object segmentation.
Takeaways & Limitations
The model does not detect camouflaged objects well under very low lighting or when objects closely resemble the background.
Abstract
from arXiv · showhide
Vision transformers have recently shown strong global context modeling capabilities in camouflaged object detection. However, they suffer from two major limitations: less effective locality modeling and insufficient feature aggregation in decoders, which are not conducive to camouflaged object detection that explores subtle cues from indistinguishable backgrounds. To address these issues, in this paper, we propose a novel transformer-based Feature Shrinkage Pyramid Network (FSPNet), which aims to hierarchically decode locality-enhanced neighboring transformer features through progressive shrinking for camouflaged object detection. Specifically, we propose a nonlocal token enhancement module (NL-TEM) that employs the non-local mechanism to interact neighboring tokens and explore graph-based high-order relations within tokens to enhance local representations of transformers. Moreover, we design a feature shrinkage decoder (FSD) with adjacent interaction modules (AIM), which progressively aggregates adjacent transformer features through a layer-bylayer shrinkage pyramid to accumulate imperceptible but effective cues as much as possible for object information decoding. Extensive quantitative and qualitative experiments demonstrate that the proposed model significantly outperforms the existing 24 competitors on three challenging COD benchmark datasets under six widely-used evaluation metrics. Our code is publicly available at https://github.com/ZhouHuang23/FSPNet.
1. Introduction
Camouflaged object detection must recover objects whose appearance closely matches the background. FSPNet addresses transformer locality limitations and decoder aggregation gaps with locality enhancement and progressive feature shrinking.
- Camouflaged object detection segments objects that are inconspicuous because they closely resemble their backgrounds.
- CNN-based COD methods struggle with long-range dependencies, limiting global feature-relation modeling and producing incomplete object regions.
- Existing transformer-based COD methods exhibit less effective local feature modeling and limited decoder aggregation of features with substantial information differences.
- FSPNet hierarchically decodes locality-enhanced neighboring transformer features through progressive shrinking to accumulate local cues and global context.
- NL-TEM enhances transformer locality by interacting neighboring tokens and exploring graph-based high-level relations within tokens.
- FSD with AIM progressively aggregates adjacent transformer features in pairs through a layer-by-layer shrinkage pyramid for camouflaged object prediction.
- FSPNet achieves superior performance against 24 existing state-of-the-art methods on three widely used COD benchmark datasets.
2. Related Work
Prior COD research uses CNN-based feature exploration, multitask, uncertainty, and bio-inspired approaches, while decoder designs follow several recurring integration strategies. Transformer COD methods improve long-range modeling but retain locality and feature-aggregation limitations, motivating progressive adjacent-feature decoding.
- CNN-based Camouflaged Object Detection: CNN-based COD methods explore contextual, texture-aware, frequency-domain, uncertainty, and bio-inspired cues, alongside multitask learning frameworks.
- CNN-based Camouflaged Object Detection: CNN-based methods remain limited in modeling long-range dependencies because of their restricted receptive fields.
- Decoder Design: Object-segmentation decoders commonly use U-shaped, dense integration, feedback refinement, or separate low-level and high-level feature strategies.
- Vision Transformer: Transformers capture long-range dependencies through self-attention and have been applied to camouflaged object detection with promising performance.
- Vision Transformer: Transformer-based COD methods still face locality-modeling and decoder feature-aggregation limitations inherited from CNN-oriented designs.
3. Proposed Method
FSPNet combines a ViT encoder with non-local token enhancement and a progressive feature shrinkage decoder to model global context, strengthen locality, and accumulate subtle camouflaged-object cues.
- Overall architecture: FSPNet uses a ViT encoder, NL-TEM, and FSD with AIMs to hierarchically decode locality-enhanced neighboring transformer features.The encoder models global context, NL-TEM enhances local representations, and FSD progressively aggregates features for camouflaged-object segmentation.
- Transformer Encoder: The ViT serializes non-overlapping image patches into token embeddings and processes them through self-attention and MLP transformer layers.Patch size is s = 16 in the experiments; positional embeddings are added before transformer encoding.
- Non-local Token Enhancement Module: NL-TEM interacts adjacent similar tokens with a non-local operation, then uses graph convolution to explore higher-order relations and enhance subtle local features.A GCN reasons over graph-based token relations, while skip connections and deserialization produce 2D features for decoding.
- Feature Shrinkage Decoder: FSD progressively aggregates adjacent feature pairs through a four-layer shrinkage pyramid containing 12 AIMs.The decoder uses adjacent interactions to flow and accumulate cues while avoiding interference from large feature differences.
- Feature Shrinkage Decoder: AIM combines adjacent features and the previous aggregated feature through convolutional processing, concatenation, and 2× upsampling.The resulting feature is passed to the next AIM, while the current AIM output is forwarded to the next decoder layer.
- Feature Shrinkage Decoder: FSD adds within-layer and cross-layer feature interaction, lateral supervision at every layer, and non-overlapping adjacent integration to preserve subtle features and reduce aggregation operations.Each layer prediction uses binary cross-entropy, with smaller weights assigned to shallow outputs having lower detection precision.
4. Experiments and Results
FSPNet is evaluated on three COD datasets against 24 state-of-the-art methods, with consistently stronger performance and ablations supporting its decoder and token-enhancement components.
- Experiment Settings: FSPNet is evaluated on CAMO, COD10K, and NC4K using six established evaluation metrics.The experiments compare the model across three widely used camouflaged object detection benchmarks.
- Comparison with State-of-the-Art Methods: FSPNet consistently surpasses 24 competing salient-object and camouflaged-object detection methods on the three benchmark datasets.The comparison includes 10 salient object detection methods and 14 COD methods.
- Comparison with State-of-the-Art Methods: Against ZoomNet, FSPNet achieves average gains of 3.0%, 3.7%, 2.7%, 1.8%, 3.0%, and 17.7% for Sα, Fωβ, Fmφ, and M across the datasets.The cited passage reports six gains corresponding to the paper’s evaluation metrics.
- Ablation Study: FSD improves performance by aggregating and retaining critical features from different transformer layers for accurate predictions.Ablations find that aggregating all transformer feature layers provides the best baseline decoding performance, while FSD improves results further.
- Ablation Study: Progressive adjacent-feature aggregation in FSD accumulates subtle camouflage cues more effectively than U-shaped decoding, which can discard valuable details.The decoder’s lateral supervision and within-layer feature flow further improve performance, especially on NC4K.
- Ablation Study: Non-local operations and graph-based feature modeling each contribute to COD performance, while their combination significantly improves FSPNet.These components are evaluated through additional ablations of NL-TEM.
5. Conclusion
The paper concludes that FSPNet addresses transformer locality modeling and decoder aggregation limitations through NL-TEM and FSD with AIMs. Experiments and ablations report superior performance over 24 methods on three COD benchmarks.
- Conclusion: FSPNet combines NL-TEM with an FSD containing AIMs to hierarchically aggregate locality-enhanced neighboring features through progressive shrinking.The design integrates local and global cues for camouflaged object detection.
- Conclusion: FSPNet achieves superior performance over 24 competing approaches on three widely used COD benchmark datasets.The conclusion attributes the result to extensive comparison experiments and ablation studies.
Appendices
The appendices provide additional details that could not be expanded in the main text.
- Appendices: The appendices contain more details that are not expanded in the main text.No specific appendix content is described in the supplied passage.
6. Datasets and Evaluation Details
The evaluation uses CAMO, COD10K, and NC4K, with five widely used COD metrics covering structural similarity, precision-recall balance, perceptual alignment, and prediction error.
- Datasets: Experiments use CAMO, COD10K, and NC4K, spanning 2.5K, 10K, and 4,121 images, respectively.CAMO includes training and testing splits; COD10K covers 78 categories; NC4K provides binary maps and ranking annotations.
- Evaluation Metrics: Five evaluation metrics are reported: S-measure, F-measure, weighted F-measure, E-measure, and mean absolute error.The paper reports adaptive variants for F-measure and E-measure.
- Evaluation Metrics: Mean absolute error calculates the average absolute difference between predicted camouflaged-object maps and ground-truth maps across image pixels.N denotes the total number of image pixels.
- Evaluation Metrics: S-measure combines region-aware and object-aware terms to calculate structural similarity for camouflaged objects with complex shapes.The balance parameter α is set to 0.5.
- Evaluation Metrics: F-measure jointly accounts for precision and recall, with β^2 set to 0.3 in this paper.Adaptive F-measure uses twice the average prediction-map pixel value as its threshold.
- Evaluation Metrics: E-measure combines pixel-level matching with image-level statistics based on human visual perception.The enhanced-alignment matrix is denoted by φFM.
7. More Comparisons
Additional comparisons evaluate FSPNet against SOD and COD methods, subclass performance, challenging visual cases, and decoder variants. The results report strong overall performance, reduced decoder computation, and remaining failures in especially difficult scenes.
- Comprehensive Evaluation: FSPNet achieves the best detection performance overall across comprehensive evaluations on three COD datasets.The comparisons include 23 state-of-the-art methods in the quantitative tables.
- Evaluation for Subclasses: FSPNet outperforms other competitors on most COD10K subclasses, while subclass-specific adaptation remains future work.Subclass results are reported using S-measure.
- Visual Comparisons: Visual comparisons cover small, large, obscured, and boundary-indistinguishable camouflaged objects, where FSPNet produces more accurate and complete predictions.The supplementary figures compare FSPNet with multiple SOD and COD competitors.
- Decoder Ablation: The decoder ablation compares variants with overlapping aggregation, lateral supervision, and within-layer feature interaction against the non-overlapping proposed design.Other modules are retained in these experiments.
- Decoder Ablation: The proposed decoder uses 4 layers and 12 AIMs instead of 11 layers and 66 AIMs, while achieving slightly better performance than the alternative.Adjacent features are fused without overlapping to reduce aggregation operations and computation.
- Failure Cases: FSPNet still struggles in some very challenging scenes, particularly under very low lighting or when objects closely resemble their backgrounds.These cases are identified as potential directions for future improvement.