PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale and Detail-Enhanced Segment Anything Model for Salient Object Detection

Shixuan Gao, Pingping Zhang, Tianyu Yan, Huchuan Lu

arXiv:2408.04326v1cs.CVcs.MM

TL;DR

SOD methods can perform poorly and generalize weakly in complex cases, while SAM requires target prompts unavailable for SOD and lacks multi-scale, multi-level, and fine-grained information. MDSAM adapts SAM with LMSA, MLFM, and DEM, and is reported to perform well on multiple SOD datasets and generalize to other segmentation tasks.

  • Problem

    SOD methods can have low performance and poor generalization in complex cases, while SAM requires accurate target prompts unavailable in SOD and lacks multi-scale, multi-level, and fine-grained information.

    Source: §1 · pp. 1–2

  • Method

    MDSAM adapts SAM for SOD using LMSA for multi-scale learning with few trainable parameters, MLFM for encoder-level feature fusion, and DEM for incorporating fine-grained details.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    MDSAM is reported to perform well on multiple SOD datasets and show strong generalization on other segmentation tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework preserves SAM's pre-trained weights while incorporating multi-scale and fine-grained information, and experiments verify its effectiveness and generalization.

    Source: §5 · p. 8

Abstract

from arXiv · show

Salient Object Detection (SOD) aims to identify and segment the most prominent objects in images. Advanced SOD methods often utilize various Convolutional Neural Networks (CNN) or Transformers for deep feature extraction. However, these methods still deliver low performance and poor generalization in complex cases. Recently, Segment Anything Model (SAM) has been proposed as a visual fundamental model, which gives strong segmentation and generalization capabilities. Nonetheless, SAM requires accurate prompts of target objects, which are unavailable in SOD. Additionally, SAM lacks the utilization of multi-scale and multi-level information, as well as the incorporation of fine-grained details. To address these shortcomings, we propose a Multi-scale and Detail-enhanced SAM (MDSAM) for SOD. Specifically, we first introduce a Lightweight Multi-Scale Adapter (LMSA), which allows SAM to learn multi-scale information with very few trainable parameters. Then, we propose a Multi-Level Fusion Module (MLFM) to comprehensively utilize the multi-level information from the SAM's encoder. Finally, we propose a Detail Enhancement Module (DEM) to incorporate SAM with fine-grained details. Experimental results demonstrate the superior performance of our model on multiple SOD datasets and its strong generalization on other segmentation tasks. The source code is released at https://github.com/BellyBeauty/MDSAM.

1 Introduction

SOD methods face poor performance and generalization in complex cases, while applying SAM to SOD is hindered by its prompt requirements and limited multi-scale and detail information. MDSAM adapts SAM with multi-scale, multi-level, and detail-enhancement components, and is reported to perform well on SOD and other segmentation tasks.

1 Introduction

  • CNN- and Transformer-based SOD methods still show low performance and poor generalization in complex cases.The introduction identifies insufficient training examples and large domain gaps as challenges for these methods.Source: §1 · p. 1
  • SAM requires target-specific prompts that are unavailable at SOD inference, while grid prompts cannot identify salient objects and selected prompts require accurate placement.The introduction also notes that box prompts may be ineffective in scenes with object occlusion.Source: §1 · p. 1
  • MDSAM combines LMSA for parameter-efficient multi-scale learning, MLFM for encoder-level feature fusion, and DEM for incorporating image details and edges.The introduction describes DEM as helping generate precise, detailed predictions.Source: §1 · p. 2
  • Experiments are reported to show MDSAM's effectiveness on multiple SOD datasets and strong generalization to other segmentation tasks.The contribution summary describes the evaluation as extensive.Source: §1 · p. 2

2 Related Work

Prior SOD work uses CNNs and Transformers to capture different feature types, but the cited discussion identifies limits in global perception, fine-grained detail, or generalization. Prior SAM adaptations also leave multi-scale and detail information insufficiently addressed, motivating MDSAM's combination of these capabilities.

2.1 Salient Object Detection

  • CNN-based SOD methods aggregate multi-scale features, but the related-work discussion identifies limited global perception as a central challenge for CNNs.It also describes methods targeting multi-level features, uncertainty, refinement, edges, and detail perception.Source: §2.1 · p. 2
  • Transformer-based SOD methods use global perception and multi-scale features, yet the cited discussion says they lack fine-grained detail perception and generalize poorly in complex cases.The paper positions visual foundation models as a way to transfer feature extraction and generalization capabilities to SOD.Source: §2.1 · p. 2

2.2 Segment Anything Model

  • Prior SAM transfers to SOD leave multi-scale and multi-level information insufficiently used, while SAM's simple decoder does not incorporate enough detail for accurate segmentation.MDSAM's architecture equips SAM with LMSA, MLFM, and DEM, alongside WD and MEEM to improve feature representation.Source: §2.2 · p. 3, Figure 2 · p. 3

3 Our Proposed Method

MDSAM equips SAM for salient object detection with lightweight multi-scale adaptation, encoder-level feature fusion, and image-detail enhancement. Its modules preserve SAM’s pretrained weights while addressing the task’s need for multi-scale and fine-grained information.

3 Our Proposed Method

  • MDSAM equips SAM with LMSA, MLFM, and DEM to adapt it to salient object detection with multi-scale, multi-level, and detail-enhanced information.The framework reuses and preserves SAM’s pretrained weights while incorporating multi-scale information with very few trainable parameters.Source: §3 · p. 3, §3.1 · p. 3

3.1 Lightweight Multi-Scale Adapter

  • LMSA adapts SAM using very few parameters, combining pooled multi-scale features with depth-wise convolutions for local-detail information.It is placed before the first normalization in each encoder Transformer layer.Source: §3.1 · pp. 3–4

3.2 Multi-Level Fusion Module

  • MLFM weights features from different SAM encoder layers, integrating shallow low-level details and deep high-level semantics for the mask decoder.It addresses the limits of using only the final encoder layer and of simple concatenation fusion.Source: §3.2 · p. 4

3.3 Detail Enhancement Module

  • DEM combines progressively upsampled decoder features with an auxiliary branch that extracts multi-scale image edges and fine-grained details.MEEM uses average pooling and 1 × 1 convolutions to reduce computation, while Edge Enhancers highlight object edges.Source: §3.3 · p. 5

3.4 Loss Functions

  • MDSAM trains with BCE, IoU, and L1 losses applied to both the final prediction and a projection of the mask-decoder feature.The provided table passages identify quantitative-comparison tables but do not supply their result values.Source: §3.4 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4 Experiments

MDSAM achieves the best overall SOD results despite weaker performance on PASCAL-S, and at 384 × 384 outperforms fully fine-tuned SAM at 512 × 512 in both speed and accuracy. Ablations report benefits from LMSA, MLFM, and MEEM, while experiments use five SOD benchmarks and compare against 15 other models.

4.1 Experiment Settings

  • Experiments train MDSAM on DUTS-TR and evaluate it on five SOD benchmarks, comparing it with 15 other models.SAM in the comparison is fully fine-tuned for SOD; evaluation uses the same metric-calculation tool across methods.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 Comparison with the State-of-the-arts

  • MDSAM achieves the best overall SOD results despite inferior performance on PASCAL-S, and at 384 × 384 beats SAM at 512 × 512 in speed and accuracy.At 512 × 512, MDSAM achieves the best results on DUTS-OMRON, HKU-IS, and ECSSD and is highly competitive on DUTS.Source: §4.2 · p. 7
  • MDSAM locates salient objects of varying sizes and recognizes their shapes, fine-grained details, and edges more accurately in complex scenarios than other methods.These observations come from the qualitative comparison of predicted saliency maps.Source: §4.2 · p. 7

4.3 Ablation Studies

  • LMSA has parameter counts similar to Adapter and LoRA but produces better results, with visual comparisons indicating improved multi-scale object localization.The ablation compares encoder adaptation strategies while keeping Adapter and LoRA parameter counts similar to LMSA.Source: §4.3 · p. 7, Figure 8 · p. 8
  • MLFM significantly improves results over using no fusion, whereas simple concatenation yields only marginal gains and may introduce noisy features.Visual comparisons report that MLFM better recognizes whole-object shape and contour than simple concatenation.Source: §4.3 · p. 8, Figure 7 · p. 8
  • DEM without MEEM yields only slight improvement, while adding MEEM significantly improves performance and fine-grained detail and edge predictions.The ablation compares DEM with and without its MEEM component.Source: §4.3 · p. 8, Table 4 · p. 8

5 Conclusion

The conclusion presents MDSAM as a SOD feature-learning framework that preserves SAM’s pretrained weights while incorporating multi-scale and fine-grained information. It reports experimental evidence for the method’s effectiveness and generalization.

5 Conclusion

  • MDSAM preserves SAM’s pretrained weights while using LMSA for multi-scale learning, MLFM for encoder-layer fusion, and DEM to address missing fine-grained details.The paper concludes that experiments verify the framework’s effectiveness and strong generalization.Source: §5 · p. 8

A.1 Model Generalization

The paper evaluates MDSAM beyond SOD on camouflaged object detection and polyp segmentation. It reports comparable COD performance and superior polyp-segmentation performance, with better results than Adapter-enhanced SAM on both tasks.

A.1 Model Generalization

  • MDSAM achieves comparable performance on camouflaged object detection and shows more precise localization and fine-grained details in qualitative evaluation.The COD experiments train on COD10K and CAMO training data and test across the COD datasets.Source: §A.1 · p. 11
  • MDSAM achieves superior performance on polyp segmentation and outperforms Adapter-enhanced SAM on both polyp segmentation and COD.Polyp segmentation is evaluated on Kvasir and CVC-300.Source: §A.1 · p. 11, Table 6 · p. 12
  • Across the reported tasks, the authors conclude that MDSAM performs excellently on SOD and strongly on COD and polyp segmentation.The paper presents these results as evidence of the model’s generalization capability.Source: §A.1 · p. 12

A.2 Zero-shot Analysis

With accurate box prompts extracted from ground-truth connected components, SAM performs well on SOD. Without prompts, it performs poorly; MDSAM improves on SAM’s segmentation and learns SOD semantic information.

A.2 Zero-shot Analysis

  • With ground-truth-derived box prompts, SAM performs significantly well on SOD, but without prompts it performs poorly; MDSAM improves segmentation and learns SOD semantic information.The passage also reports experiments with SAM-HQ when accurate prompts are given.Source: §A.2 · p. 12

A.3 More Ablation Studies

The ablations examine LMSA pooling scales and the addition of local information in fully designed MDSAM at 512 × 512 resolution. They favor retaining multi-scale pooling and report improved performance with local information.

A.3 More Ablation Studies

  • The ablations use fully designed MDSAM with a 512 × 512 input to test pooling scales and local information.Average pooling obtains multi-scale features in LMSA.Source: §A.3 · p. 12

Effects of Scales

  • Multi-scale settings usually outperform single-scale settings and detect salient objects more accurately in complex scenes; differences among multi-scale choices are small.The study therefore favors retaining multi-scale pooling in LMSA.Source: §A.3 · p. 12, Figure 10 · p. 12

Effectiveness of Local Information

  • Introducing local information significantly improves performance, and the same-scale comparison favors MDSAM with local information.The visual comparison attributes more precise features and better saliency maps to local information.Source: §A.3 · p. 12

A.4 Attribute-based Analysis

The attribute-based analysis evaluates MDSAM on nine SOC categories and compares it with 17 methods. It reports superior performance in most scenarios, with average performance in two categories containing fewer images.

A.4 Attribute-based Analysis

  • The challenging SOC test dataset is divided into 9 major attribute categories, including Appearance Change (79 images), Big Object (24 images), Clutter (92 images), Occlusion (157 images), and Shape Complexity (116 images).Source: §A.4 · p. 12
  • MDSAM demonstrates superior performance in most SOC scenarios at 512 × 512 and 384 × 384 resolutions, but performs averagely in AC and BO, which have fewer images.The analysis compares MDSAM with 17 methods on the SOC test set.Source: §A.4 · pp. 12–13, Table 9 · p. 13
  • Visual comparisons show MDSAM accurately locates multi-scale objects with precise edges and details, while current methods struggle with large and small objects.Figure 11 compares saliency maps at 512 × 512 for MDSAM and at 384 × 384 for MDSAM*.Source: §A.4 · p. 12, Figure 11 · p. 13

A.5 More Comparison Results

This section adds precision-recall and F-measure curve comparisons, plus visual comparisons, on five SOD datasets and three COD datasets. The supplied passages describe the comparisons but do not report their outcomes.

A.5 More Comparison Results

Related papers

Loading 2408.04326v1…