SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More

• SAM may fail on challenging low-level structural tasks such as camouflaged object and shadow detection because general training data cannot cover every scenario.Source: §1 · p. 2
• SAM-Adapter adapts SAM to downstream tasks by injecting task-specific guidance through a lightweight additional network and visual prompts while keeping the large model’s knowledge.Source: §1 · p. 2
• The approach is designed to generalize across customized datasets and compose multiple explicit conditions for multi-condition control.Source: §1 · p. 2
• Across shadow detection, camouflaged object detection, and polyp segmentation, experiments report enhanced performance, including state-of-the-art results for the tested downstream tasks.Source: §1 · p. 2
• The study reports testing on only a few datasets, while suggesting broader applicability to fields including medical imaging and agriculture.Source: §1 · p. 2

Semantic Segmentation

• Semantic segmentation has advanced through convolutional, encoder-decoder, attention, pyramid, and transformer architectures, while SAM contributes a large ViT-based pretrained model.Source: §2 · p. 3

Adapters

• Adapters provide compact task-specific modifications for pretrained models, but prior work had not applied them to SAM for image segmentation downstream tasks.Source: §2 · p. 3

Camouflaged Object Detection

• Camouflaged object detection identifies objects that blend into surroundings, using task-relevant cues such as texture, brightness, and color.Source: §2 · p. 3
• The paper combines SAM with task-specific prior knowledge to target state-of-the-art camouflaged object detection.Source: §2 · p. 3

Shadow Detection

• Shadow detection uses cues related to chromaticity, intensity, texture, and illumination, which this work combines with large neural network models.Source: §2 · p. 3

3.1 Using SAM as the Backbone

• SAM-Adapter uses SAM’s ViT-H/16 image encoder with pretrained weights frozen, initializes the mask decoder from SAM, tunes that decoder, and supplies no original prompts.The mask decoder combines a modified transformer decoder block with a dynamic mask prediction head.Source: §3.1 · p. 3

3.2 Adapters

• The adapter converts task-specific information into prompts that are attached to SAM transformer layers, using two MLPs and an activation function.This prompting mechanism introduces task-specific knowledge while retaining the foundation model’s general knowledge.Source: §3.2 · pp. 3–4

3.3 Input Task-Specific Information

• Task-specific information F_i can be extracted as texture or frequency features, or specified through handcrafted rules, depending on the downstream task.The guidance can be derived from samples in the task dataset or designed externally.Source: §3.3 · p. 4
• Multiple guidance sources can be composed as F_i, with adjustable weights w_j controlling the strength of each feature type.This composition makes the task-specific input flexible across different segmentation tasks.Source: §3.3 · p. 4

4.1 Tasks and Datasets

• The experiments use COD10K, CHAMELEON, CAMO, ISTD, and kvasir-SEG for camouflage, shadow, and polyp segmentation evaluation.COD10K contains 3,040 training and 2,026 testing samples; ISTD contains 1,330 training and 540 test images, while CAMO, CHAMELEON, and kvasir-SEG follow stated task protocols.Source: §4.1 · p. 4

4.2 Implementation Details

• SAM uses official implementations and varied prompting approaches, while training uses task-specific losses, AdamW, cosine learning-rate decay, and task-dependent epoch counts.The implementation uses ViT-H SAM, balanced BCE for shadow detection, BCE plus IoU losses for camouflage and polyp segmentation, and 20, 90, and 120 training epochs respectively.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 Experimental Result for Camouflaged Object Detection

• +17.9% in Sα is achieved by SAM-Adapter over SAM for camouflaged object detection, while also outperforming the existing state-of-the-art method.SAM performs significantly below existing state-of-the-art methods across all evaluated camouflage metrics, whereas SAM-Adapter identifies concealed objects more successfully.Source: §4.3 · p. 5, Table 1 · p. 5

4.4 Experimental Result for Shadow Detection

• SAM-Adapter significantly improves shadow detection, producing clearer shadow regions and a quantitative performance boost over SAM.SAM struggles to distinguish shadows from background, with missing or mistakenly added regions; the improvement is reported in Table 2.Source: §4.4 · pp. 5–6, Table 2 · p. 5

4.5 Experimental Result for Polyp Segmentation

• SAM-Adapter enhances polyp identification beyond pretrained SAM, which can identify some polyps but produces inaccurate segmentation.The improvement is supported by quantitative results in Table 3 and visualization results in Figure 6.Source: §4.5 · p. 7, Table 3 · p. 7, Figure 6 · p. 8

• SAM-Adapter uses SAM as a backbone and injects customized information through simple, effective adapters for specific tasks.Source: §5 · p. 8
• The method significantly improves SAM on camouflaged object detection and shadow detection while achieving state-of-the-art results.Source: §5 · p. 8
• Polyp segmentation experiments show that SAM-Adapter can also enhance medical image segmentation performance.Source: §5 · p. 8

• Future work will extend SAM-Adapter to increasingly challenging image segmentation tasks.Source: §6 · p. 8
• The authors plan to broaden SAM-Adapter’s application to other fields.Source: §6 · p. 8
• Future versions may incorporate more specialized designs tailored to specific tasks.Source: §6 · p. 8

• SAM fails to perceive animals that are visually hidden in their natural surroundings.Source: Figure 7 · p. 13
• SAM-Adapter significantly elevates object-segmentation performance on the CAMO dataset.Source: Figure 7 · p. 13
• Figure 7 visualizes the camouflaged-image segmentation results for the CAMO dataset.Source: Figure 7 · p. 13