CVP-BAA: Efficiently Adapting Segment Anything Model for Medical Image Segmentation

• SAM's global ViT modeling lacks explicit mechanisms for local structures and fine boundaries, although clinical segmentation requires precise delineation of subtle anatomy and irregular lesions.Source: §1 · p. 2
• CVP transforms CNN-derived local features into prompts prepended to key and value sequences in selected MHSA blocks, guiding global dependency modeling with local priors.Source: §1 · p. 4
• BAA runs parallel to the FFN and amplifies high-frequency edges and textures to enrich representations along tissue boundaries and fine-grained anatomical structures.Source: §1 · p. 5
• Across five benchmarks and three modalities, CVP-BAA achieves state-of-the-art segmentation, outperforming task-specific models and PEFT-based SAM adaptations quantitatively and qualitatively.Source: §1 · p. 5

2.1. Task-Specific Approaches

• CNNs preserve spatial details but constrain long-range modeling; transformers capture global context yet can oversmooth boundaries, while hybrids add engineered fusion and remain task-tuned.Source: §2.1 · pp. 5–6

2.2. Segment Anything Model Adaptations

• SAM adaptations include prompt tuning, adapters, and LoRA; prompts have limited capacity for local detail, while generic adapters lack domain-specific inductive biases.Source: §2.2 · p. 7
• Prior methods often integrate local cues post hoc or through cross-branch fusion rather than inside self-attention, and lightweight modules lack explicit high-frequency boundary enhancement.Source: §2.2 · pp. 7–8

3.1. Overall Framework

• CVP-BAA preserves frozen SAM and combines ViT and CNN feature extraction with convolutional visual prompting and a boundary-aware adapter.The branches align at H/8×W/8; the two modules inject local structural priors into attention and refine boundary details.Source: §3.1 · pp. 8–9

3.2. Convolutional Visual Prompting

• CVP prepends CNN-derived local prompts to key and value projections in selected self-attention blocks, allowing local textures and boundary cues to inform global context modeling.Prompts are generated from CNN features and injected in ViT blocks 3, 6, 9, and 12.Source: §3.1 · p. 9, §3.2 · pp. 11–12

3.3. Boundary-Aware Adapter

• The BAA runs parallel to the FFN, isolates high-frequency features by subtracting average-pooled low-pass outputs, and scales its contribution to the transformer pathway.The scaling factor balances the adapter branch with the primary pathway; ablations report optimal performance at s = 0.3.Source: §3.3 · pp. 12–13

3.4. Loss Function

• Training combines Dice loss and binary cross-entropy to address class imbalance, with λ set to 0.8 in all experiments.Dice loss evaluates overlap between predictions and ground-truth labels.Source: §3.4 · pp. 13–14

4.1. Datasets

• The five-dataset benchmark covers ultrasound, endoscopy, and dermoscopy, with CVC-ClinicDB and ISIC-2018 included to assess cross-domain generalization.The ultrasound datasets include two breast ultrasound benchmarks and one thyroid ultrasound benchmark.Source: §4.1 · pp. 14–15

4.1.1. BUSI

• BUSI contains 780 pixel-wise annotated breast ultrasound images and uses the SAMUS 7:1:2 training, validation, and testing split.Images average 500×500 pixels and were acquired using LOGIQ E9 and LOGIQ E9 Agile scanners.Source: §4.1.1 · p. 15

4.1.2. UDIAT

• UDIAT comprises 163 pixel-wise annotated ultrasound images, split into 115 training, 16 validation, and 32 testing images.Images are from female patients and have a mean resolution of 760×570 pixels.Source: §4.1.2 · p. 15

4.1.3. TN3K

• TN3K contains 3,493 thyroid ultrasound images, divided into 2,303 training, 576 validation, and 614 testing samples.The images were collected across three ultrasound systems; the split follows SAMUS and TRFE+ protocols.Source: §4.1.3 · p. 15

4.1.4. CVC-ClinicDB

• CVC-ClinicDB contains 612 colonoscopy images, partitioned into 490 training, 61 validation, and 61 testing images.It was originally used for the MICCAI 2015 Sub-Challenge on Automatic Polyp Detection.Source: §4.1.4 · pp. 15–16

4.1.5. ISIC-2018

• ISIC-2018 contains 2,594 clinical dermoscopy images, split into 1,868 training, 467 validation, and 259 testing images.The images have varying resolutions and are used for skin lesion segmentation benchmarking.Source: §4.1.5 · p. 16

• Six segmentation metrics are used: Dice, HD, mIoU, Acc, Prec, and Rec, with dataset-specific reporting and efficiency measures.BUSI, UDIAT, and TN3K report Dice and HD; ISIC-2018 and CVC-ClinicDB report Dice, mIoU, Acc, Prec, and Rec. Efficiency is measured by #Params, FLOPs, and FPS, with results averaged over three runs.Source: §4.2 · p. 16

• The ViT-B SAM image encoder, prompt encoder, and mask decoder remain frozen; only the parallel CNN branch, CVP, and BAA are optimized.Experiments use an NVIDIA GeForce RTX 3090 GPU with 24 GB memory, 200 epochs, Adam, an initial learning rate of 1e −4, and batch size 8.Source: §4.3 · p. 16
• Training uses a Dice-plus-BCE hybrid loss, single-point prompts, and inputs resized to 256×256 pixels.The pretrained SAM backbone was trained on SA-1B.Source: §4.3 · pp. 16–17

5.1.1. Comparison on BUSI, UDIAT, and TN3K

• 86.71%, 89.72%, and 85.23% Dice on BUSI, UDIAT, and TN3K, respectively, with the lowest HD values of 24.52 mm, 13.17 mm, and 27.77 mm.The reported comparisons place CVP-BAA ahead of task-specific models and SAM adaptations across the three ultrasound benchmarks.Source: §5.1.1 · pp. 17–18, Table 1 · p. 18
• On BUSI, UDIAT, and TN3K, CVP-BAA produces clearer boundaries and fewer segmentation errors than task-specific models and existing SAM adaptations.The comparison highlights blurred edges and over- or under-segmentation in competing methods on challenging ultrasound cases.Source: Figure 5 · p. 19, §5.1.1 · p. 19

5.1.2. Comparison on CVC-ClinicDB

• CVP-BAA achieves 94.11% Dice, 89.36% IoU, 99.16% Accuracy, 94.57% Precision, and 94.21% Recall on CVC-ClinicDB.It exceeds SAMUS by +0.83% Dice, +1.27% mIoU, +0.15% Accuracy, +1.46% Precision, and +0.02% Recall.Source: Table 2 · p. 20, §5.1.2 · p. 20
• On CVC-ClinicDB, CVP-BAA’s qualitative segmentations closely match the ground truth.The visual comparison is presented as supporting the quantitative results.Source: Figure 6 · p. 21, §5.1.2 · p. 22

5.1.3. Comparison on ISIC-2018

• 90.40% Dice, 83.75% IoU, 96.27% Accuracy, 92.44% Precision, and 91.66% Recall establish CVP-BAA’s state-of-the-art result on ISIC-2018.The paper describes this dermoscopy benchmark as evidence of cross-modality generalization.Source: Table 3 · p. 22, §5.1.3 · pp. 22–23
• On ISIC-2018, CVP-BAA’s lesion regions and boundaries more closely align with the ground truth than the compared methods.The qualitative comparison supports the reported dermoscopy metrics.Source: Figure 7 · p. 23, §5.1.3 · p. 23

5.2.1. Effectiveness of CVP

• +2.12% Dice and -1.97 mm HD on BUSI, and +2.30% Dice and -1.48 mm HD on TN3K, result from adding CVP alone to the baseline.The paper attributes these gains to injecting local fine-grained details to guide global dependency modeling.Source: §5.2.1 · p. 24

5.2.2. Effectiveness of BAA

• +3.58% Dice and -2.83 mm HD on BUSI, and +3.09% Dice and -1.97 mm HD on TN3K, result from adding BAA alone.The paper links BAA to stronger high-frequency boundary delineation and fewer boundary errors.Source: §5.2.2 · p. 24

5.2.3. Effectiveness of Combinations

• 86.71% Dice and 24.52 mm HD on BUSI, and 85.23% Dice and 27.77 mm HD on TN3K, are achieved when CVP and BAA are combined.The combined configuration achieves the best performance across both datasets.Source: §5.2.3 · p. 24, Table 4 · p. 24

5.2.4. Effectiveness of Prompt Injection Schemes

• Sequence prepending outperforms element-wise addition by 0.78% Dice and -1.05 mm HD on BUSI, and 0.49% Dice and -0.38 mm HD on TN3K.The adopted scheme prepends prompts to the key and value sequences.Source: §5.2.4 · p. 25, Table 5 · p. 25

5.2.5. Effectiveness of Prompt Injection Positions

• Injecting CVP at the 3rd, 6th, 9th, and 12th ViT blocks yields the highest Dice scores and lowest HD values on both BUSI and TN3K.The ablation reports that expanding injection across more blocks progressively integrates local priors through the frozen SAM encoder.Source: §5.2.5 · p. 25, Table 6 · p. 26

5.2.6. Effectiveness of Scaling Factor s

• s = 0.3 achieves the best Dice and HD metrics on both datasets; smaller values under-inject task-specific information, while larger values perturb the pretrained backbone and degrade generalizability.The paper fixes s = 0.3 for all experiments as a balance between the adapter branch and transformer pathway.Source: §5.2.6 · p. 26, Table 7 · p. 26

5.2.7. Visualization of Component Contributions

• Grad-CAM visualizations show CVP emphasizing local fine-grained structures, BAA activating high-frequency boundaries, and the full model producing the most boundary-aligned attention maps.The baseline attention is dispersed and often focuses on irrelevant background regions.Source: Figure 8 · p. 27, §5.2.6 · p. 27, §5.2.7 · pp. 27–28

5.3. Efficiency Analysis

• 30.20M trainable parameters versus SAMUS’s 39.65M is a ∼23.8% reduction; CVP-BAA uses 143.46 GFLOPs and runs at 37.92 FPS.It has more trainable parameters than SAMed’s 18.81M, while the paper reports a favorable efficiency-capacity trade-off and substantially better segmentation accuracy than SAMed.Source: §5.3 · p. 28

6. Conclusion

• CVP injects local structural priors into global dependency modeling, while BAA enhances representations of fine-grained structures and ambiguous lesion boundaries.The conclusion presents CVP-BAA as an efficient SAM adaptation framework for medical image segmentation.Source: §6 · p. 28
• CVP-BAA consistently outperforms dedicated task-specific segmentation models and existing SAM-based adaptation methods across multiple public datasets and imaging modalities.The authors describe these results as validating the combination of explicit local priors with boundary-aware structural refinement.Source: §6 · p. 29
• Generalization to completely unseen clinical domains and data-scarce scenarios remains an open challenge for the current design.The authors identify more parameter-efficient mechanisms and few-shot learning as future directions.Source: §6 · p. 29

Data Availability Statement

• All data supporting the study's findings are included within the article.Source: Data Availability Statement · p. 29