Source-linked AI summary
Spectral Adapters for Segment Anything Model-based Segmentation of Colorectal Liver Metastases in Computed Tomography
Ramtin Mojtahedi, Mohammad Hamghalam, Jacob J. Peoples, Natalie Gangai, Mithat Gonen, Yun Shin Chun, HyunSeon Christine Kang, Richard K. G. Do, Amber L. Simpson
TL;DR
CRLM segmentation in contrast-enhanced CT is important for clinical assessment, but efficient and accurate adaptation of SAM remains challenging. The paper introduces DiSECT and SiGA, spectral adapters that update frozen SAM weights in leading spectral directions, with SiGA adding global and input-conditioned gating. SiGA achieves the best reported accuracy and reaches 0.76 test DSC under no-prompt inference, comparable to the 0.758 DSC of 3D nnU-Net.
Problem
Accurate CRLM segmentation in contrast-enhanced CT supports clinical assessment, while adapting large SAM backbones is constrained by computational cost and limited annotated data.
Method
The paper introduces DiSECT and SiGA, spectral adapters that constrain residual updates within singular-vector subspaces of frozen SAM weights, with SiGA adding global and input-conditioned gating.
Results
0.76 test DSC under no-prompt inference makes SiGA competitive with the fully trained 3D nnU-Net baseline at 0.758 DSC.
Takeaways & Limitations
Spectral adapters support accurate, parameter-efficient SAM adaptation for CRLM segmentation while preserving promptable flexibility.
Takeaways & Limitations
The study uses 2D slices from 3D volumes, focuses on portal-venous-phase CRLM, and leaves cross-site, multiphase, protocol, and broader architecture robustness for future work.
Abstract
from arXiv · showhide
Accurate segmentation of colorectal liver metastases (CRLM) in contrast-enhanced computed tomography (CT) is important for response assessment, surgical planning, and follow-up. We propose two parameter-efficient spectral adapters for the Segment Anything Model (SAM): the Directional Spectral Adapter (DiSECT) and Spectral Instance-Guided Adapter (SiGA). DiSECT uses singular value decomposition of frozen weights to constrain residual updates to leading spectral directions, while SiGA adds global and input-conditioned gating through a multilayer perceptron. We evaluate these methods on 446 contrast-enhanced CT volumes (355 training, 91 testing) and compare them with LoRA, QLoRA, convolutional adapters (CAD), and a 3D nnU-Net baseline. Experiments consider single-point, three-point, bounding-box, and no-prompt regimes. SiGA achieves the best single-point performance with a Dice score of 0.77, IoU of 0.69, and HD95 of 35.39 mm. Under no-prompt inference, SiGA reaches 0.76 Dice, 0.68 IoU, and 46.76 mm HD95, comparable to the nnU-Net baseline (0.758 Dice). DiSECT uses only 0.14 million trainable parameters. These results show that spectral adapters can efficiently adapt SAM for CRLM segmentation while retaining strong accuracy with limited trainable parameters.
1 Introduction
CRLM segmentation in contrast-enhanced CT is clinically valuable but difficult to perform efficiently and consistently. The paper introduces spectral adapters for SAM and evaluates them against established parameter-efficient methods and a 3D nnU-Net baseline.
- 1 Introduction: Accurate CRLM segmentation supports tumor-burden assessment, radiomics, and treatment-response evaluation, but manual delineation is time-consuming and variable across observers.Contrast-enhanced CT is the primary modality for colorectal liver metastasis detection, staging, and monitoring.
- 1 Introduction: Full fine-tuning of large medical foundation-model backbones is computationally expensive when GPU resources and annotated data are constrained.Parameter-efficient fine-tuning keeps most pretrained backbone weights frozen while training only a small fraction of parameters.
- 1 Introduction: Existing SAM adaptation strategies include zero-shot use, full fine-tuning, prompt tuning, decoder tuning, and lightweight adapters, but naive choices can underperform on subtle heterogeneous 3D CT lesions.The central challenge is balancing segmentation accuracy and efficiency for medical imaging.
- 1 Introduction: SiGA adds global and input-conditioned gating to spectral adaptation, while DiSECT constrains residual updates to leading spectral directions of frozen transformer weights.Both methods operate within the singular-vector subspace of pretrained weights; SiGA additionally routes spectral directions instance-wise.
- 1 Introduction: The study benchmarks DiSECT and SiGA against LoRA, QLoRA, and CAD across prompting strategies and compares them with a fully trained 3D nnU-Net.The evaluation examines both segmentation performance and computational efficiency for resource-constrained deployment.
2 Methods
The method adds lightweight spectral adapters to SAM while freezing backbone weights, using retained singular directions for parameter-efficient residual updates. DiSECT applies trainable spectral gating, whereas SiGA further uses input-conditioned gating to route spectral directions by instance.
- Spectral Adapters: DiSECT and SiGA operate within the leading singular-vector subspace of frozen transformer weights, adding parameter-efficient residual branches to SAM.The image encoder and mask decoder are adapterized while the backbone remains frozen.
- DiSECT: DiSECT uses a trainable gate vector to selectively activate retained spectral directions in its residual update.The gate is element-wise applied within the rank-r spectral subspace, and sparsity can improve parameter efficiency and stabilize adaptation.
- SiGA: SiGA extends DiSECT with global and input-conditioned gates predicted through a lightweight multilayer perceptron for instance-aware spectral routing.The mechanism can emphasize or suppress spectral components according to tumor appearance, size, and contrast.
- Low-Rank Adapters: LoRA adds a low-rank residual through trainable projection matrices, while QLoRA uses the same formulation with a 4-bit quantized frozen backbone.The adapter rank controls the width of the compact branch for these baselines.
- Convolutional Adapter: CAD combines compact learned projections with channel-wise multi-scale depthwise dilated convolutions to capture local spatial structure.For convolutional targets, the projections are implemented as 1 × 1 convolutions; transformer targets use linear projections.
2.3 Prompting Strategies
The study evaluates single-point, three-point, bounding-box, and no-prompt regimes, deriving prompts from ground-truth masks when applicable. Single-point prompting is used for training and model selection, while held-out testing uses no prompts.
- Prompting Strategies: The evaluation includes single-point, three-point, bounding-box, and no-prompt inference regimes.Bounding boxes target IoU values of 0.50 or 0.75, while no-prompt inference uses SAM’s empty-prompt embedding.
- Prompting Strategies: Single-point prompting gave the highest validation performance with minimal annotation and was selected for adapter training and model selection.The held-out test analysis is instead reported under no-prompt inference.
2.4 Reference Fully Trained Baseline
The reference baseline is a fully trained 3D convolutional model based on nnU-Net, implemented with residual encoder connections. It processes volumetric patches and produces ensemble-thresholded tumor masks on the held-out test set.
- Reference Fully Trained Baseline: The baseline processes 96 × 256 × 256 voxel patches using 3 × 3 × 3 convolutions, instance normalization, LeakyReLU activations, and strided downsampling.The downsampling convolutions use a 2 × 2 × 2 stride.
- Reference Fully Trained Baseline: Final predictions average probabilistic outputs from an ensemble of five models and threshold voxel-wise probabilities at 0.5.Evaluation uses the same 91 held-out test cases as the SAM-based models.
2.5 Dataset and Preprocessing
The dataset comprises 446 multi-institutional portal venous contrast-enhanced CT volumes from patients with CRLM, split into 355 training and 91 testing cases for SAM experiments. Preprocessing and evaluation use liver-specific CT handling and overlap and boundary metrics.
- Dataset: The cohort contains 446 portal venous phase contrast-enhanced CT volumes from Memorial Sloan Kettering and MD Anderson.The data include public TCIA cases and retrospective or prospective cohorts spanning cancer stages and imaging conditions.
- Dataset: SAM-based experiments use 355 training volumes and 91 held-out test volumes with expert-verified tumor masks.Mean voxel spacing is (0.822, 0.822, 4.164) mm.
- Preprocessing and Training: The SAM adapters are initialized from public Medical Adapter Zoo weights and trained with frozen pretrained weights, standard augmentations, and weighted binary cross-entropy.Experiments use SAM ViT-B on one NVIDIA A100 GPU with up to 20 AdamW epochs.
- Evaluation: Segmentation quality is reported with DSC and IoU for overlap and HD95 for boundary accuracy.Training metrics are generally reported for single-point prompting, while held-out test metrics use no-prompt inference.
3 Results
SiGA delivers the strongest segmentation results across the reported adapter comparisons, while DiSECT minimizes trainable parameters at the cost of overlap and boundary accuracy. Against the fully trained 3D nnU-Net baseline, SiGA reaches a similar tumor Dice score under no-prompt inference.
- 3.1 Training Performance and Compute Trade-Offs: 0.77 DSC, 0.69 IoU, and 35.39 mm HD95 make SiGA the best single-point method.CAD has comparable overlap but higher computational cost, while LoRA and QLoRA achieve lower DSC values.
- 3.2 Test Performance Under No-Prompt Setting: SiGA achieves the highest overlap performance under no-prompt inference on the held-out 91-case test cohort.CAD and LoRA are competitive in overlap, whereas DiSECT has the lowest overlap and largest boundary error.
- 3.3 Contextual Comparison with nnU-Net: 0.76 DSC for SiGA approaches the 0.758 DSC achieved by the fully trained 3D nnU-Net baseline.The comparison uses tumor DSC despite differing 2D slice-wise and 3D volume-based evaluation settings.
4 Discussion
SiGA provides the strongest segmentation accuracy among evaluated adapters, while its efficiency trade-offs differ from LoRA, QLoRA, CAD, and DiSECT. The study is limited by 2D slice processing, portal-venous-phase data, and unexplored adapter configurations.
- SiGA consistently achieves the highest segmentation accuracy, reaching 0.76 DSC on the held-out no-prompt test set.Its spectral dual-gating mechanism combines global spectral importance with instance-specific modulation.
- LoRA and QLoRA offer efficiency advantages, with QLoRA providing the fastest throughput and memory savings but lower overlap accuracy.
- Single-point prompting provides strong tumor-focused supervision, whereas no-prompt testing better reflects high-throughput clinical workflows.
- The evaluation is constrained to 2D slices, portal-venous-phase CRLM data, and non-exhaustive exploration of ranks, spectral dimensions, and gating capacity.
- Future work includes 3D or hybrid architectures, multi-phase or multimodal inputs, and federated or edge deployment.
5 Conclusion
The study evaluates spectral and low-rank parameter-efficient strategies for adapting a frozen SAM backbone to CRLM segmentation in contrast-enhanced CT. SiGA achieves the best accuracy, including 0.76 test DSC under no-prompt inference, while LoRA and QLoRA provide alternatives when throughput or memory matters.
- The evaluation integrates LoRA, QLoRA, CAD, DiSECT, and SiGA into a frozen SAM backbone for CRLM segmentation.
- SiGA reaches 0.76 test DSC under realistic no-prompt inference, comparable to the fully trained 3D nnU-Net baseline at 0.758 DSC.
- SiGA is preferred when accuracy is the priority, while QLoRA and LoRA offer high-throughput alternatives when memory or latency dominate.
CRediT Author Statement
The CRediT statement assigns contributions across conceptualization, methodology, analysis, data curation, funding, supervision, and writing.
- Contributors are assigned roles spanning conceptualization, methodology, analysis, visualization, data curation, funding, resources, supervision, and manuscript preparation.
Funding
The study acknowledges support from an NIH/NCI grant.
- The reported funding source is NIH/NCI grant R01CA233888.
Ethics Approval and Consent
The study used de-identified human CT images and masks under Queen’s University HSREB approval, with consent and waiver requirements governed by the approved protocol.
- The study used de-identified human CT images and masks under Queen’s University HSREB approval for DMED-2441-21.The protocol was associated with TRAQ #6031742.
- Consent and waiver requirements followed the approved research protocol.
- No recruitment, contact, intervention, identifiable-information collection, or animal studies occurred.The approval was renewed on October 30, 2025, and is valid through December 8, 2026.