PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model

Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, Chun Yuan

arXiv:2401.17868v1cs.CVcs.LG

TL;DR

SAM’s generic zero-shot segmentation performance does not transfer reliably to real-world and specialized domains, while its encoder and pretraining leave limitations for dense prediction and high-level semantics. Conv-LoRA adds lightweight multi-scale convolutional experts to LoRA for parameter-efficient SAM adaptation, and the paper reports that it consistently outperforms other PEFT techniques across diverse downstream tasks. The authors position it as a general-purpose approach, not a guarantee of outperforming domain-specific state-of-the-art models.

  • Problem

    SAM performs well on generic zero-shot segmentation but underperforms on real-world domain tasks, while its plain ViT lacks dense-prediction inductive biases and foreground-background pretraining hinders high-level semantics.

    Source: §1 · p. 1

  • Method

    Conv-LoRA adds lightweight convolutional operations and multi-scale convolutional experts within LoRA to inject local priors into SAM’s image encoder.

    Source: §1 · p. 2

  • Results

    Conv-LoRA consistently exhibits superior performance over other PEFT techniques across downstream tasks in natural images, agriculture, remote sensing, and healthcare.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper presents Conv-LoRA as a general-purpose PEFT method for adapting SAM across multiple domains, rather than directly competing with domain-specific models.

    Source: §5 · p. 9

    Conv-LoRA may not consistently outperform domain-specific state-of-the-art models, since the work focuses on general PEFT rather than direct competition with them.

    Source: §5 · p. 9

Abstract

from arXiv · show

The Segment Anything Model (SAM) stands as a foundational framework for image segmentation. While it exhibits remarkable zero-shot generalization in typical scenarios, its advantage diminishes when applied to specialized domains like medical imagery and remote sensing. To address this limitation, this paper introduces Conv-LoRA, a simple yet effective parameter-efficient fine-tuning approach. By integrating ultra-lightweight convolutional parameters into Low-Rank Adaptation (LoRA), Conv-LoRA can inject image-related inductive biases into the plain ViT encoder, further reinforcing SAM's local prior assumption. Notably, Conv-LoRA not only preserves SAM's extensive segmentation knowledge but also revives its capacity of learning high-level image semantics, which is constrained by SAM's foreground-background segmentation pretraining. Comprehensive experimentation across diverse benchmarks spanning multiple domains underscores Conv-LoRA's superiority in adapting SAM to real-world semantic segmentation tasks.

1 INTRODUCTION

SAM performs well on generic zero-shot segmentation but struggles on real-world and specialized-domain tasks, while its plain ViT encoder and foreground-background pretraining leave gaps for dense predictions and high-level semantics. The introduction presents Conv-LoRA as a parameter-efficient adaptation that adds convolutional local priors to LoRA and reports stronger performance than other PEFT methods across diverse downstream benchmarks.

1 INTRODUCTION

  • SAM’s generic zero-shot performance does not extend reliably to real-world domains, and its foreground-background pretraining hinders high-level semantics needed for multi-class segmentation.The paper also identifies the plain ViT encoder’s lack of vision-specific inductive biases useful for dense prediction.Source: §1 · p. 1
  • In the road-segmentation example, Conv-LoRA separates roads from adjacent buildings where LoRA and VPT struggle; in transparent-object segmentation, it yields cleaner masks than LoRA.Both LoRA and Conv-LoRA correct VPT’s bowl-to-jar/kettle misclassification.Source: Figure 1 · p. 2
  • Conv-LoRA adds lightweight convolution layers within LoRA’s bottleneck to inject image-related local priors into SAM’s plain ViT encoder.LoRA’s trainable projections can help recover high-level semantic information, and the paper reports that LoRA surpasses VPT particularly on multi-class semantic segmentation.Source: §1 · p. 2
  • Multiple convolutional experts specialize in distinct feature scales, and MoE dynamically selects a scale to inject vision-specific inductive biases.Each expert resizes features before convolution and restores them to ViT’s default scale.Source: §1 · p. 2
  • Removing the prompt encoder and adding lightweight mask-decoder MLPs turns SAM into an end-to-end model for binary and multi-class segmentation.The prompt encoder is removed and lightweight MLPs are added in the mask decoder.Source: §1 · p. 2
  • Conv-LoRA consistently outperforms other PEFT techniques across downstream tasks in benchmarks spanning natural images, agriculture, remote sensing, and healthcare.The introduction describes the benchmark as extensive and cross-domain.Source: §1 · p. 3

2 RELATED WORK

The related-work section situates Conv-LoRA within parameter-efficient tuning, computer-vision segmentation, and prior SAM fine-tuning approaches. It distinguishes the method by targeting multi-scale local priors and SAM’s encoder limitations while retaining a parameter-efficient design.

2 RELATED WORK

  • PEFT adapts foundation models by tuning a small parameter subset while keeping most parameters fixed, through approaches including adapters, selective tuning, prompts, and LoRA.LoRA uses trainable low-rank matrices in transformer layers.Source: §2 · p. 3
  • Unlike Convpass’s convolutional bottleneck for image classification, this work targets SAM semantic segmentation and enforces multi-scale local priors beyond the default scale..Source: §2 · p. 3
  • Prior SAM fine-tuning work tunes the mask decoder or applies parameter-efficient methods to the image encoder; Conv-LoRA adds convolution to address encoder inductive-bias limits.The paper also presents SAM pretraining as hampering high-level semantic learning and makes minor architectural adjustments for end-to-end segmentation.Source: §2 · p. 3
  • Conv-LoRA uses MoE to dynamically inject local priors at different feature scales, with experts that differ in their scaling operations.The paper uses MoE for downstream parameter-efficient tuning, rather than to expand model capacity as in the original MoE goal.Source: §2 · pp. 3–4

3 METHOD

Conv-LoRA adds lightweight convolutions within LoRA to inject local image priors at selected feature scales, while retaining parameter-efficient fine-tuning. For multi-class segmentation, SAM uses a classification branch in its mask decoder alongside its existing mask prediction.

3.1 CONV-LORA

  • Conv-LoRA places convolution between LoRA’s encoder and decoder, injecting a local image prior while the low-rank constraint keeps the convolutions lightweight.LoRA freezes pretrained weights and adds trainable rank-decomposition matrices; Conv-LoRA inserts convolution operations into that bottleneck.Source: §3.1 · p. 4
  • A gating module dynamically selects scale-specialized convolution experts, which reconstruct feature maps at their assigned scale and return them to ViT’s default scale.Each expert applies convolution at its selected feature scale; the gating network activates only top-k experts, defaulting to 1.Source: §3.1 · p. 4, Figure 3 · p. 5
  • MoE selectively activates sparse experts to reduce computational overhead compared with applying local priors through concurrent multi-scale branches.The paper favors MoE for efficient fine-tuning.Source: §3.1 · p. 5

3.2 END-TO-END MULTI-CLASS SEGMENTATION WITH SAM

  • For multi-class segmentation, SAM freezes the prompt encoder with constant prompt tokens, adds a classification branch to the mask decoder, and fully fine-tunes that lightweight decoder.The added branch predicts classification scores; the original decoder predicts binary foreground-background masks.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments evaluate Conv-LoRA across segmentation tasks and domains, comparing it with parameter-efficient and other methods. Results examine binary- and multi-class segmentation, SAM’s local attention, and dynamic selection of feature scales.

4 EXPERIMENTS

  • Experiments cover natural, medical, agricultural, and remote-sensing datasets, with PEFT methods run three times and averages and standard errors reported.The default setup uses 30 epochs, batch size 4, Adam, and a learning rate of 1×10−4, with 3×10−4 used for agriculture and remote sensing.Source: §4 · pp. 5–6

4.1 BINARY-CLASS SEMANTIC SEGMENTATION

  • Conv-LoRA surpasses other PEFT techniques across diverse datasets, while all PEFT methods outperform decoder-only fine-tuning.The reported binary-segmentation comparison also notes a clear performance boost over LoRA with negligible parameter overhead.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 MULTI-CLASS SEMANTIC SEGMENTATION

  • PEFT fine-tuning of SAM’s image encoder significantly boosts mIoU, while decoder-only approaches show a substantial mIoU gap against domain-specific methods.The authors interpret the mIoU boost as a restoration of high-level semantic learning; they report SAM’s ImageNet-1K accuracy as 54.2%, versus 67.7% for MAE.Source: §4.2 · pp. 7–8, Table 2 · p. 7

4.3 ABLATION STUDY

  • SAM has short mean attention distances in many deep-block heads, unlike MAE’s consistently long distances in later layers.The paper interprets this contrast as segmentation pretraining shifting attention toward local information.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Dynamic MoE outperforms simultaneous multi-scale feature addition, with a 1.54x speedup and 1.7GB less training memory.The authors attribute the performance difference to input-dependent scale preferences: selecting top-1 experts prioritizes crucial feature maps rather than diluting them across scales.Source: §4.3 · pp. 8–9, Table 3 · p. 9
  • The best tested feature-map scaling ratio differs by dataset: 4 for Leaf Disease Segmentation and 2 for ISIC 2017.The experiment compares single-expert Conv-LoRA variants using scaling ratios 1, 2, and 4.Source: §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSION

Conv-LoRA uses lightweight convolutions and encoder fine-tuning to reinforce SAM’s local prior and address limits on high-level semantic learning. Its gating network selects feature-scale experts, while a modified decoder supports multi-class segmentation.

5 CONCLUSION

  • Conv-LoRA injects lightweight convolutions to strengthen local-prior exploitation, while fine-tuning relatively few encoder parameters can alleviate the semantic-learning constraint of foreground-background pretraining.The conclusion presents Conv-LoRA as a general parameter-efficient method for downstream segmentation across natural images, agriculture, remote sensing, and healthcare.Source: §5 · p. 9

A GATING NETWORK OF CONV-LORA

  • The gating network keeps only the top k expert scores, setting the others to −∞ so their softmax gates are zero and those experts need not be computed.The scores are derived from pooled and reshaped features using trainable gating weights and a noise term.Source: A GATING NETWORK OF CONV-LORA · p. 16

B MASK DECODER FOR MULTI-CLASS SEGMENTATION

  • SAM’s decoder lacks a classification module, so the multi-class design adds a classification branch to predict classes for its N masks.Training matches mask predictions to ground-truth segments, includes a no-object category for unmatched masks, and uses mask and classification losses.Source: B MASK DECODER FOR MULTI-CLASS… · p. 16

C DATASET DETAILS

The evaluation dataset details cover medical, natural-image, agriculture, remote-sensing, and multi-class transparent-object segmentation, with distinct training and testing setups.

Polyp Segmentation (medical images)

  • Polyp segmentation uses Kvasir and CVC-ClinicDB, with additional CVC-ColonDB, EndoScene, and ETIS images for testing.Kvasir contains 1,000 images and CVC-ClinicDB 612; the setup uses a 9:1 training-test split and reserves 20% of training images for validation.Source: C DATASET DETAILS · pp. 16–17

Skin Lesion Segmentation (medical images)

  • Skin-lesion segmentation uses ISIC 2017, with 2,000 training, 150 validation, and 600 testing images.The task is described as challenging because of ambiguous boundaries and color variation.Source: C DATASET DETAILS · p. 17

Camouflaged Object Segmentation (natural images)

  • Camouflaged-object segmentation trains on COD10K and CAMO and tests on COD10K, CHAMELEON, and CAMO.The training set combines COD10K and CAMO images, with 10% held out for validation.Source: C DATASET DETAILS · p. 17

Shadow, Leaf, and Road Segmentation

  • The remaining binary tasks use SBU for shadow detection, a leaf-disease dataset for leaf segmentation, and Massachusetts Roads for road segmentation.Their stated training-set sizes are 4,085, 498, and 1,107 images, respectively.Source: C DATASET DETAILS · p. 17

Multiclass Semantic Segmentation (natural images)

  • Trans10K-v1 and v2 provide transparent-object benchmarks, with v2 adding 11 fine-grained object categories.Both use 5,000 training, 1,000 validation, and 4,428 testing images; the text describes v2 as more challenging due to its fine-grained labels and diversity.Source: C DATASET DETAILS · p. 17

D FULL EXPERIMENT RESULTS

The full binary-segmentation experiments report that Conv-LoRA is superior to other PEFT methods across diverse datasets and domains; the supplied passages do not provide cell values for individual comparisons.

D FULL EXPERIMENT RESULTS

  • Conv-LoRA demonstrates superiority over other PEFT methods across diverse binary-segmentation datasets from various domains.The authors state that they reran experiments using provided model code, except for the Kaggle-sourced leaf dataset, for which they used their own experiments.Source: D FULL EXPERIMENT RESULTS · pp. 17–18
  • The listed binary-result tables cover polyp, skin-lesion, camouflaged-object, and shadow segmentation.The supplied table captions identify these task scopes but do not state individual results.Source: Table 5 · p. 18, Table 6 · p. 18, Table 7 · p. 18
  • Additional binary-result tables cover leaf and road segmentation.The supplied captions identify the tasks but provide no comparison values.Source: Table 9 · p. 19, Table 10 · p. 19

E ADDITIONAL ABLATION

The additional ablations examine LoRA rank, combining Conv-LoRA with VPT, multi-scale convolution design, and computational cost.

The Rank of LoRA

  • Increasing LoRA rank can improve SAM’s twelve-class segmentation performance, but the added parameters may conflict with PEFT’s parameter-efficiency aim.The paper identifies a more efficient way to introduce classification prior as a future research direction.Source: E ADDITIONAL ABLATION · p. 19, Table 11 · p. 20

Combined with Other PEFT Methods

  • Combining Conv-LoRA with VPT achieves superior or competitive polyp-segmentation performance, indicating potential for combining it with other PEFT methods.The authors suggest such combinations may help reduce trainable parameters while maintaining enhanced performance.Source: E ADDITIONAL ABLATION · p. 20, Table 12 · p. 20

MoE-Conv vs. Blocks with Various Kernel sizes

  • MoE-Conv outperforms multi-kernel Inception-style convolution blocks on the Leaf and ISIC 2017 datasets.The authors attribute the difference to multi-kernel blocks operating only at the default feature scale, whereas larger feature scales are generally more useful for injecting priors.Source: Table 13 · p. 20, E ADDITIONAL ABLATION · p. 21

Computational Cost

  • Conv-LoRA is slower than other methods, with upscale and downscale operations accounting for the main computational cost.The paper identifies avoiding explicit feature scaling and token merging as possible directions for acceleration.Source: E ADDITIONAL ABLATION · p. 21, Table 14 · p. 21

F LOCAL PRIOR ANALYSIS

The analysis measures feature inductive biases through attention distance and Fourier-based relative log amplitudes. Across randomly initialized, MAE-pretrained, and SAM ViTs, local-information indicators strengthen with pretraining, with SAM showing the strongest biases.

F LOCAL PRIOR ANALYSIS

  • The analysis averages attention distance across heads and 500 sampled images, and Fourier relative log amplitudes across layers and 100 sampled images.Mean attention distance reflects local or global aggregation; relative log amplitudes reflect high-frequency signal reduction or amplification.Source: F LOCAL PRIOR ANALYSIS · p. 21
  • SAM's large-scale segmentation pretraining further amplifies inductive biases already associated with large-scale ViT pretraining.The paper frames this analysis as investigating biases in features that Conv-LoRA is designed to reinforce.Source: F LOCAL PRIOR ANALYSIS · p. 21
  • Mean attention distance decreases and high-frequency feature signals increase from randomly initialized ViT to MAE ViT to SAM ViT.The authors interpret both trends as indicating greater focus on local information.Source: F LOCAL PRIOR ANALYSIS · p. 21

G MOE ANALYSIS

Conv-LoRA's MoE selects among convolutional experts to inject local priors at feature scales suited to input features. Expert-selection patterns differ across datasets, consistent with adaptive selection for sample properties.

G MOE ANALYSIS

  • MoE dynamically selects a feature scale for injecting local priors based on input features.Conv-LoRA uses multiple parallel convolutional experts, each specializing in a distinct feature scale.Source: G MOE ANALYSIS · p. 21
  • SAM's features show stronger inductive biases than randomly initialized ViT and MAE features, including greater focus on local information.Figure 7 characterizes this through mean attention distance and Fourier-transformed feature maps.Source: Figure 7 · p. 22
  • Leaf Segmentation favors experts with upsampling ratios 3 and 4, while ISIC 2017 favors ratio 2.The authors connect these dataset-level selection patterns with optimal-scale ablation results and interpret MoE as adapting to each sample's properties.Source: G MOE ANALYSIS · p. 22, Figure 8 · p. 22

H LOW-DATA REGIME EXPERIMENTS

The few-shot experiments test Conv-LoRA's data efficiency on multi-class transparent-object segmentation. The reported improvement over LoRA is especially pronounced at 1-shot.

H LOW-DATA REGIME EXPERIMENTS

  • The low-data experiments are motivated by the possibility that convolutional layers' inductive biases enhance data efficiency during fine-tuning.Source: H LOW-DATA REGIME EXPERIMENTS · p. 23
  • The few-shot study uses Trans10K-v2 for 12-class transparent-object segmentation, with 1, 2, 4, 8, or 16 labeled examples per class.Experiments run for 100 epochs; the table caption defines a shot as the number of labeled training examples per class.Source: H LOW-DATA REGIME EXPERIMENTS · p. 23, Table 15 · p. 23
  • Conv-LoRA's improvement over LoRA is particularly pronounced in the 1-shot setting.The authors interpret the result as evidence that introducing inductive biases contributes to improved data efficiency.Source: H LOW-DATA REGIME EXPERIMENTS · p. 23, Table 15 · p. 23

I MORE VISUALIZATION RESULTS

The visualizations compare Conv-LoRA and LoRA feature maps and show segmentation outputs across multiple datasets. Conv-LoRA features are described as carrying finer details, including slim edges.

I MORE VISUALIZATION RESULTS

  • Conv-LoRA feature maps provide more fine-grained information, such as slim edges, than LoRA feature maps.The authors describe these details as beneficial to later mask prediction and connect them to reinforcing image-related local priors.Source: I MORE VISUALIZATION RESULTS · p. 23, Figure 9 · p. 23
  • Mask-prediction visualizations compare VPT, LoRA, and Conv-LoRA across datasets, including skin lesions, camouflaged objects, leaf disease, remote sensing, and transparent objects.The supplied figure captions identify the task domains but do not state specific comparative outcomes.Source: I MORE VISUALIZATION RESULTS · p. 23, Figure 10 · p. 24, Figure 11 · p. 24

Related papers

Loading 2401.17868v1…