PaperlayerView exact revision on arXiv

Source-linked AI summary

FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptation

Ziquan Liu, Zhewei Zhu, Xuyang Shi

arXiv:2608.26531v1cs.CV

TL;DR

Direct SAM transfer to medical imaging is hindered by domain gaps, while existing PEFT methods entangle heterogeneous frequency components in shared low-rank spaces. FAN-LoRA separates adaptation into B-spline low-pass structural alignment and discrete Fourier high-pass compensation, and experiments show competitive accuracy and boundary precision across challenging domain shifts.

  • Problem

    Medical domain gaps and frequency entanglement in existing PEFT methods limit reliable SAM adaptation under cross-modality and cross-center shifts.

  • Method

    FAN-LoRA uses separate B-spline low-pass and discrete Fourier high-pass branches for structural alignment and local textural compensation.

  • Results

    FAN-LoRA consistently outperforms state-of-the-art PEFT baselines in average Dice and boundary errors across three challenging benchmarks.

  • Takeaways & Limitations

    Explicit frequency decoupling provides a compact PEFT framework with accuracy and boundary-precision advantages for medical foundation model adaptation.

Abstract

from arXiv · show

The advent of vision foundation models, notably the Segment Anything Model (SAM), has catalyzed significant advancements in natural image segmentation. However, their direct transfer to medical imaging remains severely bottlenecked by profound domain gaps, such as cross-modality and cross-center shifts. Existing Parameter-Efficient Fine-Tuning (PEFT) methods facilitate the adaptation of SAM to medical domains; nevertheless, they frequently suffer from performance degradation under severe distribution shifts. This vulnerability primarily stems from the implicit entanglement of heterogeneous frequency components within a shared low-rank subspace, which directly exacerbates sub-optimal structural alignment and localized boundary blurring. To overcome this representational bottleneck, we propose the Fourier-Adaptive Nonlinear Low-Rank Adaptor (FAN-LoRA), a novel frequency-decoupled fine-tuning architecture. FAN-LoRA explicitly separates the optimization space by employing a B-spline-driven low-pass branch for global structural alignment, synergistically coupled with a discrete Fourier high-pass branch for local textural compensation. Extensive experiments across three challenging cross-modality and cross-center benchmarks demonstrate that FAN-LoRA consistently outperforms state-of-the-art PEFT baselines. Compared to the strongest competitors, our method achieves consistent improvements in average Dice scores and notable reductions in boundary errors, while maintaining a compact module size without compromising computational efficiency.

I. INTRODUCTION

Medical domain gaps limit direct SAM transfer, while existing PEFT methods entangle heterogeneous frequency components. FAN-LoRA addresses this bottleneck with explicitly separated structural and detail adaptations.

  • Direct SAM transfer to medical imaging suffers from domain gaps, causing sub-optimal boundary delineation and degradation in low-contrast regions.
  • Existing PEFT methods entangle low-frequency structural shifts and high-frequency texture variations within one low-rank subspace.This can produce over-smoothed representations or noise overfitting through gradient conflicts.
  • FAN-LoRA reframes adaptation as synergistic optimization of low-frequency structural alignment and high-frequency residual compensation.
  • Its dual-branch design uses a B-spline-driven low-pass branch for global structure and a discrete Fourier branch for local texture.
  • FAN-LoRA demonstrates competitive accuracy and robustness against state-of-the-art PEFT methods across cross-modality and cross-center scenarios.

II. RELATED WORK

Medical segmentation progressed from task-specific supervised models to SAM-based universal segmentation, but domain gaps and computational costs motivate parameter-efficient adaptation. Existing nonlinear adapters improve approximation capacity yet retain unified parameter spaces that do not explicitly disentangle conflicting frequency components.

  • Task-specific architectures such as nnU-Net historically dominated medical image segmentation through empirical pipeline optimization.
  • SAM shifted segmentation toward promptable universal models, but zero-shot application to medical modalities exposed significant domain gaps.
  • MedSAM showed that domain-specific fine-tuning improves segmentation accuracy, while its computational requirements limit flexible continuous adaptation.
  • PEFT adapts frozen large-scale models by updating minimal parameters through lightweight trainable modules, including adapters and LoRA.
  • AuroRA expands low-rank approximation capacity with learnable B-spline bases but remains in a unified parametric space without explicit frequency disentanglement.

C. Frequency-Domain Learning for Domain Adaptation

Frequency-domain methods address medical domain shifts by modeling low-frequency style or structural changes alongside high-frequency details. FAN-LoRA extends this direction by enforcing explicit feature-level separation between spline-based structural alignment and Fourier-based residual compensation.

  • FDA transfers visual styles by swapping low-frequency Fourier amplitude spectra while preserving high-frequency semantic structures.
  • Fourier-aware PEFT methods optimize sparse spectral parameters or use frequency consistency, but do not necessarily separate structural and textural optimization.
  • The architecture projects input features into parallel B-spline ANL and discrete Fourier branches for global structure and local detail, respectively.
  • FAN-LoRA explicitly assigns continuous splines to low-frequency structural alignment and discrete Fourier transforms to high-frequency residual compensation.

A. Overall Architecture

FAN-LoRA adds low- and high-frequency adaptation branches to a frozen base mapping. The low-pass branch uses a low-rank B-spline ANL to model smooth global domain changes while regularizing the bottleneck.

  • The forward pass combines the frozen base linear mapping with low-pass structural adaptation and high-pass residual compensation.
  • The low-pass branch projects features into rank r, applies the ANL, and maps them back through the up-projection matrix B.
  • Dropout is applied before low-rank projection to regularize the bottleneck and mitigate B-spline overfitting to source-domain noise.
  • The ANL combines LayerNorm, nonlinear transformation, and a B-spline residual mapping, using spline order p = 3 and grid size G = 5.
  • B-spline basis functions provide smooth low-pass behavior that suppresses high-frequency oscillations and noise while preserving low-frequency structure.

C. High-Pass Compensation via Lightweight FourierFT

FAN-LoRA uses a lightweight high-pass Fourier branch to compensate for texture and boundary information lost by low-pass structural alignment. Fixed sinusoidal bases project features into a high-frequency subspace, where learnable spectral modulation reconstructs residuals efficiently.

  • The high-pass branch compensates for minute textures and sharp lesion boundaries smoothed by low-pass filtering.
  • FAN-LoRA samples n = 16 high frequencies by default from the band ω ∈ [π/2, π) to emphasize higher-frequency components while avoiding the singularity at π.
  • Fixed sinusoidal projection matrices map features into an implicit spectral subspace, and their columns represent distinct sampled frequency components.
  • The high-pass forward process uses a learnable spectral vector Sfreq to modulate projected features and reconstruct a residual in the original feature space.
  • The Fourier projection basis exhibits regular low-frequency sinusoidal patterns but waveform distortion at some mid-frequencies because of discrete sampling mismatch.
  • Frequency decoupling assigns large-scale domain shifts to the B-spline branch and local texture mutations to the Fourier branch.

A. Datasets

The evaluation covers three medical segmentation domain-adaptation settings spanning cross-modality and cross-center shifts. The benchmarks include cardiac, prostate, and abdominal organ segmentation tasks.

  • MM-WHS 2017 evaluates cross-modality cardiac segmentation from MR as source domain to CT as target domain.
  • Promise 12 and NCI-ISBI evaluate cross-center prostate segmentation under different scanner conditions.
  • FLARE 22 and CHAOS are used for cross-modality abdominal organ segmentation.

B. Implementation Details

Experiments use a frozen MedSAM backbone and train only the inserted FAN-LoRA modules. Inputs are standardized to 1024×1024 pixels, with source-domain supervised fine-tuning and consistent evaluation splits.

  • All experiments use the officially pre-trained MedSAM model as the backbone, with backbone parameters frozen during fine-tuning.
  • Gradients are computed exclusively for the newly inserted FAN-LoRA modules.
  • Table I summarizes the datasets and domain-adaptation settings used in the experiments.
  • Input images and segmentation masks are resized to 1024×1024 pixels, and training runs for 100 epochs with AdamW, batch size 2, and learning rate 1 × 10^-4.
  • Supervised fine-tuning is performed only on the source domain using identical data splits across methods for fair comparison.

C. Experimental Results

Across cross-modality and cross-center evaluations, FAN-LoRA improves segmentation quality, boundary precision, convergence stability, source-domain retention, and parameter efficiency versus PEFT baselines. Its frequency-decoupled branches balance anatomical structure alignment with local detail preservation under domain shifts.

  • Target-domain comparisons: FAN-LoRA achieves competitive accuracy and robustness across four target-domain evaluations against linear, nonlinear, and frequency-based PEFT methods.The comparison includes LoRA, DeLoRA, SHiRA, AuroRA, FourierFT, and FreqFiT.
  • Cross-center robustness: FAN-LoRA consistently achieves the highest Dice scores and lowest HD95 across 1.5T and 3T NCI-ISBI settings, with larger gains in HD95 than Dice.The result indicates stronger boundary precision under cross-center intensity changes and acquisition noise.
  • Cross-modality robustness: In the MM-WHS cross-modality setting, FAN-LoRA outperforms baselines by jointly optimizing global anatomical alignment and local organ-boundary refinement.The design is especially beneficial for complex cardiac structures requiring both global topology and local boundary precision.
  • Retention and efficiency: FAN-LoRA preserves source-domain performance without catastrophic forgetting and uses approximately 111.5K parameters, about four times fewer than standard LoRA.It can slightly improve over full fine-tuning on some source domains, while retaining superior segmentation accuracy and boundary precision.
  • Training dynamics: FAN-LoRA shows smoother, more stable convergence and less overfitting than AuroRA and FourierFT, while fine-tuning exhibits overfitting on both FLARE 22 and Promise 12.Its validation loss shows fewer oscillations, consistent with reduced overfitting to high-frequency noise.
  • Boundary precision: FAN-LoRA achieves the lowest HD95 across target domains, supporting more precise organ-boundary delineation and recovery of thin or low-contrast structures.Qualitative comparisons report sharper boundaries and more anatomically consistent structures than existing PEFT methods.

D. Ablation Study

The ablations show that FAN-LoRA’s low-pass and high-pass branches are complementary, while structured spectral design and restrained capacity improve boundary refinement and robustness. Performance can still diminish in extremely low-contrast regions where frequency cues are weak.

  • Core Module Ablation: The full FAN-LoRA model achieves the highest performance because global structural alignment and local detail compensation are complementary.Using either branch independently yields only marginal average-Dice improvements, whereas the combined design is most effective.
  • Core Module Ablation: The high-pass branch alone reduces HD95 from 30.51 to 30.19 on 1.5T and from 28.41 to 27.13 on 3T.These reductions demonstrate independent boundary-refinement benefits without structural guidance.
  • High-Pass Component Ablation: Fixed sinusoidal projections with learnable Sfreq achieve the best HD95 values of 26.96 on 1.5T and 26.48 on 3T.Learnable scaling degrades performance with random projections but improves results when paired with the sinusoidal basis.
  • Hyperparameter Sensitivity: A lower rank dimension combined with concise frequency sampling achieves peak Dice scores, whereas excessive dimensions marginally degrade performance.The analysis attributes this degradation to over-parameterization and renewed high-frequency noise fitting.
  • Hyperparameter Sensitivity: The high-frequency sampling count n = 16 balances representation capacity and over-parameterization.Smaller values under-represent high-frequency content, while n ≥64 adds redundant spectral components and overfitting risk without measurable gains.
  • Error Analysis: FAN-LoRA may struggle in extremely low-contrast regions where structural and textural cues are weak.The high-pass branch has limited signal to enhance, while the low-pass branch may over-smooth ambiguous boundaries.

V. CONCLUSION

FAN-LoRA is presented as a frequency-decoupled PEFT framework for medical foundation-model adaptation. Experiments show advantages in accuracy and boundary precision, although gains may diminish in extremely low-contrast regions.

  • FAN-LoRA explicitly separates low- and high-frequency optimization for medical foundation model adaptation.
  • Extensive experiments demonstrate competitive advantages over existing PEFT baselines in accuracy and boundary precision.
  • Performance gains may diminish in extremely low-contrast regions where frequency cues are intrinsically weak.
Loading 2608.26531v1…