PaperlayerView exact revision on arXiv

Source-linked AI summary

FedSSMCoOp: SSM Encoders for light-weight Federated Prompt Learning for Few-shot Classification

Ankita Das, Ambarish Parthasarathy, Sumohana S. Channappayya, C. Krishna Mohan

arXiv:2610.09907v1cs.CV

TL;DR

Biomedical VLM adaptation faces restricted data sharing, scarce labels, and institutional heterogeneity, while common alignment approaches rely on cosine similarity. FedSSMCoOp combines Vision Mamba and Cross-Mamba interaction with federated prompt updates. The authors report stable performance relative to baselines and an average model size that is 1.96 times lighter.

  • Problem

    Biomedical VLM adaptation must address restricted data sharing, scarce labeled samples, and substantial variation across institutions.

    Source: §1 · p. 2

  • Method

    FedSSMCoOp uses Vision Mamba and Cross-Mamba interaction for visual and cross-modal processing, while federated training communicates prompt updates.

    Source: §3.1 · p. 4, §4 · p. 5, Algorithm 1: FedSSMCoOp Federated… · p. 7

  • Results

    The authors report stable performance relative to baselines and that FedSSMCoOp is, on average, 1.96 times lighter than other models.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The authors describe FedSSMCoOp as suitable for early detection and inference based on UMAPs and activation maps.

    Source: §6 · p. 13

    The framework setup assumes K clients, each holding a private dataset.

    Source: §4 · p. 5

Abstract

from arXiv · show

Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information contained in the respective domains. Despite the performance gains, most of these approaches rely on aligning these domains using the cosine similarity metric, which fails to capture token-level structure and cross-modal interactions prior to the classification stage. This is especially critical in biomedical applications under federated constraints, where data sharing is restricted, labeled data is scarce at each site, and it differs widely across institutions, leading to substantial statistical heterogeneity. To overcome this issue, we propose FedSSMCoOp, a federated few-shot image classification framework that enables multimodal learning while preserving data privacy. With the help of the SSM-based Vision Mamba and Cross Mamba blocks, and by optimizing only the soft-prompt and communication-prompt updates in the federated setting, the framework prioritizes both computation and performance. Importantly, this eliminates the need to use an external Large Language Model (LLM) for feature alignment. The framework is further trained and evaluated on various biomedical image datasets, and its performance is assessed. The proposed framework delivers stable performance relative to the baselines and is, on average, 1.96 times lighter. The corresponding script will be made available soon.

1 Introduction

Vision-language models combine complementary modalities, but biomedical adaptation must contend with privacy restrictions, scarce labels, and substantial cross-institution variation. FedSSMCoOp combines Mamba-based visual and cross-modal processing with federated prompt updates to balance computation and performance.

1 Introduction

  • CLIP aligns image and text features through contrastive learning inspired by InfoNCE (Oord et al. (2018)), while BLIP supports retrieval and generation (Li et al. (2022)).The introduction also describes SigLIP as replacing CLIP's contrastive approach with a sigmoid objective (Zhai et al. (2023)).Source: §1 · pp. 1–2
  • Biomedical VLM adaptation is difficult because institutional privacy limits data sharing, labels are scarce, and images vary with devices, protocols, and patient populations.These factors make the setting highly heterogeneous, especially in few-shot adaptation.Source: §1 · p. 2
  • Mamba reduces attention's quadratic computation complexity to almost linear, but may be less effective at fine-grained features than attention-based models.The cited comparison attributes this trade-off to attention extracting fine-grained features while Mamba emphasizes global features (Parthasarathy & Channappayya (2026)).Source: §1 · p. 2
  • FedSSMCoOp combines Vision Mamba and Cross-Mamba interaction with federated learning that communicates only soft-prompt deltas.The approach targets performance and computational complexity while reducing per-round communication cost.Source: §1 · p. 2

2 Related Works

Prior work addresses federated learning, prompt learning, or state-space models, but the supplied review identifies fewer approaches combining multimodal interaction with federated SSMs. FedSSMCoOp is presented as addressing that intersection without an external LLM for feature alignment.

2 Related Works

  • FedAvg (McMahan et al. (2017)) trains across clients, while FedProx (Li et al. (2020)) and SCAFFOLD (Karimireddy et al. (2020)) address non-IID heterogeneity by sharing full model updates.CoOp (Zhou et al. (2022b)) optimizes learnable context tokens; PromptFL (Guo et al. (2023a)) aggregates soft prompts.Source: §2 · p. 2
  • Table 1 compares methods by federated learning, prompt learning, Vision Mamba, and Mamba Interaction Block support, using ✓, ✗, and ∼ for supported, unsupported, and partial support.The caption identifies the comparison scope but does not provide specific method-by-method outcomes.Source: Table 1 · p. 3, §2 · p. 3
  • Prior federated prompt methods share prompts or use LLM-derived attributes, while many Mamba approaches are unimodal and centralized or lack cross-modal interaction.The review frames FedSSMCoOp as combining federated learning, Vision Mamba, and a Cross-Mamba block to capture cross-modal cues.Source: §2 · pp. 2–3

3 Preliminaries

The preliminaries describe selective state-space processing for image patches and Cross-Mamba processing for cross-modal features. They also introduce federated learning, its client heterogeneity and communication constraints, and prompt-based communication as a way to share fewer parameters.

3.1 State Space Models (SSMs)

  • Vision Mamba embeds image patches with positional information and models the sequence recurrently, using input-dependent projections for selective gating.The hidden state captures context from preceding patches, while the input-dependent matrices allow informative patches to enter memory and background patches to be suppressed.Source: §3.1 · p. 4
  • The SSM output can be computed in convolutional or sequential mode, supporting optimized training and inference.The passage identifies M as input sequence length and K as the corresponding kernel.Source: §3.1 · p. 4
  • The Cross-Mamba block processes high-level spatial features from both modalities, adds positional embeddings, and scans features in four directions through an SSM structure.The approach is described as a lighter alternative to attention-based cross-modal processing.Source: §3.1 · p. 4

3.2 Federated Learning

  • Federated learning trains a shared model across clients while keeping their datasets private, with the server aggregating client updates by weighted averaging.Clients perform local SGD on private data before the server aggregates their updates.Source: §3.2 · p. 5
  • Client data can be non-IID, and sharing full vision-encoder weights makes communication costly; prompt-based federated learning shares a small set of learnable parameters while keeping encoders frozen and local.Class labels are partitioned across clients using a Dirichlet distribution to simulate medical-imaging heterogeneity.Source: §3.2 · p. 5

4 Proposed Approach

FedSSMCoOp combines Mamba-based vision and cross-modal interaction with federated prompt learning for zero- and few-shot image classification. It keeps image data local and communicates prompt updates between clients and the server.

4 Proposed Approach

  • FedSSMCoOp pairs Vision Mamba with bidirectional Cross Mamba fusion for zero- and few-shot classification, proposing the interaction block as a substitute for an LLM’s feature-alignment role.The architecture fuses image and text streams into text-aware and image-aware features.Source: §4 · p. 5, Figure 1 · p. 6
  • Each client keeps medical images local and trains on at most K-shot samples per class, with a Dirichlet partition simulating non-IID clinical distributions.Smaller α induces greater client heterogeneity; the goal is generalization across all classes, including classes unseen at individual clients.Source: §4 · p. 5
  • Clients initialize local prompts from the global prompt and upload only prompt deltas, making communication cost scale with prompt size.The global prompt is maintained by the server; clients send prompt deltas rather than full model weights.Source: §4 · p. 5

Algorithm 1: FedSSMCoOp Federated Training

  • Local training uses frozen Vision Mamba and BiomedCLIP encoders, while soft prompts, text projection, and MIB parameters receive gradient updates.MIB fuses image and text tokens bidirectionally, and mean-pooled image-aware features are scored against text-aware class tokens.Source: Algorithm 1: FedSSMCoOp Federated… · pp. 6–7
  • Only the soft prompt is aggregated; client-specific text projection and MIB parameters remain local to adapt to non-IID data, and performance is averaged across clients.Inference combines the aggregated global prompt with each client’s locally learned parameters.Source: Algorithm 1: FedSSMCoOp Federated… · p. 7

5 Experiments

Experiments evaluate FedSSMCoOp across 11 biomedical datasets and compare its few-shot performance, stability under client and data heterogeneity, and computational complexity. The reported model is competitive on most datasets, with lower complexity and generally stable performance across non-IID conditions.

5 Experiments

  • The evaluation covers 11 diverse biomedical datasets; the reported few-shot setup uses 8 clients with α = 0.5 and communicates lightweight soft-prompt updates.The experiments average test results over three random seeds.Source: §5 · pp. 7–8

5.1 Comparison with Baselines

  • Nearly 1.96× reduction in computational complexity while remaining competitive on most datasets; gains were roughly 4% on CHMNIST, COVID, and LC25000, with underperformance on BUSI and CTKidney.The authors attribute the latter shortfalls to difficulty capturing fine-grained representations and localized pathological patterns.Source: §5.1 · pp. 8, 10
  • Activation maps show localized baseline attention and broader FedSSMCoOp attention; its weaker BUSI and CTKidney results occur on tasks relying on localized pathological patterns.Figure 3 compares baseline visualizations at 16 shots with FedSSMCoOp at 1 shot.Source: §5.1 · pp. 9–10, Figure 3 · p. 9
  • LLM use yielded only minor improvement; the LLM-free model was in-par or sometimes better, and remained within 5% on RetinaMNIST, OCTMNIST, and CHMNIST where LLM use helped.The proposed framework is LLM-free during training and inference; the LLM appears only in the ablation.Source: §5.1 · pp. 10–11

5.2 Effect of Client Participation

  • Across 4-, 8-, and 16-client settings, performance was generally stable across Dirichlet α; 4-client deviation was below 2%, with minor 8-client spikes on BTMRI and KVASIR.The 16-client system showed minimal fluctuations across α values.Source: §5.2 · p. 11, Table 8 · p. 12

5.3 Efficiency Analysis

  • Despite increased total parameters, FedSSMCoOp reports lower GFLOPs than most baselines, measured for a 224×224 input across the full inference pipeline.The reported pipeline includes the frozen visual and text encoders, prompt-learning modules, and Mamba Interaction Block.Source: §5.3 · p. 12, Table 9 · p. 12

6 Conclusion & Future Directions

FedSSMCoOp combines Vision Mamba with a Mamba interaction block for cross-modal learning, and reports stable performance relative to baselines at roughly 1.96 times lighter weight.

6 Conclusion & Future Directions

  • FedSSMCoOp uses Vision Mamba and a Mamba interaction block to capture cross-modal information beyond cosine similarity between pooled modality features.The block captures cross-modal information and feature alignments, allowing the LLM used for the same task to be removed.Source: §6 · pp. 12–13
  • The framework communicates soft-prompt deltas between clients and server to optimize computation and performance.The conclusion reports stable performance against baselines and says the framework is roughly 1.96 times lighter than other models.Source: §6 · p. 13
  • The authors identify fine-grained representation capture as a weakness of Mamba-based approaches and suggest future hybrid attention–state-space work.They also report that server-side LLMs aided performance on Kvasir and RetinaMNIST, motivating further study of LLM contributions to alignment.Source: §6 · p. 13

B Notation

Table 10 summarizes the notation used throughout the paper.

B Notation

B.1 Few-shot Results Across Shot Settings

Across 11 datasets, FedSSMCoOp is best on more datasets as the shot count rises, though its performance varies with dataset characteristics.

B.1 Few-shot Results Across Shot Settings

  • FedSSMCoOp is best on 4 of 11 datasets at 1-shot, 7 at 2- and 4-shot, 8 at 8-shot, and 10 at 16-shot.The reported few-shot accuracy results span k = 1, 2, 4, 8, and 16 shots.Source: §B.1 · p. 17, Table 12 · p. 17
  • FedSSMCoOp achieves the best performance across all shot settings on CHMNIST, COVID, and LungColon.On LungColon, accuracy rises from 68.6% at 1-shot to 92.4% at 8-shot.Source: §B.1 · p. 17
  • Performance is less consistent on datasets requiring fine-grained or localized visual discrimination.KneeXray is not best at any shot level, while BUSI, CTKidney, and DermaMNIST show larger gaps at lower shot levels.Source: §B.1 · p. 17

B.2 Dataset Characteristics

The reported backbone preference varies with dataset characteristics: global-context datasets generally favor Vision Mamba, while localized-discrimination datasets favor attention-based backbones.

B.2 Dataset Characteristics

  • Datasets characterized by global morphology or whole-image context generally favor Vision Mamba, with FedSSMCoOp best across most shot settings.Examples include CHMNIST, COVID, LC25000, and RetinaMNIST.Source: §B.2 · p. 17, Table 17 · p. 19
  • Datasets requiring fine-grained or localized discrimination, including BUSI and KneeXray, favor attention-based backbones.Table 17 summarizes the observed relationship between dataset characteristics and best-performing backbone family across shot settings.Source: §B.2 · p. 17, Table 17 · p. 19
Loading 2610.09907v1…