PaperlayerView exact revision on arXiv

Source-linked AI summary

BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

Payam Abdisarabshali, Fardis Nadimi, Kasra Borazjani, Naji Khosravan, Richeng Jin, Seyyedali Hosseinalipour

arXiv:2610.12907v1cs.CVcs.LG

TL;DR

Because VLM data are distributed across entities and few FedVLM studies address personalization and generalization together, BLEND introduces a selective global-local adaptation strategy with a loss that combines their outputs. The paper reports that BLEND consistently outperforms state-of-the-art baselines, while noting that its scope and deployment settings remain limited in several ways.

  • Problem

    VLM data are distributed across entities, and balancing personalization with generalization in the emerging FedVLM area has only a few relevant studies.

    Source: §1 · p. 1

  • Method

    BLEND aggregates global vision and text adapters across clients, keeps a personalized vision projection adapter local, and uses a dual-objective loss to combine their representations.

    Source: §1 · p. 2, §3.2.1 · p. 4

  • Results

    BLEND consistently outperforms state-of-the-art FedVLM baselines, with experiments validating its introduced model components.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    BLEND’s global-local design and loss formulation are reported to improve the balance between personalization and generalization in FedVLMs.

    Source: §6 · p. 10

    BLEND focuses on CLIP and text–image modalities; broader modalities and tasks, adaptive hyperparameter selection, serverless settings, and privacy-preserving aggregation remain open directions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs). In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client. BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM. Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings. (Code is available at: https://github.com/payamsiabd/BLEND.git)

1 Introduction

Distributed VLM data motivates collaborative fine-tuning, while FedVLM research has few studies addressing the joint balance of personalization and generalization. BLEND responds with global and client-specific adaptation components and a loss designed to combine their representations while preserving generalized knowledge.

1 Introduction

  • Collaborative fine-tuning of VLMs across distributed clients is motivated by data being inherently distributed across numerous entities.The paper describes FedVLMs as a rapidly emerging research area.Source: §1 · p. 1
  • BLEND combines global vision and text adapters with a client-local vision projection adapter, using a dual-objective loss to balance general and client-specific representations.The design also seeks to preserve generalized knowledge encoded in CLIP’s text branch.Source: §1 · p. 2
  • The research question is how to adapt CLIP’s vision and text branches to preserve vision generality, avoid text overpersonalization, and enable client-specific personalization.The motivating intuition is that the vision branch has lower dataset-level recognition performance and more generalizable representations, whereas the text branch has higher recognition and may overfit with further personalization.Source: §1 · pp. 1–2
  • BLEND selectively aggregates global adapters across clients while keeping the personalized vision projection adapter local, and its experiments report consistent gains over state-of-the-art FedVLM baselines.The paper characterizes the framework as a simple global-local decomposition paired with a modified loss function.Source: §1 · p. 2

2 Related Work

Prior work spans centralized prompt and adapter fine-tuning as well as federated prompt and adapter methods, with only a few FedVLM studies targeting both personalization and generalization. BLEND is presented as a systematic design for that joint objective in federated fine-tuning.

2.1 Centralized VLM Fine-Tuning Techniques

  • Centralized VLM fine-tuning includes prompt methods, which are lightweight but often have limited performance relative to more expressive adapter-based approaches.Adapter-based methods introduce trainable modules for task-specific information while keeping most parameters frozen.Source: §2.1 · p. 2
  • BLEND targets the underexplored joint design of parameter-efficient adapters, CLIP architectural adjustments, and a loss explicitly balancing personalization and generalization in federated VLM fine-tuning.The cited centralized studies instead focus on centralized fine-tuning strategies.Source: §2.1 · p. 2

2.2 Federated Vision-Language Models (FedVLMs)

  • FedPGP, pFedMMA, and PromptFolio are identified as among the few FedVLM studies aiming to improve both personalization and generalization.The area also includes federated prompt optimization and adapter-based fine-tuning for clients with non-IID data.Source: §2.2 · pp. 2–3
  • BLEND’s experiments consistently outperform relevant FedVLM baselines, and the paper presents it as a new state of the art in the literature.Its architecture, local training and aggregation procedures, and loss formulation are all tailored to balance generalization and personalization.Source: §2.2 · p. 3

3 Method

BLEND balances client-specific personalization with recognition of unseen classes by combining locally retained vision-projection adapters with globally aggregated vision and text adapters, fusing global and personalized visual representations with an anchor loss. Its training process alternates local refinement with server aggregation and redistribution of shared parameters while clients retain their localized parameters.

3.1 Background: CLIP in a Nutshell

  • CLIP uses pretrained vision and text encoders to produce shared-space embeddings, whose cosine similarities support classification across candidate labels.Its encoders are transformer stacks, pretrained on image–text pairs with a contrastive objective.Source: §3.1 · p. 3

3.2.1 BLEND in a Nutshell

  • BLEND pairs globally aggregated vision and text adapters for generalizable representations with a client-local personalized projection adapter, then balances their outputs through a novel loss.The global adapters learn collaboratively across clients; the projection adapter remains local to capture client-specific representations.Source: §3.2.1 · p. 4
  • BLEND’s fused global and personalized representations achieve the highest reported personalization while maintaining superior generalization and the highest harmonic mean among the compared strategies.The comparison averages results over 10 datasets.Source: Figure 2 · p. 4, §3.2.1 · p. 5

3.2.2 Theoretical Motivations

  • Under the Markov chain S → V → T, the data-processing inequality gives I(S; T) ≤ I(S; V), motivating stronger client-specific adaptation of visual representations than textual ones.Prior work cited in the paper reports sparse textual captions relative to images [32] and stronger dataset-specific behavior in textual representations.Source: §3.2.2 · p. 5

3.2.3 Local Model Architecture

  • Vision and text adapters are residually added within their transformer blocks, while a two-branch vision head combines the frozen CLIP projection with a personalized projection adapter using a weighted average.The trade-off weight γ ∈ [0, 1] controls the balance between generalization and personalization.Source: §3.2.3 · pp. 5–6

3.2.4 Local Model Fine-Tuning

  • Client training minimizes a representation-fusion cross-entropy loss together with a weighted running-anchor loss that aligns the general vision representation with text representations from frozen zero-shot CLIP projection heads.The anchor evolves as the global vision and text adapters update the representations.Source: §3.2.4 · p. 7

3.2.5 Global Aggregation

  • BLEND aggregates global vision and text adapters but keeps each personalized projection adapter local, motivated by greater diversity in clients’ visual data than in their label- or prompt-based text data.Source: §3.2.5 · p. 7
  • Clients transmit only aggregatable parameters after local training and retain localized parameters for personalization.The transfer follows J local SGD iterations.Source: §3.2.5 · p. 7
  • The server aggregates shared parameters and redistributes the update for the next communication round, while clients retain localized parameters for refinement.Source: §3.2.5 · p. 7
  • The next local training round follows equation (19), with k replaced by k + 1 and initialization set to PTrain,(k+1,0).Source: §3.2.5 · p. 8

4 Experiments

Experiments evaluate BLEND across multiple datasets and system settings, finding strong personalization–generalization balance and examining the effects of adapter ranks, placement, loss weights, and aggregation choices.

Experimental Setup

  • The experiments use 10 datasets, base/novel splits, 16 shots per class, and a 10-client system, comparing BLEND with zero-shot CLIP and federated baselines.The main setting uses highly non-IID data and reports personalization, novel-class generalization, and their harmonic mean.Source: §4 · p. 8

Main Results

  • Across datasets, BLEND achieves the best H-Mean, improving the average by +4.81 and yielding a 2.06% lower accuracy standard deviation than the best baseline.The main comparison follows 50 global rounds, when all methods have converged.Source: §4 · pp. 8–9

Ablation Studies

  • Injecting adapters at all vision layers gives the highest accuracy, but layers 5–12 achieve an H-Mean very close to the all-layer setting.Adapter placement has a notable impact on average performance.Source: Figure 3 · p. 9, §4 · p. 9
  • γ=0.09 and λ=0.2 yield the highest H-Mean in their respective sensitivity analyses; excessive λ weakens zero-shot anchoring and gives the worst generalization.Larger γ values worsen the personalization–generalization trade-off.Source: Figure 3 · p. 9, §4 · pp. 9–10

Robustness

  • BLEND maintains superior H-Mean across data heterogeneity, client counts, and few-shot settings, with larger gains as heterogeneity, system size, and data scarcity increase.Tests include iNaturalist and systems with 100 and 200 clients.Source: §4 · p. 9

Ablation Studies

  • Global adapter ranks consistently outperform no adapters, with H-Mean rising through rank 16; personalized rank 256 gives the best personalization–generalization trade-off.Increasing personalized rank improves personalization accuracy while gradually reducing generalization.Source: Table 2 · p. 9, Table 3 · p. 9, §4 · p. 9
  • Aggregating global vision/text adapters while keeping the personalized projection adapter local gives the best overall H-Mean among the tested aggregation strategies.Full aggregation improves generalization but substantially harms personalization; no aggregation notably reduces generalization.Source: §4 · p. 10
  • Other ablations find that unfreezing the text projection head mostly degrades H-Mean, the running anchor beats two alternatives, and explicit personalization slightly lowers overall H-Mean.The optimal fusion weight varies with data heterogeneity and client count; held-out validation recovers the reported optimal γ, λ, and adapter ranks.Source: §4 · p. 10

5 Limitations

The paper identifies scope and deployment constraints for BLEND, alongside open directions for extending the method.

Scope

  • BLEND is built on CLIP and focuses on text–image modalities, leaving audio/video and retrieval or generative tasks as open extensions.Multi-task learning is also identified as an open direction.Source: §5 · p. 10

Hyperparameter Selection

  • Adaptive selection of γ and λ in new environments remains open because BLEND’s performance depends on these personalization–generalization trade-off hyperparameters..Source: §5 · p. 10

Deployment and Privacy

  • Fully decentralized or serverless FedVLM deployment and integrating privacy-preserving aggregation, such as differential privacy, remain open directions.Source: §5 · p. 10

6 Conclusion

The conclusion presents BLEND as a FedVLM framework combining globally aggregated CLIP adapters with a locally optimized vision projection head, and reports consistent gains over state-of-the-art baselines.

Conclusion

  • BLEND combines globally aggregated adapters in CLIP’s vision and text encoders with a locally optimized vision projection head.Source: §6 · p. 10
  • A novel loss formulation uses these components’ outputs to improve the balance between personalization and generalization.Source: §6 · p. 10
  • Experiments validate the introduced components and show that BLEND consistently outperforms state-of-the-art baselines.Source: §6 · p. 10

A Communication and Computation Overhead

Under matched communication and trainable-parameter budgets, BLEND maintains strong performance and outperforms competing methods, with computational costs comparable to existing methods.

Communication budgets

  • Under comparable communication overhead, BLEND achieves the highest H-Mean in the high-budget regime while transmitting fewer bits than MMA+FedAVG, and outperforms prompt-based and lightweight adapter methods in the low-budget regime.The tables measure communication overhead as total bits transmitted to the server over 50 global rounds, reported in MB.Source: A Communication and Computation Overhead · p. 15, Table 5 · p. 15, Table 6 · p. 15

Trainable-parameter budgets

Computational efficiency

B Data Heterogeneity Variations

BLEND maintains a strong personalization–generalization balance as data heterogeneity and client counts increase. Its margin over baselines grows with client count, while the heterogeneity sweep compares performance across Dirichlet settings.

B Data Heterogeneity Variations

  • The heterogeneity sweep compares personalization, generalization, and H-Mean across datasets as αDir changes, with lower αDir corresponding to more heterogeneous data.Source: B Data Heterogeneity Variations · p. 16, Figure 4 · p. 16
  • Across all experiments, BLEND consistently achieves the highest H-Mean, and its performance gap over baselines widens as data heterogeneity becomes more severe, indicating robustness under harsh non-IID conditions.Source: B Data Heterogeneity Variations · p. 16

C Varying the Number of Clients

D Large-Scale Setting with Real-World Data Distribution

BLEND retains strong personalization–generalization balance in real-world large-client settings, limited-shot learning, and global-adapter rank tests. The results also identify settings where performance trade-offs depend on the dataset or adapter rank.

D Large-Scale Setting with Real-World Data Distribution

  • For N = 100 clients, BLEND achieves the highest H-Mean of 63.37, outperforming all baselines; at N = 200, it achieves the best H-Mean of 57.39, maintaining a consistent gap over baselines.At N = 100, BLEND's Base and New accuracies are 86.96 and 49.84, respectively.Source: D Large-Scale Setting with Real-World… · p. 20

E Different Few-shot Settings

  • Across datasets, BLEND has the best average H-Mean among all other methods; across individual datasets, it is best at ≤16 shots, but is marginally outperformed on two datasets at 32 shots.Source: E Different Few-shot Settings · p. 20

F Different Bottleneck Sizes for Global Adapters

  • The average personalization–generalization balance is best at global adapter rank r = 16, although dataset-specific best ranks vary and r = 16 requires low computation and communication budgets.For example, Flowers has its best generalization with no global adapter, while DTD has its best generalization at full rank.Source: F Different Bottleneck Sizes for… · p. 24, Table 14 · p. 24, Table 19 · p. 24

G Different Bottleneck Sizes for Personalized Projection Adapter

Personalized projection-adapter rank and adapter placement both affect BLEND’s balance of personalization and generalization. The best settings vary across datasets, while broader adapter placement performs best on average.

G Different Bottleneck Sizes for Personalized Projection Adapter

  • Across datasets, the personalized projection adapter’s best average H-Mean occurs at rp = 256, while its best personalization occurs at full rank; generalization often peaks at rp = 4 or 8.Full-rank settings yield the best personalization in 7 of 10 cases.Source: G Different Bottleneck Sizes for… · p. 25, Table 25 · p. 25

H Adapter Injection at Different Layers

I Effect of γ on Performance

Across datasets, small γ values provide the best balance of personalization and generalization, with average H-Mean optimal at γ = 0.09. Increasing γ favors personalization while higher values can reduce generalization.

I Effect of γ on Performance

J Effect of λ on Performance

Across the examined settings, BLEND retains the best H-Mean as federation difficulty increases, while sensitivity analyses show how aggregation choices and loss components shift personalization and generalization. Held-out validation supports the selected hyperparameters.

J Effect of λ on Performance

K Effect of Aggregation on Various Components

L Effect of Trainable Text Projection Head

  • The best average H-Mean occurs with the text projection head frozen; adapting it is redundant on average and adds training overhead without H-Mean gains.Without text projection adaptation, personalization is best in 6/10 scenarios and generalization in 5/10.Source: L Effect of Trainable Text Projection… · p. 36, Table 47 · p. 36

M Performance Variation Across Different Settings

N Effect of the Running Anchor Loss

O Effect of the Adaptive Fusion Weight

P Effect of Explicit Personalization

  • Explicit P-head CE raises personalization accuracy from 92.91% to 93.56%, but lowers generalization accuracy from 74.62% to 70.96% and H-mean from 82.15% to 79.78%.The comparison supports optimizing the personalized representation through the fusion objective instead.Source: P Effect of Explicit Personalization · p. 40, Table 61 · p. 40

Q Hyperparameter Selection and Held-Out Validation

R Limitations and Generalization Beyond CLIP

BLEND’s demonstrated scope is currently CLIP-based image–text classification. Extensions to other architectures, downstream tasks, and multi-task or multi-modal settings require changes to module placement, fusion, or training objectives.

Extension to Other Multimodal Architectures

  • BLEND currently focuses on CLIP-based image–text models; other architectures require reconsidering where global and personalized modules enter the pipeline.For Qwen2.5-VL, one proposed extension uses a global and a personalized vision merger.Source: R Limitations and Generalization… · p. 41

Extension to Other Downstream Tasks

  • BLEND currently focuses on classification; retrieval requires adapting fusion and training objectives, while generative tasks require adapting fusion and anchor objectives.Retrieval must preserve cross-modal alignment while balancing generalization and client-specific personalization.Source: R Limitations and Generalization… · p. 41

Extension to Multi-Task and Multi-Modal Settings

  • Multi-task and multi-modal extensions must address task-dependent modality imbalance and account for task similarity, modality relevance, and potential negative cross-task transfer.The authors identify task-dependent module placement and adaptive fusion weights as possible requirements.Source: R Limitations and Generalization… · p. 41
Loading 2610.12907v1…