PaperlayerView exact revision on arXiv

Source-linked AI summary

UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

Pengyu Wang, Baochen Xiong, Xiaoshan Yang, Yifan Xu, Zhang Qimeng, Haifeng Chen, Changsheng Xu

arXiv:2608.15516v1cs.LG

TL;DR

Federated instruction tuning for VLMs must accommodate privacy constraints alongside heterogeneity in tasks, modalities, and model architectures. UniFed-VLM combines compensated subspace aggregation with collaborative distillation, achieving better average performance across tasks than representative baselines on two heterogeneous federated benchmarks.

  • Problem

    Existing federated instruction-tuning methods do not adequately address the joint heterogeneity of tasks, modalities, and model architectures in VLM clients.

  • Method

    UniFed-VLM combines subspace-aligned adapter aggregation with dynamic weighting and compensation, plus two-stage collaborative distillation for cross-architecture knowledge transfer.

  • Results

    UniFed-VLM achieves better average performance across tasks than representative baselines on two heterogeneous federated benchmarks.

  • Takeaways & Limitations

    Combining compensated subspace aggregation with collaborative distillation is effective for federated VLM instruction tuning under heterogeneous tasks, modalities, and architectures.

  • Takeaways & Limitations

    TCoD adds client-side computation, FedCSA adds server-side overhead, and TCoD performance may degrade when architectural gaps become extremely large.

Abstract

from arXiv · show

Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation. However, fine-tuning of VLMs typically relies on centralized data, which raises privacy concerns in certain domains (e.g. healthcare). Federated Learning (FL) provides a natural solution by enabling model training without sharing raw data. However, applying FL to VLM instruction tuning is highly challenging. VLMs have substantial parameter scales, and in real-world scenarios, clients exhibit significant heterogeneity in tasks, modalities, and model architectures. Existing methods mainly focus on simplified settings and are unable to handle such multi-dimensional heterogeneous scenarios. In this work, we study federated instruction tuning under joint heterogeneity in tasks, modalities, and model architectures. We propose UniFed-VLM, a unified federated instruction tuning framework for VLMs that addresses multiple types of heterogeneity. It consists of two key components: 1) Federated Compensated Subspace Aggregation (FedCSA), which performs subspace-aligned aggregation of parameter-efficient adapters with dynamic weighting and compensation to mitigate heterogeneity-induced conflicts; 2) Two-stage Collaborative Distillation (TCoD), which enables effective knowledge transfer across heterogeneous models via a Mutual Distillation Adapter (MDA) and a mixture-of-experts-based distillation strategy. We conduct experiments on multiple benchmark datasets, and the results show that UniFed-VLM achieves stronger average performance across diverse tasks compared with existing FL methods. The source code is available at: https://github.com/wangpengyu2004/UniFed-VLM.

Introduction

The introduction frames federated instruction tuning for VLMs as difficult because clients exhibit intertwined task, modality, and model heterogeneity. UniFed-VLM addresses this setting with compensated subspace aggregation and collaborative distillation, supported by experiments on two public datasets.

  • Motivation: Centralized VLM fine-tuning creates privacy concerns because distributed data must be collected centrally, making aggregation impractical in sensitive domains such as healthcare.The introduction motivates combining parameter-efficient tuning with federated learning to avoid centralized data aggregation.
  • Challenge: Client heterogeneity is identified as the core challenge because it hinders effective collaboration in federated instruction tuning and parameter-efficient tuning.The paper emphasizes heterogeneity as an inherent obstacle to collaborative learning.
  • Problem Setting: The target setting jointly involves task, modality, and model heterogeneity, unlike existing methods that typically address only one or two types.Examples include VQA, visual grounding, and classification tasks; text and image modalities; and architectures with varying dimensions and layers.
  • Method: UniFed-VLM introduces FedCSA for same-architecture parameter aggregation using SVD-based subspace alignment, dynamic weighting, and compensation.FedCSA is designed to reconcile task and modality discrepancies during federated aggregation.
  • Contributions: UniFed-VLM combines FedCSA with Two-stage Collaborative Distillation to support knowledge transfer across heterogeneous architectures and addresses simultaneous task, modality, and model heterogeneity.The authors claim to be the first to explore this simultaneous setting and report extensive experiments on two public datasets.

Related Work

Prior work spans multimodal VLM architectures, privacy-preserving federated learning, and methods for adapting or aggregating models under heterogeneity. Existing approaches address parts of these challenges, motivating unified federated instruction tuning for heterogeneous VLMs.

  • Vision-Language Models: VLMs integrate vision and text for multimodal capabilities and include dual-stream architectures that separately encode modalities before cross-modal fusion.CLIP is cited as an example of a dual-stream architecture.
  • Federated Learning: Federated learning enables decentralized collaborative training while preserving data privacy and local autonomy.FedAvg, FedAdam, and FedProx address non-IID data distributions through iterative parameter aggregation.
  • Heterogeneous Federated Learning: Existing federated methods for model heterogeneity include parameter compression, public-dataset distillation, and prototype-based knowledge exchange.FedConv targets heterogeneous models through convolutional compression and decompression, FedMD uses server-maintained public data, and FedProto and FedTGP reduce communication or exchange knowledge through prototypes.
  • Heterogeneous Federated Learning: LoRA-based methods enable knowledge exchange across heterogeneous local models through homogeneous adapters, shared-rank matrices, or gradient-aligned mixtures of adapters.The cited approaches include a personalized LoRA framework, H2Tune, and HeteroTune with Dense Mixture of Adapters and CMGA.

Methodology

UniFed-VLM addresses federated VLM instruction tuning with joint heterogeneity in models, tasks, and modalities through PEFT-based aggregation and cross-model knowledge distillation. Its two modules, FedCSA and TCoD, align and compensate LoRA updates while transferring knowledge through mutual distillation adapters.

  • Problem setting: The framework considers K clients with private datasets, VLMs, and tasks, targeting heterogeneous global models that perform strongly on corresponding client data.Task differences can induce modality differences, such as classification versus VQA.
  • FedCSA: FedCSA aggregates LoRA parameters from clients sharing an architecture by concatenating them, applying randomized SVD, and partitioning singular vectors into task-specific feature representations.Aligned singular-vector bases encode task variations in consistent directions, reducing aggregation interference from task and modality heterogeneity.
  • FedCSA: FedCSA dynamically weights client updates according to variability, favoring stable updates, then learns complementary LoRA parameters to recover information lost during subspace aggregation.The compensation step is optimized with Adam for a few hundred steps and acts as low-rank error recovery.
  • TCoD: TCoD attaches a Mutual Distillation Adapter to each client and uploads only core attention parameters, enabling knowledge transfer across heterogeneous architectures without server-side shared data or parameter aggregation.The adapter uses globally shared attention layers and client-specific projection layers.
  • TCoD: TCoD first distills each VLM into its adapter, then distills back using a mixture-of-experts adapter with heterogeneous-client attention experts and gated routing.The original client attention module serves as an anchor expert, while task and MoE balancing losses jointly optimize the gating network and VLM.

Experiments

Experiments across Fed-Nature and Fed-Crossdomain evaluate UniFed-VLM under heterogeneous tasks, modalities, and models. UniFed-VLM improves average performance, remains stable under challenging heterogeneity, and benefits from its aggregation, distillation, and adapter-design choices.

  • Datasets and Models: Fed-Nature assigns each of four tasks to one client, whereas Fed-Crossdomain assigns each task to three clients across multiple datasets.Both datasets include VQA, visual grounding, caption generation, and classification.
  • Overall Results: UniFed-VLM improves performance across all Fed-Nature Scenario 1 tasks, exceeding FlexLoRA by 2.5% on classification and most baselines by approximately 4% on VQA.The advantage also appears under the more challenging heterogeneous settings.
  • Overall Results: 91.8% classification accuracy on Fed-Nature in Scenario 1 surpasses the centralized result of 91.3%, while UniFed-VLM maintains balanced performance across heterogeneous tasks.It approaches or surpasses the centralized upper bound on some individual tasks.
  • Ablation Study: 3.5% and 3.2% VQA accuracy decreases follow FedAvg and FedProx replacements, while classification accuracy decreases by 1.1% and 1.7%, respectively.Removing compensation or dynamic weighting also reduces performance, supporting both mechanisms within FedCSA.
  • Ablation Study: 3.6% lower VQA accuracy and 0.061 lower visual grounding IoU result when TCoD is removed, demonstrating the contribution of cross-model knowledge transfer.Without TCoD, the framework lacks cross-model knowledge transfer.
  • Convergence and Stability: The method reaches baseline-comparable convergence despite starting VLM training from the 5th epoch, achieving better average performance and improved stability across tasks.FedAdam instead exhibits severe instability under Scenario 2.
  • Architecture Analysis: dh= 2048 achieves the highest VQA accuracy of 0.743, visual grounding performance of 0.519 IoU, and classification accuracy of 0.918.Reducing or increasing the hidden dimension does not consistently improve performance.

Conclusion · APPENDIX

UniFed-VLM is a unified federated instruction-tuning framework for VLMs facing heterogeneity across tasks, modalities, and model architectures. Across two heterogeneous federated benchmarks, it achieves better average task performance than representative baselines by combining compensated subspace aggregation with collaborative distillation.

  • Conclusion: UniFed-VLM addresses joint heterogeneity across tasks, modalities, and model architectures in federated VLM instruction tuning.The framework is designed for multi-dimensional heterogeneous federated settings.
  • Conclusion: Experiments were conducted on two heterogeneous federated benchmarks.The reported evaluation covers heterogeneous federated benchmark settings.
  • Conclusion: UniFed-VLM achieves better average performance across tasks than representative baselines.The conclusion reports an average-performance advantage across tasks.
  • Conclusion: The framework unifies parameter aggregation and knowledge transfer for heterogeneous federated VLMs.Its approach combines compensated subspace aggregation with collaborative distillation.
  • Conclusion: Compensated subspace aggregation facilitates parameter aggregation under heterogeneous tasks, modalities, and model architectures.The method is presented as an effective mechanism for aggregating parameters despite these heterogeneous dimensions.
  • Conclusion: Collaborative distillation facilitates knowledge transfer under heterogeneous tasks, modalities, and model architectures.The conclusion identifies collaborative distillation as the complementary mechanism for transferring knowledge across heterogeneous federated VLMs.

A Implementation Details

The implementation uses 20 communication rounds with staged distillation, while specifying model dimensions, distillation hyperparameters, and MoE balancing settings for heterogeneous clients.

  • Training schedule: Training runs for 20 communication rounds, with one local epoch per round; the first five rounds perform only first-stage distillation with the VLM frozen.During this phase, the frozen VLM is distilled into the DMA.
  • Model configuration: The attention adapter hidden dimension is set to dh = 2048 for LLaMA 3.2-3B and Show-O 1.5B, whose hidden dimensions are 3072 and 2048.The adopted models are LLaMA 3.2-3B and Show-O 1.5B.
  • Distillation and MoE settings: Both distillation stages use temperature 2.0, with weights λS1 = 0.7 and λS2 = 0.2; the smaller second-stage weight limits interference with the task loss.The MoE expert count equals the number of heterogeneous clients, while the gating-network learning rate starts at 1 × 10−4 and decays over rounds.
  • MoE losses: The MoE objective includes a standard autoregressive task loss and a balance loss, whose weight is set to λMoE = 0.01.The supplied implementation text identifies the task loss as identical to the standard autoregressive loss.

B Derivation of the Compensation Objective

The compensation objective preserves the efficiency of separately aggregated low-rank factors while recovering information lost from decoupling their bilinear coupling. It optimizes compensation parameters so the global update better approximates client-side behavior in loss and update space.

  • Motivation and formulation: Separate aggregation of low-rank factors reduces storage and communication costs but can lose information by decoupling their bilinear coupling.Compensation parameters ∆B^l and ∆A^l are introduced to refine the global low-rank representation.
  • Loss consistency objective: The loss consistency objective minimizes the discrepancy between compensated global and client-side updates at the loss level.A smaller discrepancy indicates that the global model better approximates the performance of local models.
  • Interpretation of the compensation objective: Compensation terms restore interaction information lost when independently averaging the left and right low-rank factors.The aggregated factors define a shared low-rank subspace, while ∆B^l and ∆A^l recover lost coupling within that space.
  • Interpretation of the compensation objective: The objective performs error recovery in low-rank space instead of reconstructing the full high-dimensional update matrix, making it suitable for large-scale federated settings.It searches for a better global representation in the low-dimensional factor space.
  • Optimization procedure: In practice, Adam optimizes only ∆B^l and ∆A^l for a few hundred iterations while the client and aggregated factors remain fixed.The resulting correction mitigates information loss from factor-wise aggregation and improves approximation of client behaviors.

C Analysis of the Number of Iterations

Experiments on Fed-Nature compare 100, 200, and 300 optimization iterations, finding that I = 200 offers the best overall performance–efficiency trade-off. Although I = 300 improves Caption Generation, I = 200 outperforms it on other tasks and reaches near-convergence.

  • Performance comparison: When I = 100, performance is consistently lower across all tasks than with I = 200.I = 200 is the setting adopted in the main paper.
  • Performance comparison: Increasing iterations to I = 300 improves Caption Generation but reduces performance on other tasks relative to I = 200.Thus, the highest iteration count does not provide the best overall results.
  • Convergence analysis: The loss curve has not fully converged at I = 100, whereas the model is already largely converged at I = 200 with only marginal differences from I = 300.This indicates limited optimization benefit from extending iterations beyond 200.
  • Final selection: Considering performance and computational efficiency, I = 200 provides the best trade-off and is selected as the optimal setting.The selected iteration count balances accuracy with optimization cost.

D Hyperparameter Analysis … Impact of λS1 and λS2

The hyperparameter analysis evaluates MoE weighting, collaborative-knowledge coefficient, MDA projection depth, and distillation weights under Fed-Nature Scenario 1. Results identify settings that balance task optimization, feature alignment, and cross-model knowledge transfer.

  • Effect of MoE-related Hyperparameters: MoE-related hyperparameters govern the balance between task optimization, expert collaboration, and feature transformation capability.Experiments use Fed-Nature under Scenario 1 with the main-experiment evaluation protocol.
  • Effect of MoE-related Hyperparameters: λMoE values of 0.1 or 0.2 reduce VQA and CLS performance, while 0.4 degrades most tasks by overemphasizing MoE optimization.These results indicate that insufficient or excessive MoE task supervision can interfere with the original task objective.
  • Effect of MoE-related Hyperparameters: λMoE = 0.3 achieves the highest VQA accuracy and competitive performance on other tasks, so it is adopted as the default.The setting provides the best overall balance among the evaluated values.
  • Impact of α: The coefficient α produces slight performance variation: α = 0.3 lowers CLS accuracy, whereas α = 0.5 improves visual grounding but slightly reduces VQA accuracy.The coefficient regulates collaborative knowledge during optimization, and larger knowledge transfer can trade off task performance.
  • Impact of Projection Layer Depth: Increasing MDA projection depth from 2 to 3 consistently improves most tasks by enabling more effective feature alignment between heterogeneous models.Further increasing depth to 4 does not produce consistent improvements, although it benefits caption generation.
  • Effect of Distillation Hyperparameters: λS1 and λS2 control knowledge transfer from VLMs to MDA and from MDA back to VLMs, respectively, in the two-stage distillation process.The analysis is conducted on Fed-Nature under Scenario 1 using the main-experiment evaluation protocol.
  • Impact of λS1 and λS2: With λS1 = 0.7, increasing λS2 from 0.1 to 0.2 consistently improves performance across tasks, while λS2 = 0.3 causes degradation, particularly in caption generation.The weights jointly determine the trade-off between cross-model knowledge transfer and task-specific optimization.

E Another Scenario

UniFed-VLM remains robust when LLaVA handles VQA and visual grounding while Show-O handles classification and caption generation. It achieves the best results on three tasks and remains comparable for caption generation, indicating effective task adaptation and cross-model knowledge transfer.

  • Task-model assignment: Under a heterogeneous task-model assignment, LLaVA handles VQA and visual grounding while Show-O handles classification and caption generation.This scenario tests collaboration under different task distributions across heterogeneous VLMs.
  • Performance: 0.738 Acc, 0.365 IoU, and 0.899 Acc are UniFed-VLM’s best results on VQA, visual grounding, and classification, respectively.FedAvg performs slightly better for caption generation, while UniFed-VLM remains comparable with other methods.
  • Method analysis: UniFed-VLM combines FedCSA and TCoD to balance task-specific adaptation with cross-model knowledge transfer under heterogeneous configurations.FedAvg and FedProx are limited by direct parameter aggregation, while HetLoRA and pFedLoRA face challenges transferring complementary knowledge across heterogeneous VLMs.

F Privacy Issues Analysis · G Federated Optimization

UniFed-VLM incorporates differential privacy with limited performance variation and organizes training into two-stage local distillation followed by architecture-aware server aggregation. The optimization combines LoRA, MDA, MoE gating, subspace alignment, and compensation across communication rounds.

  • F Privacy Issues Analysis: Differential privacy adds Gaussian noise to clipped LoRA updates before aggregation to protect sensitive information while preserving personalized local adaptation.The privacy mechanism is applied during federated fine-tuning.
  • F Privacy Issues Analysis: 0.019 VQA accuracy decreases from 0.743 to 0.724 with DP, while caption generation, visual grounding, and classification remain comparable to the non-private setting.The passage identifies reasoning-oriented tasks as relatively more sensitive to perturbations.
  • G Federated Optimization: UniFed-VLM optimizes over T communication rounds, each containing local two-stage collaborative distillation and server-side aggregation.Clients receive global LoRA parameters and the shared MDA module at the beginning of each round.
  • G Federated Optimization: Clients perform E local epochs, first optimizing MDA representations with the VLM frozen using LS1, which combines task and knowledge-distillation losses.The MDA learns task-specific representations from the VLM during Stage 1.
  • G Federated Optimization: In Stage 2, clients fix the MDA attention module and optimize LoRA parameters with the MoE gating network through reverse distillation.The LoRA-adapted VLM learns from the MDA.
  • G Federated Optimization: FedCSA aggregates LoRA parameters among clients sharing a VLM architecture after transforming LoRA matrices into a shared low-rank subspace using randomized SVD.Adaptive subspace aggregation uses update stability.
  • G Federated Optimization: FedCSA applies lightweight compensation optimization to reduce information loss, while heterogeneous clients’ uploaded MDA attention modules update each local model.The passage describes both same-architecture LoRA aggregation and cross-architecture MDA utilization.

H Limitations

UniFed-VLM remains limited by added computation on both clients and the server. TCoD adds client-side training stages, while FedCSA increases server-side aggregation overhead compared with simpler baselines.

  • Computational Overhead: TCoD introduces additional client-side computation because it contains extra training stages, potentially challenging resource-constrained devices.The limitation arises from the two-stage collaborative distillation process.
  • Computational Overhead: FedCSA incurs extra server-side overhead from SVD decomposition and compensation parameter optimization.These operations increase aggregation cost relative to methods with negligible aggregation overhead.
  • Aggregation Cost: FedAvg and FedProx introduce negligible aggregation costs, whereas FlexLoRA and HetLoRA require approximately 1 minute per communication round and FedCSA takes around 2 minutes.FedCSA’s higher cost is attributed to additional optimization operations.
Loading 2608.15516v1…