PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Domain-Level Adaptation: Margin-Oriented Semantic-Appearance Interaction Correction for Personalized Federated Vision-Language Models

Wentao Yue, Qingyu Mao, Tianyou Lai, Ahmed M. Abdelmoniem, Qilei Li

arXiv:2610.01625v1cs.CV

TL;DR

Personalized federated VLM tuning often treats domain effects as class-agnostic, despite class-dependent shifts and residuals that vary in their effect on image-text margins. MOSAIC models these interactions and selectively corrects harmful pairs using shared class structure and private domain factors. It improves macro-client top-1 accuracy across the evaluated datasets and domain-shift settings.

  • Problem

    Personalized federated VLM methods largely model each domain as a class-agnostic transformation, although a fixed domain can shift semantic classes differently.

    Source: Introduction · pp. 1–2

  • Method

    MOSAIC scores class-domain residuals by their true-versus-competitor margin effect and uses a low-rank adapter with shared class structure, private domain factors, and gated candidate-wise correction.

    Source: Introduction · p. 2, Factorized Class-Domain Residual Adapter · p. 5

  • Results

    MOSAIC improves macro-client top-1 accuracy across the three datasets and both client-partition protocols, with the largest gains on Office31 and smaller positive gains on OfficeHome and DomainNet100.

    Source: Comparisons under Domain and Label… · p. 7

  • Takeaways & Limitations

    The results support targeted correction of decision-harmful class-domain interactions while preserving shared class structure and client-private domain activation.

    Source: Introduction · p. 2, Conclusion · p. 7

    The formulation associates each client with a domain and retains a client-private domain factor.

    Source: Problem Formulation · p. 3

Abstract

from arXiv · show

Federated parameter-efficient fine-tuning enables distributed clients to adapt pretrained vision-language models without sharing raw data or updating the full backbone. Its effectiveness, however, is limited by domain heterogeneity across clients. Existing personalized methods separate globally shared knowledge from client-specific style, but they largely treat each domain as a class-agnostic transformation. We show that this abstraction is insufficient: the cross-domain displacement associated with a fixed domain varies across semantic classes, and only a subset of these class-domain residuals damages the image-text decision margin. We therefore propose Margin-Oriented Semantic-Appearance Interaction Correction (MOSAIC), which first constructs a decision-aware harmfulness score that measures whether a training-derived class-domain residual favors a competing text prototype over the true class. It then models fine-grained class-domain interactions with a low-rank residual adapter whose class factors and residual basis are globally shared while domain factors remain client-private. An image-conditioned gate further controls candidate-wise correction, and harmful-pair-aware reweighting prioritizes decision-relevant residuals during local optimization. Extensive experiments on Office31, OfficeHome, and DomainNet100 demonstrate that MOSAIC consistently improves macro-client top-1 accuracy across all evaluated domain-shift and joint domain-label-shift settings.

Introduction

Personalized federated VLM methods separate shared knowledge from client-private style but generally model domain effects as class-agnostic. MOSAIC instead targets class-domain residuals that harm image-text margins, using shared class structure and private domain factors for selective correction.

Introduction

  • Existing personalized federated VLM tuning treats a domain largely as a class-agnostic transformation, leaving its effect on frozen image-text representations underspecified.The paper identifies this as a coarse modeling choice for separating transferable semantic structure from domain-dependent visual variation.Source: Introduction · p. 1
  • A fixed domain can shift different semantic classes differently, leaving class-conditioned residuals beyond the common domain effect.Dog and chair shifts across DomainNet rendering domains illustrate this class-domain interaction.Source: Introduction · p. 2
  • Only residuals that favor a competing text prototype over the true class are decision-harmful; residual size alone does not determine classification harm.The harmfulness score measures this effect on the true-versus-competitor margin; PCA is used only for visualization, while calculations use the original feature space.Source: Introduction · p. 2
  • The correction must model class-domain interactions, use decision-aware selection, preserve client-private domain effects, and avoid perturbing harmless residuals.These requirements follow from the observed class-dependent shifts and the distinction between geometric residuals and margin harm.Source: Introduction · p. 2
  • MOSAIC uses a low-rank residual adapter with shared class structure and private domain factors, plus harmful-pair-aware optimization and an image-conditioned candidate-wise gate.Only shared class-side structure is aggregated; the gate subtracts a residual only when useful.Source: Introduction · p. 2

Related Work

Prior work adapts frozen VLMs with compact prompts or adapters and addresses client and domain heterogeneity through personalized states or domain-level transfer. MOSAIC distinguishes itself by modeling class-specific domain residuals and selecting corrections by their image-text margin effect.

Parameter-Efficient VLM Tuning

  • PEFT methods such as CoOp and CoCoOp adapt frozen VLMs with learned context tokens, while later methods use regularization or conditioning to improve transfer.These methods operate at task, image, or class-prompt levels rather than identifying harmful class-domain residuals.Source: Related Work · p. 2

Personalized Federated VLM Tuning

  • Personalized federated VLM methods balance shared and client-specific prompts or adapters, but do not test class-specific domain residuals against the true-versus-competitor margin.The passage describes methods including PromptFL, FedOTP, FedPGP, DiPrompT, FedPHA, FOCoOp, and FedDDA.Source: Related Work · p. 2

Domain Shift and Fine-Grained Interactions

  • Federated domain-generalization methods use alignment, regularization, prototype exchange, aggregation, prompts, or adapters, yet target domain-level invariance or adaptation rather than margin-tested class-conditioned residuals.The described distinction motivates MOSAIC’s separation of a common domain effect from the residual and its selective correction.Source: Related Work · p. 3

Problem Formulation

MOSAIC formulates personalized VLM adaptation around shared and private states so class-conditioned domain effects can be corrected without averaging them across clients. Clients retain domain factors locally, while the server aggregates shared parameters.

Problem Formulation

  • The formulation seeks to correct class-conditioned domain effects without averaging them away, separating transferable federation-wide state from domain-local state.It frames MOSAIC as personalized VLM adaptation with explicitly partitioned shared and private parameters.Source: Problem Formulation · p. 3
  • Each client owns a private dataset associated with its domain and retains a private domain factor, including when multiple clients share a visual domain.The client index denotes locality of the factor, not necessarily a distinct visual domain.Source: Problem Formulation · p. 3
  • The shared state contains global prompt, visual-adapter, and class-side residual parameters; the private state contains client-specific prompt and adapter parameters plus its domain factor.The server aggregates only the shared state from participating clients.Source: Problem Formulation · p. 3
  • Before residual correction, classification uses normalized visual and textual features produced by frozen encoders conditioned on MOSAIC’s adaptation states.The standard VLM classifier uses a learned logit scale.Source: Problem Formulation · pp. 3–4

Methodology

MOSAIC estimates class-domain residuals, scores their decision harm, and learns gated, factorized corrections while keeping domain factors client-private.

Methodology

  • Training-only feature statistics define a fixed harmful-pair set, then MOSAIC learns candidate-wise corrections and aggregates only shared parameters.The harmfulness score is based on the most adverse competitor-versus-true projection of each residual.Source: Methodology · p. 4

Class-Conditioned Domain Drift

  • A class-domain residual captures how a domain affects one class beyond its common effect across classes.The interaction is estimated from training features only.Source: Class-Conditioned Domain Drift · p. 4

Decision-Aware Harmful Drift Scoring

  • MOSAIC ranks class-domain pairs by whether their residual favors a competing text prototype over the true class, selecting a top quantile per domain.This avoids treating residual magnitude alone as evidence of classification harm and prevents higher-variation domains from dominating selection.Source: Decision-Aware Harmful Drift Scoring · pp. 4–5

Factorized Class-Domain Residual Adapter

  • A low-rank adapter forms each residual from a global class factor, a client-private domain factor, and a global basis.The factorization shares how classes use the residual basis while preserving each client's domain-specific activation.Source: Factorized Class-Domain Residual Adapter · p. 5

Image-Conditioned Candidate-Wise Correction

  • An image-conditioned gate controls each candidate class's correction, which subtracts the undesirable residual and normalizes the resulting feature.The gate begins near identity to preserve the pretrained representation early in training.Source: Image-Conditioned Candidate-Wise… · p. 5

Harmful-Pair-Aware Federated Optimization

  • Local optimization upweights samples from the harmful set while retaining cross-entropy learning for other data, and the server aggregates shared but not private parameters.The local objective also regularizes the class factors, private domain factors, global basis, and image gate.Source: Harmful-Pair-Aware Federated… · p. 5

Experiments

MOSAIC improves macro-client accuracy across the evaluated datasets and client-partition protocols, while its component ablations support selective class-domain correction. The experiments cover domain shift, joint domain and label shift, and parameter sensitivity.

Experimental Setup

  • The evaluation uses Office31, OfficeHome, and DomainNet100 with one-client-per-domain and two-clients-per-domain partitions, including Dirichlet label imbalance in Setting 2O.Results are macro-client top-1 accuracy, averaged over three seeds; the selected round maximizes macro-client test accuracy.Source: Experimental Setup · pp. 5–6

Comparisons under Domain and Label Shifts

Component Ablation and Parameter Sensitivity

Conclusion

MOSAIC targets class-conditioned domain drift in personalized federated vision-language tuning through selective correction of harmful class-domain residuals. The paper reports that its main results and component ablations support this targeted correction.

Conclusion

  • MOSAIC separates the common domain effect from decision-harmful class-domain residuals, using shared class structure and private domain activation for selective correction.The conclusion frames this as personalized federated VLM tuning under class-conditioned domain drift.Source: Conclusion · p. 7
  • The authors state that MOSAIC supports federated VLM adaptation across heterogeneous institutions without sharing raw images.They identify scaling harmful-set construction and protecting exchanged statistics as future work.Source: Conclusion · p. 7
  • Future work will extend the approach to prototype learning and develop benchmarks with explicitly curated harmful sets.The stated aim is systematic evaluation of how such interactions affect existing prototype-based methods.Source: Conclusion · p. 7

Supplementary Material

The supplementary material consolidates evidence for the motivating phenomenon, harmfulness score, and MOSAIC method, then provides implementation, analysis, and detailed-result material.

A Guide to the Supplementary Material

  • Sections B–D cover datasets and protocols, comparison methods, and MOSAIC's parameter, communication, computational, and privacy considerations.Section B also describes handling missing class-domain pairs and dataset-specific hyperparameters.Source: A Guide to the Supplementary Material · p. 10
  • Sections E–F extend evidence for class-conditioned drift and analyze the training-only harmfulness diagnostic, including its counterfactual audit and quantile trends.The guide says these analyses motivate the two-level selective-correction design.Source: A Guide to the Supplementary Material · p. 10
  • Section G reports complete domain- and class-level results, discusses domain-specific regressions, and clarifies the scope of the empirical claims.Source: A Guide to the Supplementary Material · p. 10

B Detailed Experimental Setup

The experiments use shared evaluation protocols across three benchmarks and compare adaptation methods that differ in where and how they personalize. The section also details MOSAIC’s communication costs and qualitative evidence of class-conditioned domain drift across benchmarks.

B Detailed Experimental Setup

  • MOSAIC is evaluated on Office31, OfficeHome, and DomainNet100 under single-client-per-domain and multiclient-per-domain protocols, with macro-client accuracy averaged across three seeds.The multiclient protocol includes domain-only partitions and Dirichlet label imbalance; all compared methods use the same frozen CLIP backbone and client partitions.Source: B Detailed Experimental Setup · p. 10

C Detailed Comparison Baselines

  • Among the compared methods, only MOSAIC combines private domain state with decision-aware, image–candidate correction, beyond global prompts or client-level personalization.The baselines span zero-shot CLIP, federated prompts, proximal stabilization, global–local prompt interaction, and prompt personalization.Source: C Detailed Comparison Baselines · p. 11

D Parameter, Communication, and Computational Cost

  • The domain factor is never uploaded, while residual parameters are exchanged each round and class-feature sums and counts are exchanged once for harmful-set construction.The reported payload excludes raw images; the experiments do not implement secure aggregation and do not claim the exchanged statistics are free of privacy risk.Source: D Parameter, Communication, and… · pp. 11–12

E Additional Qualitative Evidence of Class-Conditioned Domain Drift

  • Across Office31, OfficeHome, and DomainNet100, qualitative examples show that domain shifts alter different classes through different visual cues.Examples include changes to mug material and chair geometry in Office31, refrigerator outlines and drill contours in OfficeHome, and distinct cues for couches and chandeliers in DomainNet100.Source: E Additional Qualitative Evidence of… · p. 12

F Training-Only Diagnosis of Harmful Class-Domain Residuals

The diagnosis estimates class-domain residuals from training data and keeps held-out test images out of residual estimation, scoring, and harmful-set construction. It examines whether a domain’s displacement varies by class beyond a common domain effect.

F Training-Only Diagnosis of Harmful Class-Domain Residuals

  • Residuals, harmfulness scores, and the harmful set are estimated exclusively from deterministic training subsets; held-out test data measures outcomes only.The test split does not enter residual estimation, scoring, or harmful-set construction.Source: F Training-Only Diagnosis of Harmful… · p. 12
  • Figure 7 arranges classes in rows and domains in columns to show that the same domain changes different classes through different visual cues.The figure provides qualitative evidence of class-conditioned domain drift.Source: Figure 7 · p. 13

F.1 Leave-One-Domain-Out Residual

  • The leave-one-class-out residual removes the domain’s average displacement of other classes to isolate the interaction specific to a class.The analysis uses at most 80 deterministic training images per pair for Office31 and OfficeHome and 100 for DomainNet100, yielding 93, 260, and 600 class-domain pairs, respectively.Source: §F.1 · p. 13

F.2 Decision-Aware Score and Held-Out Counterfactual Audit

The audit tests whether the decision-aware score tracks held-out margin gains from removing training-derived residuals, while keeping the test data out of harmful-set construction. The score tracks this margin response more directly than residual norm, but its association with held-out error is not universal.

F.2 Decision-Aware Score and Held-Out Counterfactual Audit

  • The harmfulness score’s rank correlation with oracle margin response rises from 0.724 on Office31 to 0.897 on DomainNet100, while residual norm has a weaker, dataset-dependent relationship.The audit is a geometric consistency check; the oracle uses the ground-truth class only as a diagnostic, not for MOSAIC inference or accuracy reporting.Source: §F.2 · pp. 13–14
  • The score’s association with held-out error is weaker and less stable on small, low-error benchmarks, so it is not a universal causal error predictor.Spearman correlations are −0.144, 0.364, and 0.619 on Office31, OfficeHome, and DomainNet100, respectively; indiscriminate residual removal decreases aggregate margin on all three datasets.Source: §F.2 · p. 14

F.3 Construction and Fixed Use of the Harmful Set

  • The harmful set is selected offline from training data and remains fixed during federated training; held-out test labels are reserved for audits and correlation analyses.Selection takes the top-q class-domain pairs independently within each domain, using the round-49 FedDDA diagnostic space for reported runs.Source: §F.3 · p. 14
  • The harmfulness score remains associated with held-out counterfactual margin gain in frozen CLIP, although top-20% harmful-set overlap with FedDDA is only 9, 27, and 69 pairs.The results indicate that adverse class-conditioned geometry is measurable before adaptation, while exact pair membership is partly representation-dependent.Source: §F.3 · p. 14, Table 7 · p. 14

F.4 Harmfulness-Quantile Trends and Cases

Within-domain harmfulness quintiles show that the response to residual removal varies with score, but high scores do not ensure a top-1 flip for every pair. Examples also show variation within the highest quintile, motivating image-conditioned rather than constant correction.

F.4 Harmfulness-Quantile Trends and Cases

  • In the highest harmfulness quintile, removal improves both average margin and top-1 accuracy on DomainNet100, while Office31 mainly shows a margin benefit.Across quintiles, removal transitions from harmful in lower groups to useful in Q5; a high score does not guarantee a top-1 flip for every pair.Source: §F.4 · p. 14
  • Within DomainNet100 Q5, Quickdraw–aircraftcarrier gains +8.15 margin and +86.0 accuracy points, while real–blackberry loses −3.67 margin and −17.5 accuracy points.The blackberry category includes brand-related imagery as well as fruit appearance.Source: §F.4 · p. 14
  • Within-domain quantile selection can admit near-zero-score pairs, so MOSAIC uses an image-conditioned candidate gate and harmful-pair reweighting to regulate correction.Pair-level selection proposes where correction may be needed; the gate and reweighting determine whether and how strongly it acts on each image.Source: §F.4 · p. 15

G Complete Domain- and Class-Level Results

MOSAIC improves macro-client averages across both client-partition settings, while domain-level results show uneven gains and terminal-checkpoint tables report class-level accuracy across all three datasets.

G Complete Domain- and Class-Level Results

  • MOSAIC's Setting 1 gains cluster on OfficeHome Clipart, Art, and Product (+2.03, +1.53, +1.08 points) and DomainNet100 Clipart and Infograph (+0.56, +1.46 points).OfficeHome Real World decreases by 0.12 points, while other DomainNet100 changes are within 0.37 points.Source: G Complete Domain- and Class-Level… · p. 15
  • Setting 2 is not uniformly superior: DSLR falls 1.20 points and DomainNet100 Quickdraw 2.31, despite gains on Webcam (+3.04) and Infograph (+2.23).The reported regressions bound the conclusion to a better macro-client trade-off, not dominance on every client domain.Source: G Complete Domain- and Class-Level… · p. 15
  • Terminal checkpoints at rounds 79, 59, and 79 report MOSAIC accuracy across 93, 260, and 600 class-domain pairs for Office31, OfficeHome, and DomainNet100, respectively.The class-level tables include held-out sample counts.Source: G Complete Domain- and Class-Level… · p. 15, Table 10 · p. 18, Table 11 · p. 18
  • The domain profiles place MOSAIC outermost on every Office31 domain and three of four OfficeHome domains; on DomainNet100, its contour largely overlaps FedDDA and FedPGP.DomainNet100 improvements on Clipart, Infograph, and Sketch are balanced by small regressions on Painting, Quickdraw, and Real.Source: G Complete Domain- and Class-Level… · p. 15
  • In domain-balanced quintiles, higher harmfulness generally corresponds to a more favorable response to oracle residual removal, while low-score pairs are frequently damaged by indiscriminate correction.Figure 8 compares training-derived harmfulness with held-out counterfactual margin gain for each class-domain pair.Source: Figure 9 · p. 16, Figure 8 · p. 16
  • MOSAIC raises macro-client accuracy by 1.96 points on Office31 in Setting 1 and by 1.15, 0.27, and 0.12 points on Office31, OfficeHome, and DomainNet100 in Setting 2.Table 9 defines Average as macro-client accuracy and Δ as MOSAIC minus FedDDA.Source: G Complete Domain- and Class-Level… · p. 15, Table 9 · p. 17

Related papers

Loading 2610.01625v1…