PaperlayerView exact revision on arXiv

Source-linked AI summary

FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learning

Xinyuan Zhao

arXiv:2609.37582v2cs.CVcs.AI

TL;DR

Clients with different modalities, private architectures, and tasks need federated knowledge they can execute through their own inputs and outputs. FedSocket connects them through a shared predictor Q, and across six datasets Joint improves missing-modality performance and exceeds independent ensembles at matched inference capacity.

  • Problem

    Federated learning must make complementary knowledge useful to recipients with different modalities, private architectures, and tasks, using inputs they actually observe and task-valid outputs.

    Source: §1 · p. 1

  • Method

    FedSocket uses Q with recipient-computable modality features, task-owned outputs, ownership-aware aggregation, and bidirectional teaching between private models and local Q copies.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across six datasets, Joint improves missing-modality accuracy over Local by 14.44 pp on MELD and 15.51 pp on UCF-51, and beats matched-capacity ensembles by 11.06 pp in UCF-51 pooled accuracy and 4.87 pp in mean bidirectional Flickr30k R@1.

    Source: §8 · p. 8

  • Takeaways & Limitations

    Recipient-executable exchange connects federated training with recipient inference, with Joint combining exchanged Q predictions and local expertise.

    Source: §1 · p. 1, §8 · p. 8

Abstract

from arXiv · show

Federated knowledge must remain usable by recipients with different modalities, private architectures, and tasks. We present FedSocket, which makes recipient execution a design requirement of the exchanged model. A shared Q combines recipient-computable inputs, task-owned outputs, and ownership-aware aggregation, connecting heterogeneous private models through a common prediction interface. Private models teach local Q copies; the returned Q supports local learning and Joint inference, with only Q parameters and counts exchanged. Across six datasets, FedSocket improves missing-modality recipient accuracy over Local by 14.44 and 15.51 percentage points on MELD and UCF-51. Under matched inference capacity, Joint exceeds independent ensembles by 11.06 points in UCF-51 accuracy and 4.87 points in mean bidirectional Flickr30k R@1. Joint also improves over Q alone on all four heterogeneous endpoints, demonstrating the value of combining local and exchanged predictions. Teacher controls, sharing-path interventions, and component factorials identify the roles of supervision, sharing, and deployment. FedSocket makes exchanged knowledge directly usable from federated training to recipient inference.

1. Introduction

FedSocket makes recipient execution the organizing objective of knowledge exchange across clients with different modalities, architectures, and tasks. Its shared predictor links recipient-computable inputs and task-owned outputs to aggregation and deployment.

1. Introduction

  • Recipients need predictors that use their available modalities, remain compatible with different private encoders, and preserve task-specific outputs and parameter ownership.These requirements apply both to one task with unequal modality access and to different tasks using overlapping modalities.Source: §1 · p. 1
  • FedSocket’s Q reads fixed modality features, keeps task-specific outputs, and uses task-owned components, shared modality embeddings, and a gated residual for cross-task knowledge.The server aggregates blocks according to task or modality ownership, linking recipient execution to federation updates.Source: §1 · p. 1
  • Private models teach local Q copies, while returned Q supports private learning and Joint inference alongside clients’ native architectures.Clients exchange Q parameters and counts.Source: §1 · pp. 1–2
  • At matched inference capacity, Joint exceeds independent ensembles by 11.06 pp in UCF-51 pooled accuracy and 4.87 pp in mean bidirectional Flickr30k R@1.Across six datasets, Joint also improves missing-modality recipient accuracy over Local by 14.44 pp on MELD and 15.51 pp on UCF-51.Source: §1 · p. 2

2. Related Work

Related work studies shared-model federation, multimodal collaboration, cross-task representation exchange, and transfer objectives. FedSocket focuses on structuring and aggregating an exchanged predictor for recipients with coupled modality, architecture, and task differences.

2. Related Work

  • FedAvg and FedProx aggregate or regularize shared models, while distillation, public-example prediction exchange, ensembles, and prototypes transfer knowledge across model types [11] [12] [4].These exchanged predictions or representations impose requirements on client inputs and output spaces.Source: §2 · pp. 2–3
  • FML uses mutual learning and partial-model sharing [19], FedKD uses adaptive distillation and compressed updates, and MH-pFLID uses a public-data-free messenger for heterogeneous architectures.FedSocket instead ties Q’s input interface and task outputs to parameter ownership for both local learning and recipient deployment.Source: §2 · p. 3
  • Multimodal methods address heterogeneous sensors, modality bias, incomplete inputs, or prompt alignment [30]; FedSocket targets shared predictor execution across private architectures and tasks.Its task-owned heads preserve prediction objectives while shared modality embeddings and a residual carry cross-task knowledge.Source: §2 · p. 3
  • Cross-task methods use public image–text representations or combine distillation with task-conflict handling, while other approaches address modality dominance or combine alignment, fusion, and distillation.Contrastive distillation and cross-modal KD support transfer, while OGM and PMR address imbalanced multimodal optimization.Source: §2 · p. 3

3. Recipient-Executable Knowledge Exchange

Recipient-executable exchange makes the shared predictor Q runnable through each client’s available modalities and task output space. The design connects locally computable inputs and task-valid outputs to ownership-aware aggregation.

3. Recipient-Executable Knowledge Exchange

  • Clients differ in modality access, private architecture, and task output space, while Q serves participating clients through their available modalities and task outputs.Private models retain their own parameters and native inputs.Source: §3 · p. 3
  • Within one task, complete-modality donors teach Q while recipients execute it through observed modalities; across tasks, clients share compatible Q components and retain task-specific outputs.In cross-task collaboration, every client receives global Q and teaches a local Q copy.Source: §3 · p. 3
  • Fixed modality mappings provide Q with common input coordinates independent of private encoder states, and task and modality identifiers accompany the features.Each task supervises its own classification or retrieval outputs, while the server aggregates blocks according to task or modality ownership.Source: §3 · p. 3
  • Task-owned adapters, experts, and heads specialize within tasks, while shared modality embeddings and residuals carry cross-task updates.For example, classification and retrieval clients can share image embeddings and residual modules while keeping separate prediction heads.Source: §3 · p. 4

4. FedSocket: Learning and Deploying Q

FedSocket trains Q through fixed recipient-computable features, bidirectional teaching between Q and private models, and aggregation aligned with parameter ownership. At inference, recipients can use Q alone or combine it with their private model.

4.1. Recipient-Computable Q and Task-Owned Outputs

  • Fixed modality features feed task- and modality-specific adapters and embeddings; task experts and heads specialize outputs, while a gated shared residual carries cross-task information.For missing-modality transfer, complete-input donor predictions supervise Q using observed-modality features; the conditional target averages predictions for examples with the same observed features.Source: §4.1 · p. 4

4.2. Bidirectional Local Knowledge Transfer

  • Each round’s frozen Teacher Q first supervises Private, then the updated Private teaches a trainable Student Q through detached predictions.Label-smoothed cross-entropy, reliability-weighted distillation, and label-supervised fusion shape the objectives; retrieval applies these roles to score distributions and multi-positive matching.Source: §4.2 · p. 4

4.3. Compatible Server Aggregation

  • The server averages task-owned blocks within their tasks, modality embeddings across tasks using that modality, and shared residual updates across tasks.Classification-head rows use clients with positive counts for that class; PCGrad-style projection removes residual-update components opposed to another task’s update.Source: §4.3 · p. 4

4.4. Reusing Q at Recipient Inference

  • Recipients can run Private, Q, or Joint through the same observed inputs, with development data selecting inference weights.For task-heterogeneous classification, Q receives greater weight for classes with fewer local examples and full weight for absent classes; UCF and retrieval use their stated development-selected mixture weights.Source: §4.4 · p. 5

5. Experimental Design

The experiments evaluate recipient-executable exchange across six datasets, using protocol-specific comparisons and controls to test recipient gains, sharing, and deployment. Checkpoint and fusion selection, evaluation groups, and aggregation rules are specified to make these comparisons consistent.

5.1. Protocol Instances

  • Six datasets test complete-modality supervision for missing-audio recipients and task-structured exchange across classification, retrieval, and heterogeneous private architectures.Same-task clients use sample-weighted Q averaging; heterogeneous tasks separately aggregate task-owned blocks and shared residuals.Source: §5.1 · p. 5

5.2. Statistical and Selection Protocol

  • Primary results use five paired training seeds; reported means and sample standard deviations are computed over five seeds on fixed primary partitions.Paired effects include unadjusted 95% Student-t confidence intervals and exact two-sided sign-flip p-values.Source: §5.1 · p. 5, §5.2 · p. 5
  • Table 1 summarizes dataset interfaces, including modality abbreviations and donor-to-recipient access, while full protocols are referenced in Sec. D.Flickr30k uses five-fold 200-image evaluation.Source: Table 1 · p. 5, §5.1 · p. 5
  • DEV selects checkpoints and fusion coefficients before TEST, with Private, Q, and Joint evaluated on the same test examples within each run.All, Recipient, and Backbone group denote pooled clients, missing-modality recipients, and clients with a specified private architecture, respectively.Source: §5.2 · p. 5

5.3. Baselines, Controls, and Ablations

  • Controls compare knowledge sources, sharing, training, deployment, and alternative shared-model designs under matched parameter budgets.Task-isolated Q disables cross-task exchange; No return disables Q→Private KD and fusion training, while independent predictors and checkpoint ensembles test matched-capacity deployment.Source: §5.3 · pp. 5–6

6. Results

Joint prediction improves over both Private and Q across heterogeneous classification and retrieval endpoints, while FedSocket also strengthens recipient prediction when audio is missing. Controls show benefits from complete-modality teacher supervision, matched-capacity deployment with Q, cross-task sharing, and combining Q supervision with fusion.

6.1. Recipient Gains with Missing Modalities

  • Joint accuracy reaches 68.24% on MELD and 86.83% on UCF-51, exceeding Local by 14.44 ± 6.59 pp and 15.51 ± 1.35 pp, respectively.On CREMA-D, recipient Joint accuracy is 75.24%, the highest mean in Tab. 2.Source: §6.1 · p. 6, Figure 2 · p. 6
  • Complete-modality teacher supervision raises recipient Joint accuracy over Label Q by 1.27 ± 0.84 pp on MELD, 0.76 ± 0.76 pp on CREMA-D, and 2.22 ± 0.35 pp on UCF-51.Recipient inputs remain fixed; the paired differences are positive in all MELD and UCF seeds and four CREMA-D seeds.Source: §6.1 · p. 6

6.2. Joint Prediction Across Classification and Retrieval

  • Joint improves over both Private and Q on all six classification and retrieval metrics, with gains over Q of 3.27 pp on CIFAR, 2.65 pp on AG, and 4.50±0.41 pp/4.33±0.82 pp on Flickr I2T/T2I.Gains over Q are positive in every seed, and Joint leads at all reported retrieval ranks.Source: §6.2 · p. 6, Table 3 · p. 6
  • FedSocket leads FML-adapted, FedKD-adapted, and Naive Shared on all four mean Joint metrics; its gains over Naive Shared are 14.24/8.36 pp on CIFAR/AG and 1.29/0.98 pp on Flickr I2T/T2I.The shared-model controls have matched parameter counts and use the same partitions, private backbones, recipient interfaces, rounds, and Joint selection rules.Source: §6.2 · p. 6, Table 4 · p. 7

6.3. Deployment Gains at Matched Inference Capacity

  • At matched inference capacity, Joint exceeds independent ensembles by 11.06 pp in UCF pooled accuracy and 4.87 pp in mean bidirectional Flickr30k R@1.The UCF systems each use 0.60M parameters; Flickr Joint and its independent Q-trained Private ensemble each use 36.15M parameters.Source: §6.3 · p. 7, Table 5 · p. 7

6.4. Cross-Task Sharing and Bidirectional Teaching

  • Full Q exceeds task-isolated Q by 1.42±0.40 pp on CIFAR and 2.06 ± 0.13 pp/1.34 ± 0.37 pp on Flickr I2T/T2I, with positive differences in all five pairs.The control retains within-task training and aggregation, isolating the overall benefit of cross-task exchange.Source: §6.4 · p. 7, Figure 4 · p. 8
  • Returning aggregated knowledge to Private improves it over No return by 6.58 pp on CIFAR and 16.87 pp/15.29 pp on Flickr I2T/T2I.Writing local knowledge into Q also improves Q over Label Q, while Private improves over Label Q across the reported tasks.Source: §6.4 · p. 7

6.5. Interactions That Strengthen Joint Prediction

  • KD and fusion have positive average effects on Private and Joint metrics, respectively, and their Joint interactions are positive across all four metrics.On AG, fusion adds 0.88 pp without KD and 4.36 pp with KD, a 3.48 pp interaction; learning from Q increases the deployment benefit of training Private and Q to predict together.Source: §6.5 · pp. 7–8, Table 6 · p. 8

7. Analysis and Discussion

Recipient compatibility can make teacher correctness an insufficient guide to supervision. Development results also examine whether Q complements local predictions and whether Joint gains persist across private architectures.

7. Analysis and Discussion

  • When Private assigns greater true-class probability than Q, label and distillation gradients can oppose each other.Figure 5 plots Private-minus-Q true-class probability against gradient cosine for 6,096 correct-teacher CREMA-D TRAIN observations; negative cosine indicates opposing output-loss directions.Source: §7 · p. 8, Figure 5 · p. 8
  • Conditional Q yields 76.60 ± 0.42% mean Joint accuracy, versus 76.28% for Private and 69.44% for Q.The endpoint audit uses 430 examples held from previously observed DEV.Source: §7 · p. 8
  • Joint improves mean macro F1 and UAR across all three UCF backbone groups; Transformer gains reach 10.92 pp and 10.68 pp.MLP and CNN groups gain 15.02 pp/18.57 pp accuracy, positive in every seed, while retaining their private architectures.Source: §7 · p. 8

8. Conclusion

FedSocket connects heterogeneous private models through Q by jointly designing recipient-computable inputs, task-owned outputs, and compatible aggregation. Its architecture links task-specific prediction components with cross-task sharing, while its recipient-interface projection characterizes what a recipient can express.

8. Conclusion

  • Across six datasets, Joint improves missing-modality recipient accuracy over Local by 14.44 pp on MELD and 15.51 pp on UCF-51.FedSocket also leads FML/FedKD adaptations and Naive Shared on all four primary classification and retrieval metrics, and beats independent ensembles at matched inference capacity on two reported comparisons.Source: §8 · p. 8

A.1. Recipient-Conditioned Task Projection

  • For a fixed within-task teacher, Equation (2) defines the recipient-interface prediction minimizing unrestricted forward-KL risk by averaging teacher predictions over examples with the same interface value.FedSocket approximates this projection using teacher predictions, labels, and reliability weights.Source: §A.1 · p. 10

A.2. Interfaces and Private Models

  • Q reads fixed local features independently of private encoders, while private models retain native architectures suited to their tasks and resources.Inputs include precomputed text or visual features for same-task experiments and fixed 256-dimensional features for task-heterogeneous experiments.Source: §A.2 · p. 10

A.3. FedSocket Architecture

  • Task and modality embeddings augment each adapted input, then a task expert combines with a gated low-rank shared residual.Classification uses separate linear heads; Flickr30k uses a shared 256-dimensional normalized retrieval head, and the server shares the residual and modality embeddings across compatible tasks.Source: §A.3 · p. 10

A.4. Stage I: Private Warm-Up and Same-Task Cross-Fitting

The appendix describes a three-stage exchange in which private models and Q teach one another, with ownership-aware aggregation and development-selected inference. It also gives a population interpretation of the recipient-interface target and its label-anchored variant.

A.4. Stage I: Private Warm-Up and Same-Task Cross-Fitting

  • Same-task teachers produce out-of-fold logits from full-modality models that exclude the held dialogue or video group from target-producing training.The teacher combines an observed-modality public prior with a bounded private correction; cached logits remain local.Source: §A.4 · p. 10

A.5. Stage II: Q as Teacher of the Private Model

  • Q first teaches Private using reliability-weighted distillation and fusion; retrieval additionally combines multi-positive contrastive loss with symmetric KL distillation.The training fusion weight is distinct from Q’s task gate on hidden features.Source: §A.5 · pp. 10–11

A.6. Stage III: Q as Student of the Private Model

  • After Private updates, a local Q copy learns from detached Private outputs; same-task Q training also uses a label anchor and reliable out-of-fold teacher correction.The appendix distinguishes the directions: Q teaches Private in Eqs. (14) and (16), while Private teaches Q in Eqs. (17) and (18).Source: §A.6 · p. 11

A.7. Heterogeneous Server Aggregation

  • Adapters, experts, task embeddings, gates, and heads aggregate within tasks; modality embeddings share across users of that modality, and residual updates merge across tasks.Task-level residual updates use a PCGrad-style projection for conflicting directions; same-task implementation instead uses sample-weighted Q averaging and a server stabilizer.Source: §A.7 · p. 11

A.8. Development Calibration and Inference

  • Development calibration selects inference weights for class-dependent Joint scores and separate retrieval directions; same-task UCF selects a scalar Q weight on DEV and freezes it for TEST.Methods and seeds use the same calibration grid.Source: §A.8 · p. 12

A.9. Training and Deployment Protocol

  • Each round broadcasts Q, updates Private, trains a local Q copy, and aggregates Q blocks by ownership; only Q parameters and permitted counts are communicated.Deployment retains Private, Q, or both predictors for Joint inference.Source: §A.9 · p. 12

B. What Related Methods Exchange

  • Table 7 compares closely related methods by inputs, communication, output spaces, and inference options.Its caption defines a sample-indexed upload as logits or representations tied to shared examples, and a recipient branch as a returned model with computable inputs and task-valid outputs.Source: B. What Related Methods Exchange · p. 12, Table 7 · p. 13

C. Population Interpretation of the Conditional Interface

  • The population recipient target is the conditional mean teacher prediction given the recipient interface; under stated alignment conditions, donor projection matches the recipient label conditional.With label anchoring, the teacher contribution is controlled by reliability-weighted mixture weight λ; under label shift, recipient risk depends on mismatch between donor and recipient label conditionals.Source: C. Population Interpretation of the… · pp. 12–13

D. Experimental Protocol Details

The appendix specifies evaluation populations, replication, and controls, then describes supplementary endpoint metrics and the context for published comparisons.

D. Experimental Protocol Details

  • The protocol uses fixed primary client partitions and independent training seeds; paired comparisons enumerate all 2^n sign flips, with minimum two-sided p = 0.0625 for five seeds.Intervals are unadjusted per-contrast summaries without multiplicity correction, and Flickr ensemble comparisons are descriptive.Source: D. Experimental Protocol Details · p. 13

E. Supplementary Endpoint Metrics

  • Table 8 reports same-task recipient accuracy, macro F1, and UAR; Flickr retrieval ranks; and paired recipient accuracy gain decompositions.Values are means ± sample SD across five seeds; mean R is a point estimate averaged from six displayed retrieval recall means.Source: Table 8 · p. 14, E. Supplementary Endpoint Metrics · p. 14

F. Published Reference Context

  • Published results are collected under their cited benchmark conditions; protocol-matched shared-model comparisons appear separately in Tables 4 and 15.Source: F. Published Reference Context · p. 14

G. Paired Contributions and Training Effects

This section reports paired sharing and supervision controls, defines how KD and fusion contributions are separated, and describes deployment-capacity and evaluation comparisons. It also documents implementation and reproducibility details.

G. Paired Contributions and Training Effects

H. Absolute Cross-Task Sharing and Supervision Controls

I. KD and Fusion Effect Definitions

  • Within-seed factorials separate balanced KD and fusion effects from their interaction; examples include 3.26 pp on CIFAR Private and 3.48 pp on AG Joint.On AG Joint, KD increases fusion’s contribution from 0.88 pp to 4.36 pp.Source: §I · p. 15

K. Interpreting Training and Inference Gains

  • Joint-minus-Local performance decomposes into Private’s retained Q-training gain and the additional contribution of keeping Q at inference.The identity is defined for a fixed dataset, seed, population, and metric.Source: K. Interpreting Training and Inference… · p. 16

L. Implementation and Reproducibility / M. Evaluation Aggregation and Release

N. Backbone Groups

  • UCF backbone-group results report accuracy, macro F1, and UAR, with paired Joint-minus-Private accuracy differences within each fixed client group.Table 14 also reports confidence intervals and positive-seed counts for those paired differences.Source: N. Backbone Groups · p. 16, Table 14 · p. 17

O. Direct Comparisons with Shared-Model Neighbors

  • FML-adapted [19], FedKD-adapted, and Naive Shared are compared under a common 40-round protocol; FedKD’s compression reduces upload by 34.45%.Tables 15 and 16 report endpoints and paired Joint contrasts; controls use nearly the same parameter count as Q.Source: O. Direct Comparisons with… · pp. 16–17

J. Capacity Controls and Inference Costs

  • Table 13 pairs deployment scores with inference costs; Flickr ensemble gains are descriptive because ensembles reuse cyclic replicas and omit independent-pair tests.Profiling used an RTX 4090 at batch size 64 and excludes data loading and host–device transfer.Source: J. Capacity Controls and Inference Costs · p. 16, Table 13 · p. 17

P. Recipient-Side Development Diagnostics

Recipient-side development diagnostics examine teacher compatibility, Q checkpoint selection, and the return path from Q to recipient models. The selected Conditional Q candidate has the highest mean Joint accuracy among the reported development candidates.

Recipient-side development diagnostics

  • The diagnostics use 27 multimodal donors and 64 video-only recipients, with 813 historical recipient DEV examples split into selection and audit sets.The audit is a holdback from previously observed DEV data; primary TEST and Set A results retain their original protocols.Source: P. Recipient-Side Development… · p. 17

Selecting useful Q checkpoints

  • Recipient DEV NLL selects Q checkpoints after 100 supervised warm-up rounds and a 200-round branch budget; Figure 7 tracks candidate optimization curves.The curves compare persistent or reset server Adam state and label-only or full-modality-teacher supervision.Source: P. Recipient-Side Development… · pp. 17–18, Figure 7 · p. 19

Teacher information and recipient compatibility

  • Figure 6 separates donor teacher accuracy, recipient DEV predictions, and the share of correct-teacher TRAIN observations with negative CE/KD gradient cosine.The figure’s accuracy comparison is full-modality minus video-only teacher accuracy on 1,395 OOF donor examples.Source: P. Recipient-Side Development… · p. 17, Figure 6 · p. 18

Evaluating Q as a prediction partner

  • The two-stage protocol selects Q first, then trains recipient Private models and selects their Joint rule using DEV accuracy, with NLL breaking ties.KD candidates use all-class or non-target-class KL; only correct-teacher TRAIN examples activate KD, and choices are frozen before audit inference.Source: P. Recipient-Side Development… · p. 18
  • Conditional Q reaches 76.60 ± 0.42% mean Joint accuracy, versus 76.28 ± 0.72% for its Private endpoint and 69.44 ± 0.92% for Q.These are development-candidate results, and the paired matrix reports seed-level Conditional-minus-Label changes.Source: P. Recipient-Side Development… · p. 18

Connecting supervision to the recipient’s prediction

  • Figure 8 relates Private-minus-Q true-class probability to CE/KD gradient cosine and reports DEV-selected KD coefficients and modes.Non-target KL transfers relations among alternative classes; source and seed determine the selected mode and strength.Source: Figure 8 · p. 19, P. Recipient-Side Development… · p. 19

Related papers

Loading 2609.37582v2…