PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Decentralized Federated Distillation via Multi-Modality Knowledge Collaboration

Xiao Ma, Hong Shen, Hui Tian, Wei Ke, Wenqi Lyu

arXiv:2609.07230v1cs.LGcs.DC

TL;DR

Robust collaboration remains unresolved when heterogeneous clients observe non-IID data and Byzantine peers can manipulate receiver-specific predictions. Clients compare shared-public-data predictions across three modalities, filter and weight unreliable peers, and validate the resulting distillation gradients with private supervised gradients. Across all evaluated attacks, Ours achieves the highest worst-client accuracy on CIFAR-10 and CIFAR-100; averaged over the five conditions, mean accuracy improves by 0.82 percentage points over FedDF.

  • Problem

    Robust collaboration remains unresolved when heterogeneous clients observe non-IID data and Byzantine peers can manipulate receiver-specific predictions.

    Source: §I · pp. 1–2

  • Method

    Clients compare shared-public-data predictions across three modalities, filter and weight unreliable peers, and validate the resulting distillation gradients with private supervised gradients.

    Source: §I · p. 2

  • Results

    Across all evaluated attacks, Ours achieves the highest worst-client accuracy on CIFAR-10 and CIFAR-100; averaged over the five conditions, mean accuracy improves by 0.82 percentage points over FedDF.

    Source: 1) CIFAR-10 Results · p. 12, §VII · p. 14

  • Takeaways & Limitations

    Experiments support robust collaboration among heterogeneous clients under non-IID data and multiple Byzantine attacks, including receiver-specific malicious predictions.

    Source: §VII · pp. 14–15

Abstract

from arXiv · show

This paper propose a robust decentralized federated distillation method that enables clients with heterogeneous models to collaborate through predictions on shared unlabeled public data. In the proposed method, each client first evaluates the received predictions in three modalities of class prediction, boundary decision, and prediction correlation. It then filters unreliable clients, assigns reliability-based weights to the retained clients, and constructs a teacher for each type of knowledge. Finally, the corresponding distillation gradients are validated using a supervised gradient computed from private data. Conflicting prediction and boundary gradients are removed, and conflicting relation gradients are suppressed before the final model update. We prove the convergence of the proposed method by showing stable local optimization for honest clients under Byzantine distillation. Particularly, we show that our method ensures a bounded Byzantine influence on both distillation gradients and individual client private gradients after cross-modality fusion, thereby enabling stable local optimization for honest clienunder Byzantine distillation. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed method improves the prediction accuracy of heterogeneous models of clients under non-IID data and Byzantine attacks. As the booming demands of federated learning in decentralized environments such as edge computing and mission-oriented UAV collaborations, our method has a great potential for adoption of DFL in unreliable real-world scenarios where clients are exposed to receiver-specific Byzantine messages of malicious predictions.

I. INTRODUCTION

The paper addresses Byzantine prediction manipulation in decentralized federated learning, where heterogeneous models cannot reliably collaborate through parameter aggregation. It exchanges public-data predictions and combines modality-specific robust distillation with private-gradient validation.

A. Decentralized Federated Learning

  • DFL removes the central server and distributes communication across a graph, but many methods still exchange parameters or gradients and assume compatible models.Output-space distillation instead supports collaboration between heterogeneous models through predictions on shared public data.Source: §I · p. 1, A. Decentralized Federated Learning · p. 2

I. INTRODUCTION

  • Heterogeneous architectures make parameter averaging undefined and parameter-space distance unreliable, while Byzantine peers can send receiver-specific predictions.Existing robust DFL and heterogeneous distillation methods address parts of this combined problem, but not receiver-specific manipulation in serverless networks.Source: §I · pp. 1–2

B. Robust Decentralized Federated Learning

  • Prior robustness methods screen model vectors, while output-space defenses are mainly server-coordinated or inference-oriented rather than receiver-specific serverless distillation.These approaches do not directly address clients receiving inconsistent Byzantine messages during iterative decentralized collaboration.Source: §I · p. 2, B. Robust Decentralized Federated… · p. 3

III. PROBLEM STATEMENT

  • The setting assumes clients share a C-class output space and public unlabeled data, while Byzantine clients may send arbitrary receiver-specific logits.Honest clients train locally before public-data distillation; Byzantine messages can differ across receivers.Source: §III · p. 3

A. Framework Overview

  • Each client exchanges standardized public-data logits, evaluates them across class, boundary, and correlation modalities, filters and weights peers, then validates and fuses distillation gradients.Conflicting prediction and boundary gradients are removed, while conflicting relation gradients are suppressed.Source: A. Framework Overview · pp. 3–4

B. Logit Exchange and Standardization

  • Logits are centered and rescaled across classes to reduce architecture-dependent offset and scale differences while preserving relative class ordering.The standardized outputs are made directly comparable across heterogeneous models.Source: B. Logit Exchange and Standardization · p. 4

C. Robust Distillation within Each Modality

Clients compare received logits against a median reference in three knowledge modalities. They filter and weight peers separately by modality before aggregating retained knowledge into modality-specific teachers.

1) Multi-Modality Logit Comparison :

2) Robust Neighbor Filtering and Weighting :

  • For each modality, clients filter peers whose discrepancies exceed an adaptive median-based threshold, then assign reliability weights to retained peers.The threshold uses the median absolute deviation and adapts to the current discrepancy distribution; more reliable retained peers receive larger aggregation weights.Source: 2) Robust Neighbor Filtering and… · pp. 4–5

C. Robust Distillation within Each Modality

  • Each modality aggregates retained client knowledge into its own teacher and produces a corresponding distillation gradient.Class-prediction, boundary-decision, and prediction-correlation teachers use their respective knowledge representations.Source: 3) Modality-Specific Teacher… · pp. 5–6

1) Adaptive Modality Weighting :

Modality weights adapt to the estimated reliability of each modality's retained clients.

1) Adaptive Modality Weighting :

  • A modality with more reliable-than-average retained clients receives a larger weight in the final multi-modality update.Client weights determine contributions within a modality, whereas modality weights determine each modality's contribution to the final update.Source: 1) Adaptive Modality Weighting · p. 6

2) Private-Gradient Validation and Model Update :

Private supervised gradients validate external distillation directions before they are combined for the public update. The algorithm first controls unreliable knowledge within modalities, then controls interactions among modality gradients.

2) Private-Gradient Validation and Model Update :

  • The private supervised gradient removes conflicting prediction and boundary components and retains the correlation gradient only when it agrees with the supervised direction.This gradient validates external directions but does not add a supervised update during the public phase.Source: 2) Private-Gradient Validation and… · p. 6

E. Algorithm Description

  • The algorithm standardizes exchanged logits, constructs modality-specific teachers from filtered and weighted peers, then adaptively weights, validates, and combines their gradients.The procedure controls unreliable knowledge within each modality and then controls interactions among the resulting gradients through private-gradient validation and adaptive fusion.Source: E. Algorithm Description · p. 7, 2) Private-Gradient Validation and… · p. 7

V. THEORETICAL ANALYSIS

The analysis studies whether each honest client’s own supervised objective remains stable under Byzantine public distillation, rather than seeking convergence to a common model. Under stated regularity and gradient assumptions, it bounds the average gradient norm and shows convergence to stationarity under a specified step-size choice.

V. THEORETICAL ANALYSIS

  • The analysis targets each honest client’s local supervised objective because heterogeneous architectures make convergence to a common model parameter ill-defined.It bounds residual Byzantine influence on distillation gradients and analyzes private-gradient validation before establishing bounded average stationarity.Source: §V · p. 7

A. Assumptions

B. Main Result

  • For every honest client, the average squared gradient norm of its local supervised objective remains bounded under Byzantine public distillation.The first bound term decreases as O(1/T); remaining terms capture local-update drift, stochastic gradients, validation error, and public distillation.Source: B. Main Result · p. 8
  • With γ = T^-1/2 and α ≤ T^-1, a uniformly sampled local iterate converges to stationarity in expectation.The step-size choice yields the stated corollary under the theorem’s conditions.Source: B. Main Result · p. 8
  • The proof combines local-SGD descent with bounds on Byzantine-affected distillation gradients and the validated public-distillation updates.The public update uses the validated multi-modality gradient; the private supervised gradient validates public directions rather than adding another model update.Source: §V · p. 7, Proof sketch · p. 8

C. Bounded Residual Byzantine Distillation

This section bounds the residual Byzantine effect on an honest client’s modality-wise distillation gradients. The bound is controlled by the total retained weight assigned to Byzantine sources.

C. Bounded Residual Byzantine Distillation

  • The difference between an actual modality-wise distillation gradient and the gradient induced by retained honest sources is bounded, so retained Byzantine influence is controlled by its total retained weight.The section identifies the residual Byzantine weight among retained sources as the factor controlling this effect.Source: C. Bounded Residual Byzantine… · pp. 8–9
  • After decomposing retained-source representations into honest and Byzantine components, the section bounds the resulting multi-modality distillation gradients.The resulting bound contributes to the gradient bound used in the main convergence analysis.Source: C. Bounded Residual Byzantine… · pp. 8–9

D. Bound of Private-Gradient Validation

Private-gradient validation limits the size and adverse effect of public distillation gradients. Prediction and boundary conflicts are projected away, while relation gradients are retained only when aligned with the private supervised direction.

D. Bound of Private-Gradient Validation

  • The validated multi-modality gradient remains bounded by G_i, and its possible adverse effect on the local objective is bounded by δ_iG_i.This safety result is used to control the public-distillation contribution in the convergence analysis.Source: D. Bound of Private-Gradient Validation · pp. 9–10
  • Validation does not increase modality-wise gradient norms, and prediction and boundary gradients have their components opposing the private supervised gradient removed.The relation gradient is retained only when it has a positive inner product with the private supervised gradient.Source: D. Bound of Private-Gradient Validation · p. 10
  • Because the private supervised gradient is stochastic, the analysis bounds the possible adverse effect of a validated public-distillation direction by δ_iG_i.The bound accounts for the estimation error between the stochastic private gradient and the true local gradient.Source: D. Bound of Private-Gradient Validation · pp. 9–10

E. Proof of the Main Result

The proof of the main result combines local supervised-training descent with bounds on the public-distillation stage. Together, these yield the theorem’s bound on each honest client’s local objective.

E. Proof of the Main Result

  • The proof combines local-SGD descent with bounds on Byzantine-affected distillation gradients and validated public-distillation updates.The resulting bound is summed over training rounds to obtain the theorem’s average-stationarity bound.Source: E. Proof of the Main Result · pp. 10–11
  • The local-training analysis supplies the local-update drift and stochastic-gradient terms in the bound.The local-SGD result is applied under Assumptions 1–2.Source: E. Proof of the Main Result · p. 10
  • Lemmas 3 and 4 control the public-distillation terms using bounds on the multi-modality gradients and their effect on the local objective.The proof applies these bounds successively across distillation steps and telescopes the resulting inequality across rounds.Source: E. Proof of the Main Result · pp. 10–11

A. Experimental Setup

The experiments evaluate heterogeneous client models on CIFAR-10 and CIFAR-100 under non-IID data and Byzantine attacks, using shared public data for distillation. The setup specifies decentralized participation, training and attack conditions, evaluation metrics, and public-distillation baselines.

1) Datasets:

  • CIFAR-10 and CIFAR-100 experiments use 10% of training data as unlabeled public data, 5% for validation, and the remainder as clients’ private labeled data.Public labels are not used in teacher construction or distillation; the test set is reserved for final evaluation.Source: 1) Datasets · p. 11

2) Federated Setting:

  • Ten clients operate in a fully connected decentralized system, with three Byzantine clients and honest clients indexed H = {0, 1, 2, 3, 4, 5, 6}.No central server aggregates models, gradients, or predictions; a coordinator only schedules computation and routes messages.Source: 2) Federated Setting · p. 11

6) Evaluation Metrics:

  • Reported test metrics include only honest clients; Accmean measures their overall performance, while Accworst measures the most affected honest client.Accworst is particularly useful when Byzantine clients send different messages to different receivers.Source: 2) Federated Setting · p. 11, 6) Evaluation Metrics · p. 12

3) Data and Model Heterogeneity:

  • Private data use label-skewed Dirichlet partitioning with α = 0.5, and clients use CIFAR-compatible variants of ResNet-18, MobileNetV2, ShuffleNetV2, and VGG-11.Each client receives at least 100 private training samples; Table I covers heterogeneous model architectures across clients.Source: 3) Data and Model Heterogeneity · p. 11, Table I · p. 11

4) Training and Distillation Settings:

  • Each round includes one private-data training epoch and two public mini-batches of size 256, with three distillation gradients validated by a private supervised gradient.Training runs for 120 rounds with SGD, momentum 0.9, batch size 64, initial learning rate 0.05, and weight decay 5 × 10−4; the learning rate drops at 50% and 75% of training.Source: 4) Training and Distillation Settings · pp. 11–12, Table II · p. 12

5) Byzantine Attacks:

  • The four attacks are Gaussian message replacement, sign reversal, target-class preference amplification, and fixed class-wise bias.For output-space methods, attacks target exchanged logits, and Byzantine clients may send different manipulated logits to different honest clients.Source: 5) Byzantine Attacks · p. 12

7) Compared Methods:

  • The primary comparison uses FedMD, FedDF, and the proposed method under matched heterogeneous model assignments, private-data partitions, public batches, schedules, and logit-space attacks.FedMD averages source logits, while FedDF averages source probabilities for public ensemble distillation.Source: 7) Compared Methods · p. 12

B. Main Comparison under Heterogeneous Models

The method combines three prediction-based knowledge modalities with private-gradient validation to limit Byzantine influence in decentralized distillation. Experiments report accuracy gains for heterogeneous clients across CIFAR-10 and CIFAR-100.

1) CIFAR-10 Results:

  • Ours has the highest worst-client accuracy in all five CIFAR-10 conditions, averaging 41.85%, 4.69 points above FedDF.Its mean accuracy averages 0.82 percentage points above FedDF.Source: Table III · p. 12, 1) CIFAR-10 Results · p. 12

2) CIFAR-100 Results:

  • Ours achieves the highest mean and worst-client accuracy under every evaluated CIFAR-100 attack; across conditions, it improves those metrics by 0.96 and 1.79 points over FedMD.Source: Table IV · p. 12, 2) CIFAR-100 Results · p. 12

D. Client-Level Performance

  • CIFAR-10 client averages vary substantially: C0 reaches 81.77%, versus 43.21% for C1 and 42.08% for C5; CIFAR-100 averages range from 22.65% to 27.18%.C0 and C4 both use ResNet-18 but obtain markedly different accuracies, so architecture alone does not explain the CIFAR-10 gap.Source: Table V · p. 13, Table VI · p. 13, D. Client-Level Performance · p. 13

C. Robustness across Attack Types

  • On CIFAR-10, Sign-flip reduces Ours mean and worst-client accuracy by 4.78 and 6.33 points; CIFAR-100 attack-related changes range from −1.02 to +0.11 points.The +0.11-point Gaussian difference on CIFAR-100 is within normal single-run variation, not an attack-caused improvement.Source: C. Robustness across Attack Types · pp. 12–13, Figure 1 · p. 13

E. Variant Analysis

  • Against Ours-MM+CA, Ours gains 10.00 points in CIFAR-10 worst-client accuracy without attack, 5.34 attack-averaged, and 2.20 on CIFAR-100 attack-averaged.The CIFAR-10 mean-accuracy gains are 4.18 points without attack and 0.62 attack-averaged; CIFAR-100 attack-averaged mean accuracy rises 0.96 points. The versions are cumulative designs, not one-factor variants.Source: Table VII · p. 13, E. Variant Analysis · p. 14

F. Mechanism Diagnostics

  • Adaptive modality weights stay near the prediction-dominant prior while retaining boundary and correlation contributions; CIFAR-100 has lower pseudo-label coverage and high prediction-projection frequency.The high projection frequency indicates private-gradient validation is active in the no-attack diagnostics.Source: Table VIII · p. 14, F. Mechanism Diagnostics · p. 14

VII. CONCLUSIONS

  • The analysis bounds Byzantine influence on modality distillation gradients and fused private gradients, supporting stable honest-client optimization; experiments report the highest worst-client accuracy across attacks on both datasets.The reported application scope is decentralized learning where clients may receive receiver-specific malicious predictions.Source: §VII · pp. 14–15
Loading 2609.07230v1…