Source-linked AI summary
CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
Baraa Bilbeisi, Mengchen Fan, Baocheng Geng, Qing Tian
TL;DR
Conventional federated learning assumes shared architectures and performs poorly under non-IID data, while existing distillation approaches often require centralized or public transfer infrastructure. CRAD uses decentralized peer distillation with class-wise filtering and precision weighting, and achieves the best global accuracy across CIFAR-10, CIFAR-100, and PathMNIST under heterogeneous architectures and severe non-IID skew.
Problem
The paper asks how to combine peer teacher predictions when reliability differs across teachers and classes and finite local data makes reliability estimates uncertain.
Method
CRAD evaluates peer snapshots on each student’s local data, filters unsupported or consensus-disagreeing teachers per class, and precision-weights the survivors’ class-accuracy estimates.
Results
CRAD attains the best global accuracy across CIFAR-10, CIFAR-100, and PathMNIST under heterogeneous architectures and severe non-IID skew.
Takeaways & Limitations
Decentralized, public-data-free peer distillation can combine heterogeneous models while accounting for class-specific teacher reliability.
Takeaways & Limitations
The evaluation is confined to image classification, and end-to-end differential privacy covering model snapshots remains open.
Abstract
from arXiv · showhide
Conventional federated learning (FL) relies on parameter averaging, which forces clients to be doubly homogeneous: it demands an identical architecture and degrades under non-IID data. Real-world deployments usually break both assumptions. We sidestep both by building a decentralized knowledge distillation framework in which each client evaluates its peers' model snapshots on its own local data and distills from the resulting soft predictions. Because knowledge is transferred through the shared class posterior, clients are free to run different architectures; and because every teacher is evaluated on the student's own device, raw data never leaves the client, with no central server or public dataset required. Within this setting, we identify and address an under-examined problem: how to combine the peer teacher predictions. Existing methods, like uniform averaging, ignore how knowledge reliability varies across teachers and classes. We propose Class-wise Reliability-Aware Distillation (CRAD), which, per class, first discards teachers that disagree with the peer consensus and then takes a weighted average of the rest, weighting each teacher by its per-class reliability (precision, or inverse variance). Since the variance of an accuracy from $n$ samples scales as $1/n$, support enters automatically: among the teachers that survive filtering, a teacher is trusted for a class to the degree that it is both accurate and well-evidenced for it. On three image-classification benchmarks (CIFAR-10, CIFAR-100, and PathMNIST colon pathology), across heterogeneous architectures under severe non-IID skew, CRAD consistently outperforms competing methods in global accuracy.
1. Introduction
Conventional federated learning struggles when clients use incompatible architectures and hold skewed data, while existing distillation approaches often require centralized or public transfer infrastructure. CRAD addresses the open problem of combining peer predictions by filtering disagreement and weighting teachers according to class-wise reliability and support.
- Motivation: Parameter averaging requires a shared architecture and can produce a model worse than individual specialists under skewed data.These constraints arise because incompatible weight spaces cannot be meaningfully combined and local specialization differs across clients.
- Motivation: Existing distillation methods avoid architectural homogeneity but typically rely on a public dataset or central server.This reintroduces privacy and centralization costs that federated learning is intended to avoid.
- Open Problem: Combining peer predictions remains under-examined once a client receives heterogeneous teacher outputs.The challenge is especially important when teacher reliability varies by peer and class.
- Contribution: CRAD discards insufficiently supported or consensus-disagreeing teachers per class, then precision-weights the survivors using class-accuracy estimates.Each client shares compact class-wise statistics describing validation support and correctness.
- Contribution: CRAD provides decentralized, public-data-free collaboration across architectures and data distributions through peer distillation over the shared class posterior.The framework handles model and data heterogeneity simultaneously.
2. Related Work
Related work either operates in a shared parameter space or uses knowledge distillation with auxiliary infrastructure. CRAD targets the remaining setting: decentralized, public-data-free teacher fusion that accounts for class-dependent reliability and finite-sample uncertainty.
- Parameter-Space Methods: Parameter-space methods mitigate non-IID client drift but remain restricted to a shared model space.Personalization methods likewise operate within shared parameter-space assumptions.
- Knowledge Distillation: Knowledge distillation transfers architecture-agnostic soft predictions but commonly depends on a public dataset, central server, or both.These methods therefore require infrastructure absent from the target decentralized setting.
- Teacher Fusion: Federated KD methods usually fuse peer predictions by uniform averaging or fixed data-volume weighting.Such rules treat teachers as equally trustworthy for every class despite heterogeneous capacities and non-IID training partitions.
- Teacher Fusion: Confidence-based weighting can be unreliable under distribution shift and ignores the amount of data supporting each teacher’s class belief.Reliability weighting also typically assumes trusted validation data or a shared transfer set.
- CRAD: CRAD addresses the open problem of decentralized, public-data-free reliability weighting by retaining consensus-aligned teachers and weighting them by precision.Its weighting accounts for statistical uncertainty estimated from finite local data.
3. Method
CRAD performs architecture-agnostic peer distillation in a fully decentralized, non-IID federation. It constructs class-specific targets by filtering unsupported or discordant teachers and precision-weighting the remaining accuracy estimates before combining local supervision with distillation.
- Setting: Clients may use distinct architectures and non-IID local label distributions, including classes with few or no local samples.All clients share the same label space while maintaining private datasets and models.
- Knowledge Transfer: Each client receives peer snapshots and evaluates them on its own local batches, avoiding raw-data exchange and central coordination.All clients distill from the previous round’s snapshots to preserve synchronous-round semantics.
- Knowledge Transfer: Teacher outputs remain compatible across architectures because the shared interface is the C-dimensional class posterior.Teachers operate as black-box mappings from inputs to class-space outputs.
- Class-wise Filtering: CRAD first removes teachers lacking class support or deviating beyond the per-class median peer deviation.The adaptive median retains the supported teachers most aligned with consensus without requiring a tuned threshold, and different classes can use different subsets.
- Reliability Statistics: Each teacher’s class accuracy is treated as a finite-sample Bernoulli estimate stabilized with Agresti–Coull correction and an associated variance.The shared statistics require only 2C scalars per client per round and expose no raw samples.
- Reliability Weighting: Surviving teachers receive inverse-variance weights, which increase with both estimated decisiveness and sample support.Because variance scales as 1/n_cj, few-example estimates receive smaller weights; filtering precedes weighting to reduce the influence of confident outliers.
- Student Update: The student minimizes a convex combination of local-label cross-entropy and temperature-scaled KL divergence to the aggregated target.The precision weights act only on teacher estimates, preserving architecture-agnostic aggregation.
4. Experiments
Across heterogeneous architectures and severe non-IID partitions, CRAD is evaluated against competing teacher-aggregation methods on CIFAR-10, CIFAR-100, and PathMNIST. It consistently achieves the strongest reported global accuracy, while improving rare-class performance, convergence stability, scalability, and aggregation efficiency.
- Setup: Experiments use CIFAR-10, CIFAR-100, and PathMNIST with 10 clients under severe non-IID Dirichlet partitioning and heterogeneous architectures.The architecture pool includes ResNet-18, ResNet-18-Half, and CNN-6; training runs for 300 communication rounds.
- Benchmark results: CRAD reaches 78.60% global accuracy on CIFAR-10, exceeding Uncertainty KD at 77.41% and uniform averaging at 77.19%.All aggregation variants share the same decentralized backbone, isolating the effect of reliability-aware aggregation.
- Benchmark results: CRAD ranks first on CIFAR-100 at 48.22% global accuracy, ahead of DFML at 47.29% and uniform averaging at 41.53%.The advantage over uniform averaging widens as the number of classes increases and per-class support decreases.
- Benchmark results: CRAD attains the best PathMNIST global accuracy at 89.44%, versus 87.59% for uniform averaging, while also achieving 88.14% local accuracy.The result transfers the reliability-aware aggregation advantage from natural images to colon-pathology histology.
- Per-class analysis: On the rarest CIFAR-10 class with 30 local samples, CRAD retains about 56% accuracy versus about 47% for the other aggregation variants.The agreement filter alone tracks uniform averaging, indicating that precision-weighting the surviving teachers supplies most of the rare-class gain.
- Stability and scalability: CRAD remains above every baseline from early communication rounds, converges smoothly, and improves over Uniform KD as the candidate client pool grows.Its relative global-accuracy advantage over Uniform KD increases from a factor of 1.2 at N=20 to 2.0 at N=100; its added communication is a 2C-scalar vector under 1 KB.
5. Limitations and Future Work
The evaluation is limited to image classification, and extending CRAD to other modalities and vision tasks remains future work. The shared statistics vector also exposes clients’ validation label proportions, while end-to-end differential privacy for snapshots remains open.
- CRAD is evaluated only on image-classification tasks, with extension to other modalities and vision tasks left for future work.
- The shared statistics vector reveals each client’s validation label proportions through its class-count statistics.
- At ε = 1, Laplace noise on the shared counts reduces CIFAR-100 global accuracy by 0.82 points, from 48.22% to 47.40%.
- End-to-end differential privacy covering exchanged model snapshots remains an open problem.
6. Conclusion
The paper presents a decentralized, server-free, public-data-free framework for distillation across differing architectures and identifies peer-prediction aggregation as an under-examined problem. CRAD filters disagreeing teachers and precision-weights the rest, achieving the best global accuracy across three heterogeneous, severely non-IID benchmarks.
- The framework enables clients with differing architectures to collaborate by distilling peer predictions without a server or public data.
- CRAD addresses uneven, class-dependent teacher reliability by discarding teachers that disagree with peer consensus and precision-weighting the survivors.
- Across CIFAR-10, CIFAR-100, and PathMNIST, CRAD attains the best global accuracy under heterogeneous architectures and severe non-IID skew.
A. Qualitative Mechanism on Individual Images
Figures 4 and 5 show how CRAD combines class support, per-class accuracy, consensus filtering, and precision weighting on individual CIFAR-10 predictions, including both a successful correction and a failure mode.
- Weighting context: Figure 4 contrasts per-client validation support with per-class accuracy, showing why CRAD weights both performance and the evidence behind it.Support enters the variance estimate, while accuracy is measured on the balanced global test set.
- Two-stage aggregation: Figure 5 marks each teacher as retained, support-filtered, or consensus-filtered, while reporting its normalized distillation weight and probability on the true class.The acceptance band is defined around the preliminary consensus using the class-specific threshold τc.
- Success case: 59.5% of the bird-image weight goes to the teacher with 180 bird samples, raising the true-class target from 68.3% to 79.4%.That teacher assigns 80.1% probability to the correct class; four consensus outliers and two single-sample teachers are removed first.
- Failure case: On the dog image, filtering removes the best-evidenced and most accurate teacher before weighting, leaving the aggregated target at 27.2%.The removed teacher has 258 class samples, 63.7% probability on the true class, and 90.5% class accuracy, but lies far outside the low consensus.
- Limitation: The failure illustrates the limitation of assuming unreliable teachers are a minority: a less-informed majority can suppress a useful specialist.The paper leaves support-aware consensus filtering as a future refinement.