Source-linked AI summary
CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learning
Yifan Ying, Qing Tian
TL;DR
Decentralized federated distillation supports heterogeneous clients without a server or public data, but teacher targets can be unevenly reliable under non-IID skew and current pipelines make hard trust decisions. CALM applies smooth class-wise agreement, per-sample disagreement, and label-consistency gates. Across four benchmarks, it improves on uniform and hard-filtered distillation and matches or exceeds competing methods.
Problem
Under heterogeneous, non-IID data, peer targets vary in reliability, yet existing pipelines discard outlier teachers by threshold and apply surviving targets at full strength.
Source: §1 · p. 1Method
CALM combines smooth class-wise agreement weighting with per-sample disagreement and true-label-support gates to modulate distillation.
Source: §1 · p. 1, §3.4 · p. 3Results
Takeaways & Limitations
CALM’s per-sample weight also separates wrong from right peer targets, providing an indicator of where peer supervision can be trusted.
Source: §1 · p. 1The method assumes clients’ models share a label space, despite allowing heterogeneous architectures and private datasets.
Source: §3 · p. 2
Abstract
from arXiv · showhide
Conventional federated learning relies on parameter averaging, which forces clients to be doubly homogeneous: all must run an identical architecture, and accuracy degrades when local data are non-IID. Decentralized federated distillation sidesteps both: each client runs its peers' model snapshots as teachers on its own local data and distills from their soft predictions, with no server, no public data, and no shared architecture. Under severe non-IID skew, however, the trustworthiness of the aggregated teacher target is a matter of degree, yet existing pipelines make hard, all-or-nothing decisions: outlier teachers are discarded by threshold, and whatever target survives is trusted in full. We propose CALM, which replaces every hard decision with a smooth trust gate at three levels: per class, teachers are weighted by agreement with the peer consensus; per sample, distillation is scaled by the teachers' divergence from that target; and a label gate scales it by how strongly the target supports the sample's true label. None of this adds communication or auxiliary data. On CIFAR-10, SVHN, OrganAMNIST, and Google Speech Commands with heterogeneous client architectures under Dirichlet label skew, CALM consistently outperforms uniform and hard-filtered distillation and matches or exceeds competing heterogeneous-FL methods.
1. INTRODUCTION
Decentralized federated distillation accommodates heterogeneous architectures and private, non-IID data, but its teacher targets can be unevenly reliable. CALM replaces hard teacher filtering and uniform target trust with three smooth gates, and improves global accuracy across four benchmarks under severe Dirichlet skew.
1. INTRODUCTION
- Decentralized distillation exchanges soft predictions rather than weights, enabling clients with different architectures to learn without a server, public transfer set, or raw-data exchange.Each client evaluates peer snapshots on its own private data.Source: §1 · p. 1
- Under heterogeneous, non-IID data, peer teachers can provide sharply uneven target quality, while hard filtering and full-strength distillation discard graded trust information.A teacher just past the agreement cutoff is silenced, while surviving teachers and even targets formed amid conflict are trusted fully.Source: §1 · p. 1
- CALM uses smooth class-wise agreement, per-sample disagreement, and true-label support gates to modulate teacher contributions and distillation strength.The label gate addresses cases where correlated teachers confidently agree on the wrong class.Source: §1 · p. 1
- Across CIFAR-10, SVHN, OrganAMNIST, and Google Speech Commands under heterogeneous architectures and severe Dirichlet skew, CALM improves accuracy over uniform and hard-filtered distillation and matches or exceeds competing methods.The contribution summary also reports that each gate contributes and the combination is strongest.Source: §1 · p. 1
- CALM’s per-sample weight also separates cases where the peer target is wrong from those where it is right, serving as an indicator of peer-supervision trustworthiness.Figure 1 depicts the three gates and the final weight as a smooth modulation without thresholds.Source: §1 · p. 1, Figure 1 · p. 2
2. RELATION TO PRIOR WORK
Prior federated-learning approaches address non-IID drift or architecture heterogeneity through parameter-space correction or knowledge distillation, but often retain a shared-weight space, public data, or a central coordinator. CALM shares the decentralized, public-data-free regime of DFML and CRAD while replacing hard class-wise teacher censoring with graded weighting and adding sample-level gates.
2. RELATION TO PRIOR WORK
- Parameter-space approaches such as FedProx and SCAFFOLD mitigate non-IID drift but require a shared weight space and cannot span distinct architectures.KD-based approaches exchange predictions, but public proxy sets, generated transfer data, or central-server distillation retain other dependencies.Source: §2 · p. 2
- DFML and CRAD [11] share CALM’s peer-to-peer regime without a server or public data, while CRAD censors teachers per class and CALM weights them gradually.CALM additionally gates how strongly each sample’s fused target is imposed.Source: §2 · p. 2
- Prior schemes weight teachers by data volume or class-wise reliability, weight samples by confidence against labels, or censor outliers, but do not combine CALM’s target and sample controls.The cited label-aware schemes assume co-located teachers and apply fused targets at full strength to every sample.Source: §2 · p. 2
3. METHOD
CALM operates among clients with private, non-IID data and heterogeneous models that share a label space, exchanging model snapshots rather than raw data. It replaces binary teacher filtering and equal-strength distillation with class-wise agreement weighting, sample-level disagreement and label gates, and a combined objective requiring no added communication, auxiliary data, or learned parameters.
3. METHOD
- Clients have private datasets with sharply differing class proportions, heterogeneous architectures, a shared label space, and no central server or public dataset.Peers exchange model snapshots, which each client runs locally as teachers on its own training batches; under partial participation, teachers are the active peers.Source: §3 · p. 2
- The conventional target uniformly averages peer predictions, and its KD loss treats every sample equally, with optional binary outlier filtering.CALM replaces these assumptions with three gates.Source: §3 · p. 2
3.1. Gate 1: Soft Class-wise Agreement Target
- Gate 1 weights each teacher’s class prediction by its soft agreement with the class-wise peer consensus, then renormalizes the weighted average into the distillation target.Deviating teachers are exponentially attenuated rather than removed; a teacher can be trusted on classes it knows and discounted on others.Source: §3.1 · p. 3
3.2. Gate 2: Disagreement-Modulated Distillation Strength
- Gate 2 measures each sample’s teacher divergence from the aggregated target and lowers KD strength as disagreement rises.A floor preserves some peer signal for contested samples, and the divergence reuses teacher predictions already computed for aggregation.Source: §3.2 · p. 3
3.3. Gate 3: Label-Consistency Gate
- Gate 3 uses the aggregated target’s probability on the true label to attenuate distillation when peer predictions contradict local evidence.This addresses confidently agreed wrong targets that the label-free gates cannot identify; learning then falls back on cross-entropy supervision.Source: §3.3 · p. 3
3.4. Full Objective
- The full objective multiplies the two sample-level gates to weight KD, while all three gates make trust decisions smoothly and add no communication, auxiliary data, or learned parameters.Each gate degrades toward the baseline as its scale is loosened.Source: §3.4 · p. 3
4. EXPERIMENTS
Across heterogeneous architectures and severe Dirichlet label skew, CALM is evaluated against distillation baselines on four benchmarks. It leads across the main comparisons, while ablations and trust-score analysis examine its gates and sample weighting.
4.1. Setup
- The evaluation spans four benchmarks and three modalities, with 50 or 100 clients using three structurally distinct model architectures and Dirichlet-partitioned data.Ten clients participate per round; the main partition uses αDir=0.3, described as severe label skew.Source: §4.1 · p. 3
4.2. Main Results
- CALM leads all four benchmarks at 100 clients, consistently beats hard filtering, and exceeds DFML in seven of eight settings across three modalities.At 50-client CIFAR-10, it trails DFML by 0.24 points.Source: Table 1 · p. 4, §4.2 · p. 4
4.3. Analyses
- On SVHN, soft agreement lifts Uniform KD from 44.24% to 58.53%, disagreement adds 5.3 points, and label gating completes CALM at 71.50%.Class-wise weighting repairs the target; the sample-level gates discount targets that reweighting cannot repair, with label gating yielding the larger sample-level gain (+7.7 vs. +5.3 points).Source: Table 2 · p. 4, §4.3 · p. 4
- CALM leads at every tested CIFAR-10 skew level, with its largest gain over uniform averaging (+8.5 points) at αDir=0.3; margins narrow at extreme skew.At extreme skew, fewer teachers overlap each client’s classes and little transferable signal remains to reweight.Source: Table 3 · p. 4, §4.3 · p. 4
- Wrong aggregated targets receive lower λ (AUROC 0.970), while correct targets are still only partially admitted (median 0.435).The reported AUROC has a 95% client-bootstrap CI of [0.962, 0.979].Source: Figure 2 · p. 4, §4.3 · p. 4
5. CONCLUSION
CALM replaces hard trust decisions in decentralized, public-data-free federated learning with three smooth gates. Under severe non-IID skew, it outperforms uniform distillation and hard filtering, and matches or exceeds prior decentralized KD.
5. CONCLUSION
- CALM uses class-wise agreement weighting to build targets, per-sample disagreement to modulate teaching strength, and a label gate to anchor targets in local labels.The three gates add no communication or auxiliary data.Source: §5 · p. 4
- Under severe non-IID skew, CALM outperforms uniform distillation and hard filtering.Source: §5 · p. 4
- CALM matches or exceeds prior decentralized knowledge distillation in the paper’s stated setting.Source: §5 · p. 4