Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

Motivation

• Decentralized SGD is undefined for heterogeneous architectures because parameter averaging requires peers to share the same parameter space.Gossip learning removes the central server, with peers updating locally and communicating over an asynchronous P2P network.Source: Introduction · p. 1
• KD enables heterogeneous peers to communicate by exchanging soft predictions in a common output space rather than parameters.Existing federated KD methods use this output space but rely on a coordinator or synchronized rounds.Source: Introduction · p. 1, Related Work · p. 2

Function-space framework

• The analysis measures consensus in L2(µ; R^C), where a KD exchange acts as a logit-space function-mixing step.Its contraction rate is proportional to ρ_f = Θ(ηαp_minλ_2(L)/|E|), and the proof couples this contraction with task-risk descent.Source: Introduction · p. 1

Contributions

• The main theorem controls time-averaged functional stationarity and function-space disagreement within an O(η) + O(B_f^2) + O(ζ_f^2) neighbourhood.The terms represent stochastic discretization, representational error, and persistent local-task heterogeneity.Source: Introduction · p. 1
• The paper characterizes a communication operator that remains defined for heterogeneous peers and states two modelling bridges linking parameter updates, functional effects, and reachable-class constraints.Parameter averaging is retained as a homogeneous-model control.Source: Introduction · p. 1

Notation and geometry

• The framework models N peers on a graph with algebraic connectivity λ2(L), and represents predictive distributions in the Hilbert space H = L2(µ; R^C).Prediction disagreement is measured through squared distances under the reference measure µ.Source: Important Notations · p. 2

Notation and diagnostics

• The preliminaries define the global objective F, consensus error Ψf, mean predictor, KD contraction rate ρf, teacher lag, and capacity floor Bf.Bf measures how far the required optimum lies outside the weakest architecture’s reachable class.Source: Important Notations · p. 2

Representational capacity

• A3 introduces peer-specific reachable prediction classes and the oracle capacity floor Bf as the irreducible approximation error for a common output-space optimum.The constrained task minimizer is defined over bounded-logit softmax predictors, and Bf enters the terminal neighbourhood as an approximation-error parameter.Source: Core Assumptions · p. 2

Parameter-to-function realizability

• A4 requires parameter SGD steps to induce controlled finite-support logit and prediction updates through local kernel geometry and bounded Taylor and noise terms.A4 bridges actual parameter SGD with the functional recursion used in the proof, without asserting full-space surjectivity or implying A10.Source: Core Assumptions · p. 2

Restricted task/KD alignment

• A10 limits task-gradient and directed KD-pull components outside each peer’s realizable tangent directions, linking common output-space pulls to peer-specific updates.Its small-gain conditions depend on transient coefficients γtask and γKD, while coefficients multiplying Bf need only be finite.Source: Core Assumptions · p. 3

Heterogeneous peer communication

• Peers may have different parameter dimensions, so parameter averaging is undefined and they exchange temperature-scaled output distributions instead.The temperature τ controls distribution softness.Source: Network Topology and Heterogeneous Peers · p. 3

Asynchronous gossip

• Each asynchronous event activates one graph edge, whose endpoints act once as student and teacher without requiring a global clock.The model does not represent concurrent writes; bounded races can be incorporated through the lag bound A.Source: Random-Edge Asynchrony and EMA Teachers · p. 3, Figure 1 · p. 3

Teacher updates

• Selected peers exchange soft probabilities on a shared reference batch, using either EMA parameters or the current parameters in the live-network variant.The EMA variant supplies probabilities from an exponential moving average, whereas the live-network variant uses current parameters.Source: Random-Edge Asynchrony and EMA Teachers · p. 3

Student update

• The student takes an SGD step combining supervised cross-entropy with a teacher-to-student divergence, weighted by α between the two objectives.The system is implemented as a gossip-distillation architecture in which no parameters are exchanged.Source: The P2P-KD Loss and Student Update · p. 3, Figure 1 · p. 3, System Architecture · p. 3

Functional setup

• The analysis uses a finite reference measure μ as the common prediction space for heterogeneous peers and applies the actual preconditioned KD update.A4 separates the parameter-to-logit chain rule from the frozen-kernel condition, while A10 controls components outside peer-specific tangent spaces.Source: Theoretical Convergence Analysis · p. 4

Consensus contraction

• Random-edge KD contracts function disagreement in a uniformly equivalent logit metric, with a defect capturing task drift, stochasticity, and capacity limitations.The metric is positive definite and uniformly equivalent to arithmetic logit disagreement; the consensus recursion includes a critical first-order local task-force term.Source: Global KD Contraction · p. 4

Heterogeneity defect

• O(ζ_f^2) is unavoidable for implemented non-IID local CE updates because heterogeneous quadratic objectives can produce a nonzero consensus fixed point even when B_f = 0 and stochastic noise is zero.The theorem therefore separates stochastic discretization, capacity, and deterministic task heterogeneity rather than attributing all neighbourhood effects to η.Source: Global KD Contraction · p. 4, Coupled Bound · p. 5

Task descent

• Functional task descent holds under the task-side assumptions when the task margin is positive, with 0 ≤ α < 1 and activation correction ensuring uniform peer averaging.Inverse activation weights compensate for nonregular graphs, and the descent proof does not absorb an η-independent conditioning error into O(η).Source: Task Descent and Staleness · p. 4

Experimental design

• 25 trajectories record function-space consensus, a finite-reference stationarity proxy, rate analysis, and task accuracy every 1,000 edge activations.The experiments target observable implications of Theorem 2 rather than benchmarking specialized decentralized optimizers.Source: Experiments · p. 5

Consensus

• 40.2–61.0× contraction reduces function-space disagreement across homogeneous, width-heterogeneous, and mixed-family rosters, unlike isolated controls.Terminal KD disagreement is 2.81×10−4 to 4.10×10−4, versus 1.36 × 10−2 to 1.51 × 10−2 for isolated controls.Source: Empirical Validation of the Bound · p. 6

Stationarity and rate

• 0.88–1.90 late-transient exponents support stationarity decay at the 1/T order or faster across the eight Setting A–D main runs.Four values lie in [0.88, 1.03], while Set-C values are 1.21 and 1.90 and Set-D values are 0.88 and 0.91.Source: Empirical Validation of the Bound · p. 7

Neighbourhood signatures

• (1.10, 1.17, 1.25) × 10−3 at η = 0.0125 falls to (3.87, 3.98, 4.08) × 10−4 at η = 0.05, then rises slightly at η = 0.1.The sweep exhibits finite-horizon competition between the 1/(ηT) transient at small η and the discretization neighbourhood at large η.Source: Empirical Validation of the Bound · p. 7

Task quality and controls

• 0.535–0.729 main KD ensemble accuracies for seed 0 remain well above chance across every roster, indicating consensus does not collapse to a uniform predictor.D-SGD is reported only for Setting A as an appendix mechanism reference, not for a performance claim.Source: Empirical Validation of the Bound · p. 7

Empirical validation

• 40.2–61.0× reduction in function-space disagreement occurs across all CIFAR-10 experiments with KD.The diagnostics are consistent with the theoretical prediction across the reported network settings.Source: Conclusion · p. 7
• Late-transient exponents near or above one and the step-size sweep’s predicted transient–neighbourhood tradeoff support the convergence diagnostics.These observations accompany the reported disagreement reductions in the CIFAR-10 experiments.Source: Conclusion · p. 7

Limitations

• The theorem depends on stable kernels, bounded staleness, A4 for the parameter-to-logit bridge, and A10 for restricted alignment.These assumptions expose where the proof uses network geometry but are not automatic for arbitrary feature learning or aggressive step sizes.Source: Limitations and Future Work · p. 7

Limitations and future work

• The experiments estimate a combined terminal neighbourhood rather than separate O(η), B_f^2, and O(ζ_f^2) contributions, using finite reference support for KD and evaluation.Future work should measure A4/A10 residuals, separate KD and held-out evaluation supports, and estimate B_f through reachable-class probes.Source: Limitations and Future Work · p. 7

Appendix

• The appendix states Assumptions A1–A10, develops named lemmas and propositions, culminates in Theorem A1, and details experiment setups and results with comprehensive statistical analysis.It is explicitly described as self-contained.Source: Appendix · p. 10

Notations Clarification

• The section specifies the mathematical notation used throughout the convergence proof.It functions as a notation reference for the appendix.Source: Notations Clarification · p. 10
• It defines the variables and operators appearing in the proof.These definitions support consistent interpretation of the convergence analysis.Source: Notations Clarification · p. 10
• It records the constants used in the federated knowledge distillation appendix.The section covers every constant used throughout the proof.Source: Notations Clarification · p. 10

• N denotes the total number of peers, while i, j, and k index specific peers.Source: Network and Graph Topology · p. 10
• E is the set of undirected communication links, and directed edge set E⃗ contains both orientations for every undirected edge.Source: Network and Graph Topology · p. 10
• The graph Laplacian is L = D − A, with eigenvalues λ1, λ2, …, λmax; algebraic connectivity λ2 dictates information-mixing speed.Here, D is the degree matrix and A is the adjacency matrix.Source: Network and Graph Topology · p. 10

• t counts individual edge activations, while T denotes the total number of events or steps in the horizon.Source: Time and Optimization Parameters · p. 10
• η is the base parameter-space SGD step size, and α ∈ (0, 1) balances local task loss against KD loss.Source: Time and Optimization Parameters · p. 10
• τ scales logits before softmax application as the softmax temperature scalar.Source: Time and Optimization Parameters · p. 10
• w = ηατ is the effective step size for KD logit contraction.Source: Time and Optimization Parameters · p. 10

Core objects

• The reference measure µ is a finite evaluation dataset used to measure logits and predictions, while θ_i denotes peer i’s network parameters.The notation also includes initialization parameters θ_i^0 and parameter updates Δθ_i.Source: Data, Functions, and Prediction Spaces · p. 10

Function representations

• Logits Z_i are pre-softmax outputs evaluated over µ, with Z_i^t denoting peer i’s logits at time t and Z̄ the network-wide mean logits.Source: Data, Functions, and Prediction Spaces · p. 10

Prediction representations

• Prediction probabilities P_i are computed as softmax(Z_i/τ), where τ is the temperature parameter.Source: Data, Functions, and Prediction Spaces · p. 10

Targets and aggregates

• Teacher targets p_j^T may use peer j’s live prediction or an exponential moving average, alongside the time-t mean prediction probability p̄^t.Source: Data, Functions, and Prediction Spaces · p. 10

Assumptions and optimum

• The framework enforces a strictly positive class-probability lower bound p_min and defines q⋆ as the hypothetical optimal target distribution for the global task.Source: Data, Functions, and Prediction Spaces · p. 10

Jacobian definitions

• Jz_i is the logit Jacobian with respect to parameters, while Jp_i is the prediction-probability Jacobian.These Jacobians describe parameter sensitivity in logit and prediction spaces, respectively.Source: Jacobians and Neural Tangent Kernels… · p. 10

Neural Tangent Kernels

• Θ_i is the logit-space Neural Tangent Kernel, and Θ0_i denotes its initial value at θ0_i.The section defines Θ_i from the logit Jacobian and separately identifies the initialization-time kernel.Source: Jacobians and Neural Tangent Kernels… · p. 10
• K_i is the Neural Tangent Kernel in prediction space, defined from Jp_i.This kernel transfers the NTK construction from logits to prediction probabilities.Source: Jacobians and Neural Tangent Kernels… · p. 10

KD update operator

• H_i maps prediction-space KD-loss gradients into parameter updates through a cross-Jacobian operator.The operator is defined as H_i = τJp_i.Source: Jacobians and Neural Tangent Kernels… · p. 10

• H_z is the full ambient vector space of evaluated logits.Source: Tangent Spaces and Projections · p. 10
• T_z_i is the logit tangent space, representing logit directions the model can physically move in.Source: Tangent Spaces and Projections · p. 10
• Π_z_i projects orthogonally onto T_z_i, while I − Π_z_i projects onto the normal space of directions the model cannot move.Source: Tangent Spaces and Projections · p. 10
• T_p_i and Π_p_i denote the prediction tangent space and its orthogonal projection matrix.Source: Tangent Spaces and Projections · p. 10
• M(P) = diag(P) − PP^T defines the softmax metric tensor.Source: Tangent Spaces and Projections · p. 10

Strict Lyapunov Coefficients (Lemma 5 in Main Paper, Lemma 9 in Appendix)

• The convergence proof combines consensus and task-descent recursions through a strict Lyapunov function under the small-gain condition bF CG < aF c0.The Lyapunov function is Lt = Rt + λVW (Zt), and the condition yields strictly positive descent factors.Source: Strict Lyapunov Coefficients (Lemma 5… · p. 12

Decentralized Optimization and the Type Mismatch

• KD replaces undefined cross-architecture parameter averaging with contraction in a shared finite-support prediction space.The analysis preserves the graph viewpoint while changing the state variable from parameters to evaluated predictions.Source: Decentralized Optimization and the… · p. 13

Assumption Validity and Diagnostics

• A4 and A10 are local, trajectory-dependent bridge assumptions connecting parameter updates to finite-support logits and requiring task gradients and KD pulls to align with realizable tangent directions.These assumptions are sufficient analytical conditions, not universal facts about finite neural networks.Source: Assumption Validity and Diagnostics · p. 14, Detailed Assumptions · p. 14

Checkpoint-Level Diagnostics

• The empirical evidence validates observable consequences of the conditional theory rather than establishing that neural CIFAR-10 trajectories satisfy A4 and A10.The narrowed interpretation emphasizes prediction-space contraction and finite-reference stationarity proxies.Source: Checkpoint-Level Diagnostics · p. 18

Convergence Proof

• Theorem A1 establishes constant-step convergence under A1–A10, connectedness, frozen-kernel conditions, threshold constraints, and 0 < η ≤ ηmax.The theorem provides positive constants controlling the convergence bounds under these assumptions.Source: Convergence Proof · p. 30

Empirical Convergence-Rate Diagnostics

• 1.21 and 1.90 are the late 85% dynamic exponents for setting C’s seed 0 and extended seed-1 histories, while settings A and B remain close to 1.00.The mixed-family setting D also reaches the consensus neighbourhood, but stationarity traces are less uniform across rosters.Source: Empirical Convergence-Rate Diagnostics · p. 32, Stored Convergence Traces · p. 32

Statistical Analysis of the Data Results

• 40.9×, 47.5×, and 41.0× reductions in disagreement occur for settings B–D, compared with isolated local training; the mean reduction across A–D is 41.4×.KD finishes between 2.88 × 10^-4 and 3.74 × 10^-4, while isolated runs finish between 1.36 × 10^-2 and 1.51 × 10^-2.Source: Statistical Analysis of the Data Results · p. 34

Scope of the Empirical Evidence

• The learning-rate sweep and four rosters support a finite-horizon neighbourhood tradeoff, with terminal consensus values remaining on the order of 10^-4 across homogeneous, width-heterogeneous, and mixed-family settings.The evidence supports the stated scope: random-edge KD contracts prediction disagreement across all three roster types, while the sweep exhibits the predicted tradeoff.Source: Sensitivity to Learning Rate and Model… · p. 35, Scope of the Empirical Evidence · p. 35