PaperlayerView exact revision on arXiv

Source-linked AI summary

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

Lucas Qingyang Fang, Tiyao Liu, Jinhao Jing, Zeji Li, Kaijie Chen, Harikrishna Kuttivelil, Katia Obraczka

arXiv:2609.01952v1cs.LGcs.AI

TL;DR

Asynchronous P2P knowledge distillation lacks convergence theory when peers use different architectures and parameter spaces. This paper analyzes KD as function-space mixing and proves conditional convergence, with experiments showing 40.2–61.0× reductions in function-space disagreement.

  • Problem

    Asynchronous, fully decentralized P2P knowledge distillation lacks convergence theory for peers with different reachable function classes and parameter spaces.

  • Method

    The paper analyzes asynchronous random-edge KD as kernel-weighted mixing in a common prediction space and couples function-space contraction with task-objective descent.

  • Results

    40.2–61.0×: KD reduces function-space disagreement across all CIFAR-10 experiments, with late-transient exponents near or above one and the predicted step-size tradeoff.

  • Takeaways & Limitations

    Under the stated regularity and realizability conditions, asynchronous P2P KD exhibits the predicted convergence transient and neighbourhood behavior in function space.

  • Takeaways & Limitations

    The theorem assumes stable kernels, bounded staleness, and realizability conditions, while experiments estimate only a combined terminal neighbourhood on finite reference support.

Abstract

from arXiv · show

Decentralized, serverless learning increasingly connects devices running different architectures, where the standard tool, decentralized SGD, is undefined as models with different parameter counts cannot be averaged. Knowledge distillation (KD) exchanges soft predictions rather than weights and sidesteps this obstacle, yet convergence theory for fully decentralized, asynchronous peer-to-peer (P2P) KD is lacking. We provide one, relocating consensus from parameter space to function (output) space: a KD event is a geometric contraction operator in logit space on the peers' predictive distributions, which we analyse in the Hilbert space of predictions on a reference measure. Under standard smoothness/variance assumptions and two realizability assumptions, one bridging parameter SGD to the functional step and one controlling restricted task/KD alignment, the time-averaged functional stationarity and function-space disagreement converge at rate $O(1/(ηT))$ to an $O(η)+O(B_f^2)+O(ζ_f^2)$ neighbourhood. Here $B_f$ is the distance from the task optimum to the peers' reachable classes and $ζ_f$ measures persistent local-task heterogeneity. Across homogeneous, width-heterogeneous, and mixed-family networks of the experiments, KD contracts function disagreement by $40-61\times$, while isolated training does not. The sampled stationarity diagnostic has late transient exponents $0.99-1.90$ on the shared-skeleton main runs, and the four-point step-size sweep exhibits the predicted transient: neighbourhood tradeoff.

Introduction

The introduction identifies a gap in asynchronous, fully decentralized KD for heterogeneous architectures and presents a function-space convergence framework to address it. It establishes convergence controls for functional stationarity and disagreement while retaining parameter averaging only as a homogeneous-model control.

  • Motivation: Decentralized SGD is undefined for heterogeneous architectures because parameter averaging requires peers to share the same parameter space.Gossip learning removes the central server, with peers updating locally and communicating over an asynchronous P2P network.
  • Motivation: KD enables heterogeneous peers to communicate by exchanging soft predictions in a common output space rather than parameters.Existing federated KD methods use this output space but rely on a coordinator or synchronized rounds.
  • Function-space framework: The analysis measures consensus in L2(µ; R^C), where a KD exchange acts as a logit-space function-mixing step.Its contraction rate is proportional to ρ_f = Θ(ηαp_minλ_2(L)/|E|), and the proof couples this contraction with task-risk descent.
  • Contributions: The main theorem controls time-averaged functional stationarity and function-space disagreement within an O(η) + O(B_f^2) + O(ζ_f^2) neighbourhood.The terms represent stochastic discretization, representational error, and persistent local-task heterogeneity.
  • Contributions: The paper characterizes a communication operator that remains defined for heterogeneous peers and states two modelling bridges linking parameter updates, functional effects, and reachable-class constraints.Parameter averaging is retained as a homogeneous-model control.

Preliminaries

The preliminaries define the graph, prediction-space geometry, objective, consensus diagnostics, and key error terms for asynchronous P2P KD. They state capacity, parameter-to-logit realizability, and restricted task/KD alignment assumptions connecting heterogeneous peer updates to functional analysis.

  • Notation and geometry: The framework models N peers on a graph with algebraic connectivity λ2(L), and represents predictive distributions in the Hilbert space H = L2(µ; R^C).Prediction disagreement is measured through squared distances under the reference measure µ.
  • Notation and diagnostics: The preliminaries define the global objective F, consensus error Ψf, mean predictor, KD contraction rate ρf, teacher lag, and capacity floor Bf.Bf measures how far the required optimum lies outside the weakest architecture’s reachable class.
  • Representational capacity: A3 introduces peer-specific reachable prediction classes and the oracle capacity floor Bf as the irreducible approximation error for a common output-space optimum.The constrained task minimizer is defined over bounded-logit softmax predictors, and Bf enters the terminal neighbourhood as an approximation-error parameter.
  • Parameter-to-function realizability: A4 requires parameter SGD steps to induce controlled finite-support logit and prediction updates through local kernel geometry and bounded Taylor and noise terms.A4 bridges actual parameter SGD with the functional recursion used in the proof, without asserting full-space surjectivity or implying A10.
  • Restricted task/KD alignment: A10 limits task-gradient and directed KD-pull components outside each peer’s realizable tangent directions, linking common output-space pulls to peer-specific updates.Its small-gain conditions depend on transient coefficients γtask and γKD, while coefficients multiplying Bf need only be finite.

System Architecture

The system uses asynchronous gossip distillation over a connected peer-to-peer graph, allowing heterogeneous models to exchange output distributions instead of parameters. Randomly activated edges alternate student and teacher roles, with supervised learning combined with a distillation pull.

  • Heterogeneous peer communication: Peers may have different parameter dimensions, so parameter averaging is undefined and they exchange temperature-scaled output distributions instead.The temperature τ controls distribution softness.
  • Asynchronous gossip: Each asynchronous event activates one graph edge, whose endpoints act once as student and teacher without requiring a global clock.The model does not represent concurrent writes; bounded races can be incorporated through the lag bound A.
  • Teacher updates: Selected peers exchange soft probabilities on a shared reference batch, using either EMA parameters or the current parameters in the live-network variant.The EMA variant supplies probabilities from an exponential moving average, whereas the live-network variant uses current parameters.
  • Student update: The student takes an SGD step combining supervised cross-entropy with a teacher-to-student divergence, weighted by α between the two objectives.The system is implemented as a gossip-distillation architecture in which no parameters are exchanged.

Theoretical Convergence Analysis

The analysis establishes asynchronous P2P KD as a function-space process: random-edge updates contract logit disagreement while task descent yields convergence under smoothness, alignment, and coupling conditions. The resulting time-averaged stationarity and disagreement reach an O(η)+O(B_f^2)+O(ζ_f^2) neighbourhood at rate O(1/(ηT)), with persistent heterogeneity contributing an unavoidable O(ζ_f^2) term.

  • Functional setup: The analysis uses a finite reference measure μ as the common prediction space for heterogeneous peers and applies the actual preconditioned KD update.A4 separates the parameter-to-logit chain rule from the frozen-kernel condition, while A10 controls components outside peer-specific tangent spaces.
  • Consensus contraction: Random-edge KD contracts function disagreement in a uniformly equivalent logit metric, with a defect capturing task drift, stochasticity, and capacity limitations.The metric is positive definite and uniformly equivalent to arithmetic logit disagreement; the consensus recursion includes a critical first-order local task-force term.
  • Heterogeneity defect: O(ζ_f^2) is unavoidable for implemented non-IID local CE updates because heterogeneous quadratic objectives can produce a nonzero consensus fixed point even when B_f = 0 and stochastic noise is zero.The theorem therefore separates stochastic discretization, capacity, and deterministic task heterogeneity rather than attributing all neighbourhood effects to η.
  • Task descent: Functional task descent holds under the task-side assumptions when the task margin is positive, with 0 ≤ α < 1 and activation correction ensuring uniform peer averaging.Inverse activation weights compensate for nonregular graphs, and the descent proof does not absorb an η-independent conditioning error into O(η).

Experiments

The experiments test Theorem 2’s observable predictions in asynchronous P2P KD, measuring function-space consensus, stationarity, rates, and task accuracy across heterogeneous peer rosters. KD contracts disagreement across all tested roster types, while stationarity follows 1/T-order-or-faster transients and the step-size sweep reveals the predicted neighbourhood tradeoff.

  • Experimental design: 25 trajectories record function-space consensus, a finite-reference stationarity proxy, rate analysis, and task accuracy every 1,000 edge activations.The experiments target observable implications of Theorem 2 rather than benchmarking specialized decentralized optimizers.
  • Consensus: 40.2–61.0× contraction reduces function-space disagreement across homogeneous, width-heterogeneous, and mixed-family rosters, unlike isolated controls.Terminal KD disagreement is 2.81×10−4 to 4.10×10−4, versus 1.36 × 10−2 to 1.51 × 10−2 for isolated controls.
  • Stationarity and rate: 0.88–1.90 late-transient exponents support stationarity decay at the 1/T order or faster across the eight Setting A–D main runs.Four values lie in [0.88, 1.03], while Set-C values are 1.21 and 1.90 and Set-D values are 0.88 and 0.91.
  • Neighbourhood signatures: (1.10, 1.17, 1.25) × 10−3 at η = 0.0125 falls to (3.87, 3.98, 4.08) × 10−4 at η = 0.05, then rises slightly at η = 0.1.The sweep exhibits finite-horizon competition between the 1/(ηT) transient at small η and the discretization neighbourhood at large η.
  • Task quality and controls: 0.535–0.729 main KD ensemble accuracies for seed 0 remain well above chance across every roster, indicating consensus does not collapse to a uniform predictor.D-SGD is reported only for Setting A as an appendix mechanism reference, not for a performance claim.

Conclusion

The paper establishes conditional convergence theory for asynchronous P2P knowledge distillation in heterogeneous serverless networks and reports diagnostics consistent with its predicted rate and neighbourhood tradeoff. It also identifies assumptions and measurement limitations that constrain the theorem’s scope and empirical interpretation.

  • Empirical validation: 40.2–61.0× reduction in function-space disagreement occurs across all CIFAR-10 experiments with KD.The diagnostics are consistent with the theoretical prediction across the reported network settings.
  • Empirical validation: Late-transient exponents near or above one and the step-size sweep’s predicted transient–neighbourhood tradeoff support the convergence diagnostics.These observations accompany the reported disagreement reductions in the CIFAR-10 experiments.
  • Limitations: The theorem depends on stable kernels, bounded staleness, A4 for the parameter-to-logit bridge, and A10 for restricted alignment.These assumptions expose where the proof uses network geometry but are not automatic for arbitrary feature learning or aggressive step sizes.
  • Limitations and future work: The experiments estimate a combined terminal neighbourhood rather than separate O(η), B_f^2, and O(ζ_f^2) contributions, using finite reference support for KD and evaluation.Future work should measure A4/A10 residuals, separate KD and held-out evaluation supports, and estimate B_f through reachable-class probes.

Appendix

The appendix is self-contained, presenting Assumptions A1–A10, a proof sequence culminating in Theorem A1, and detailed experimental setups and statistical analyses.

  • Appendix: The appendix states Assumptions A1–A10, develops named lemmas and propositions, culminates in Theorem A1, and details experiment setups and results with comprehensive statistical analysis.It is explicitly described as self-contained.

Notations Clarification

This section defines the mathematical notation, variables, operators, and constants used in the convergence proof. It serves as a reference for the federated knowledge distillation appendix.

  • Notations Clarification: The section specifies the mathematical notation used throughout the convergence proof.It functions as a notation reference for the appendix.
  • Notations Clarification: It defines the variables and operators appearing in the proof.These definitions support consistent interpretation of the convergence analysis.
  • Notations Clarification: It records the constants used in the federated knowledge distillation appendix.The section covers every constant used throughout the proof.

Network and Graph Topology

The network is modeled as an undirected graph of peers and communication links, with directed orientations used to represent active communication events. Its Laplacian spectrum, especially algebraic connectivity λ2, characterizes information mixing across the graph.

  • N denotes the total number of peers, while i, j, and k index specific peers.
  • E is the set of undirected communication links, and directed edge set E⃗ contains both orientations for every undirected edge.
  • The graph Laplacian is L = D − A, with eigenvalues λ1, λ2, …, λmax; algebraic connectivity λ2 dictates information-mixing speed.Here, D is the degree matrix and A is the adjacency matrix.

Time and Optimization Parameters

The section defines the event horizon and optimization parameters governing asynchronous updates and knowledge distillation, including the effective KD contraction step w = ηατ.

  • t counts individual edge activations, while T denotes the total number of events or steps in the horizon.
  • η is the base parameter-space SGD step size, and α ∈ (0, 1) balances local task loss against KD loss.
  • τ scales logits before softmax application as the softmax temperature scalar.
  • w = ηατ is the effective step size for KD logit contraction.

Data, Functions, and Prediction Spaces

The framework represents peers through parameters, logits, and temperature-scaled prediction probabilities evaluated on a finite reference measure. It also defines peer-averaged predictions, teacher targets, probability bounds, and the hypothetical global-task optimum.

  • Core objects: The reference measure µ is a finite evaluation dataset used to measure logits and predictions, while θ_i denotes peer i’s network parameters.The notation also includes initialization parameters θ_i^0 and parameter updates Δθ_i.
  • Function representations: Logits Z_i are pre-softmax outputs evaluated over µ, with Z_i^t denoting peer i’s logits at time t and Z̄ the network-wide mean logits.
  • Prediction representations: Prediction probabilities P_i are computed as softmax(Z_i/τ), where τ is the temperature parameter.
  • Targets and aggregates: Teacher targets p_j^T may use peer j’s live prediction or an exponential moving average, alongside the time-t mean prediction probability p̄^t.
  • Assumptions and optimum: The framework enforces a strictly positive class-probability lower bound p_min and defines q⋆ as the hypothetical optimal target distribution for the global task.

Jacobians and Neural Tangent Kernels (NTK)

This section defines the Jacobians and kernel operators used to connect parameter updates with logits and prediction probabilities. It distinguishes logit-space and prediction-space NTKs and introduces the cross-Jacobian operator for KD-driven parameter updates.

  • Jacobian definitions: Jz_i is the logit Jacobian with respect to parameters, while Jp_i is the prediction-probability Jacobian.These Jacobians describe parameter sensitivity in logit and prediction spaces, respectively.
  • Neural Tangent Kernels: Θ_i is the logit-space Neural Tangent Kernel, and Θ0_i denotes its initial value at θ0_i.The section defines Θ_i from the logit Jacobian and separately identifies the initialization-time kernel.
  • Neural Tangent Kernels: K_i is the Neural Tangent Kernel in prediction space, defined from Jp_i.This kernel transfers the NTK construction from logits to prediction probabilities.
  • KD update operator: H_i maps prediction-space KD-loss gradients into parameter updates through a cross-Jacobian operator.The operator is defined as H_i = τJp_i.

Tangent Spaces and Projections

This section defines the ambient logit space, model-reachable tangent spaces, and orthogonal projections that separate movable from immovable directions. It also introduces prediction-space tangents and the softmax metric tensor.

  • H_z is the full ambient vector space of evaluated logits.
  • T_z_i is the logit tangent space, representing logit directions the model can physically move in.
  • Π_z_i projects orthogonally onto T_z_i, while I − Π_z_i projects onto the normal space of directions the model cannot move.
  • T_p_i and Π_p_i denote the prediction tangent space and its orthogonal projection matrix.
  • M(P) = diag(P) − PP^T defines the softmax metric tensor.

Losses, Gradients, and Residuals … Scope of the Empirical Evidence

The paper formalizes asynchronous P2P knowledge distillation through prediction-space contraction, with convergence governed by structural bridge assumptions, Lyapunov recursions, and explicit error floors. Experiments support prediction disagreement contraction across homogeneous, width-heterogeneous, and mixed-family rosters, while narrowing claims to observable consequences of the conditional theory.

  • Strict Lyapunov Coefficients (Lemma 5 in Main Paper, Lemma 9 in Appendix): The convergence proof combines consensus and task-descent recursions through a strict Lyapunov function under the small-gain condition bF CG < aF c0.The Lyapunov function is Lt = Rt + λVW (Zt), and the condition yields strictly positive descent factors.
  • Decentralized Optimization and the Type Mismatch: KD replaces undefined cross-architecture parameter averaging with contraction in a shared finite-support prediction space.The analysis preserves the graph viewpoint while changing the state variable from parameters to evaluated predictions.
  • Assumption Validity and Diagnostics: A4 and A10 are local, trajectory-dependent bridge assumptions connecting parameter updates to finite-support logits and requiring task gradients and KD pulls to align with realizable tangent directions.These assumptions are sufficient analytical conditions, not universal facts about finite neural networks.
  • Checkpoint-Level Diagnostics: The empirical evidence validates observable consequences of the conditional theory rather than establishing that neural CIFAR-10 trajectories satisfy A4 and A10.The narrowed interpretation emphasizes prediction-space contraction and finite-reference stationarity proxies.
  • Convergence Proof: Theorem A1 establishes constant-step convergence under A1–A10, connectedness, frozen-kernel conditions, threshold constraints, and 0 < η ≤ ηmax.The theorem provides positive constants controlling the convergence bounds under these assumptions.
  • Empirical Convergence-Rate Diagnostics: 1.21 and 1.90 are the late 85% dynamic exponents for setting C’s seed 0 and extended seed-1 histories, while settings A and B remain close to 1.00.The mixed-family setting D also reaches the consensus neighbourhood, but stationarity traces are less uniform across rosters.
  • Statistical Analysis of the Data Results: 40.9×, 47.5×, and 41.0× reductions in disagreement occur for settings B–D, compared with isolated local training; the mean reduction across A–D is 41.4×.KD finishes between 2.88 × 10^-4 and 3.74 × 10^-4, while isolated runs finish between 1.36 × 10^-2 and 1.51 × 10^-2.
  • Scope of the Empirical Evidence: The learning-rate sweep and four rosters support a finite-horizon neighbourhood tradeoff, with terminal consensus values remaining on the order of 10^-4 across homogeneous, width-heterogeneous, and mixed-family settings.The evidence supports the stated scope: random-edge KD contracts prediction disagreement across all three roster types, while the sweep exhibits the predicted tradeoff.
Loading 2609.01952v1…