Decentralized Federated Learning for Heterogeneous Multi-Task Semantic Communication

Lin Yin; Tiejun Lv; Weicai Li; Xi Yu; Xiaoyu He

Source-linked AI summary

Decentralized Federated Learning for Heterogeneous Multi-Task Semantic Communication

Lin Yin, Tiejun Lv, Weicai Li, Xi Yu, Xiaoyu He

arXiv:2608.15256v1Aug 15, 2026cs.AIcs.LG

TL;DR

Existing semantic communication and decentralized learning must support heterogeneous concurrent tasks without centralized coordination. The paper introduces similarity-aware routing and aggregation to filter mismatched updates, achieving a 4.77% global relative improvement at the analytically derived optimal depth while outperforming decentralized baselines.

Problem
Existing semantic communication largely assumes single-task settings, leaving heterogeneous multi-task edge learning to address concurrent tasks without centralized orchestration.
Method
The paper proposes similarity-aware routing and aggregation that separates task paths and filters structurally mismatched updates in decentralized multi-task learning.
Results
4.77% global relative improvement was achieved at the analytically derived optimal depth, consistently outperforming decentralized FedAvg, FedAMP, and heuristic baselines.
Takeaways & Limitations
Aggregation depth has a U-shaped trade-off: deeper mixing reduces topology-dependent variance but amplifies overconsensus bias, while task composition constrains effective depth.

Abstract

from arXiv · show

Collaborative training in distributed semantic communication (DSC) networks typically relies on decentralized federated learning (DFL). However, pushing topology-agnostic aggregation into heterogeneous, multi-task environments creates a fundamental bottleneck: it drives negative transfer and overconsensus bias (OCB). This paper introduces a personalized DSC framework that cuts off this cross-task interference. At the node level, a policy-driven multi-path routing mechanism separates task-specific features from shared representations to preserve local fidelity. Across the network, we deploy a "communicationwhile- aggregation" protocol. It calibrates a column-stochastic consensus matrix using task affinities. This limits the system to absorbing complementary knowledge while actively blocking mismatched parameter updates. To bound the convergence, we derive a unified Lyapunov drift analysis. We reveal a strict Ushaped trade-off: deeper topological mixing reduces variance but amplifies structural OCB. Resolving this tension yields a closed-form expression for the optimal aggregation depth. We evaluate the proposed framework on NYU-v2, where the results reveal a clear trade-off between insufficient aggregation and excessive topological mixing. At the analytically derived optimal aggregation depth, our method achieves a 4.77% global relative improvement over the no-aggregation baseline and outperforms decentralized FedAvg, FedAMP, and heuristic max aggregation. We further evaluate the framework on Taskonomy and imperfect wireless links to examine the effects of network-size variation and wireless-link reliability.

I. Introduction … A. Distributed Semantic Communication Model

The paper develops a decentralized, personalized semantic-communication framework for heterogeneous multi-task edge intelligence. It combines task-aware semantic routing, similarity-aware peer aggregation, and a wireless mesh model to reduce cross-task interference while preserving local task fidelity.

I. Introduction: Semantic communication transmits task-relevant representations instead of raw bit streams, reducing bandwidth overhead but leaving most existing architectures limited to single-task settings.Real-world edge applications require concurrent heterogeneous tasks, while centralized multi-task orchestration incurs communication, latency, and privacy costs.
A. Related Work and Motivations: Mixing mismatched tasks in decentralized topologies causes negative transfer, client drift, and over-consensus bias, while dense adaptive aggregation exceeds edge bandwidth constraints.These limitations motivate screening and reweighting neighborhood updates according to semantic compatibility.
B. Contributions and Organization: The proposed framework moves personalization into semantic representation learning through a multi-path encoder with a learnable feature-routing policy.The encoder balances shared semantic extraction with task-specific features, reducing dense parameter exchange while retaining local modeling capacity.
B. Contributions and Organization: Task-specific policy vectors calibrate topological aggregation weights so mismatched updates are blocked and semantically aligned peers receive greater influence.This similarity-aware protocol is designed specifically to counter over-consensus bias and negative transfer.
II. System Model: The framework models clients as a decentralized mesh that jointly trains local multi-path encoders and task-specific decoders through direct neighborhood exchanges.Information from non-neighboring clients propagates through repeated one-hop aggregation over the connected graph.
A. Distributed Semantic Communication Model: Clients extract and exchange task-specific semantic features over scheduled one-hop links rather than transmitting raw data or dense model weights.The communication graph uses a TDMA-based schedule constructed from conflicts among feasible one-hop links.
A. Distributed Semantic Communication Model: Packet loss affects decentralized communication through retransmission latency: successful reception preserves the decoded semantic feature, while losses increase transmission time.The abstraction represents link reliability through packet-level loss and avoids detailed physical-layer modulation, coding, or outage analysis.

B. Design of Multi-Task Joint Source-Channel Coding · 1) Policy-Driven Multi-Path Encoder:

The framework uses policy-driven routing through a shared encoder to extract task-specific semantic representations while limiting gradient interference and structural overconsensus bias. Task-path overlap provides a lightweight compatibility signal for affinity-aware decentralized aggregation, complemented by independent task-specific decoders.

B. Design of Multi-Task Joint Source-Channel Coding: A shared backbone of K sequential blocks uses each task’s learnable binary policy vector u_t∈{0,1}^K to determine its execution path.The policy avoids instantiating independent backbones for heterogeneous tasks.
1) Policy-Driven Multi-Path Encoder:: For client i and task t, u_t,k activates or bypasses block k as the representation evolves to the final semantic feature X_t.The initial input is α_i,t,0=x, and each active block applies its client-parameterized transfer function F_i,k(·).
1) Policy-Driven Multi-Path Encoder:: Task-specific routing restricts active encoder parameters to a policy-governed subset of the global shared parameter space.Disentangled execution paths prevent gradient interference and structural OCB among disjoint tasks.
1) Policy-Driven Multi-Path Encoder:: The policy-driven multi-path encoder adaptively extracts task-specific semantic representations from shared blocks before independent task-specific decoders reconstruct task outputs.The architecture is summarized schematically in Fig. 2.
1) Policy-Driven Multi-Path Encoder:: Overlapping active blocks indicate overlapping shared-backbone updates, making routing-policy overlap a lightweight measure of compatibility in the shared representation space.The policy therefore serves as a compact descriptor of each task’s use of the shared encoder.
1) Policy-Driven Multi-Path Encoder:: The task-affinity cosine score normalizes shared activated-block counts, preventing tasks that activate more blocks from receiving compatibility advantages solely due to path size.Its interpretation is based on how tasks select and share encoder blocks.
1) Policy-Driven Multi-Path Encoder:: Symmetric best matching evaluates semantic alignment between neighboring clients, and S_i,n guides aggregation toward overlapping pathways while filtering disjoint task portfolios.Each task also retains an independent lightweight DeepLab-style dilated convolutional decoder with parallel multi-scale branches and linear fusion.

2) Task-Specific Decoders: · C. Personalized and Global Optimization Objectives · III. Decentralized Training for the DSC Model

The framework separates task-specific decoders from decentralized aggregation to preserve local task fidelity, while optimizing personalized client risks and a network-wide consensus objective. Its communication-while-aggregation protocol alternates local semantic updates with similarity-aware peer-parameter fusion.

2) Task-Specific Decoders:: Task-specific decoder parameters are decoupled from neighbor aggregation, preserving high fidelity to each client’s local tasks.Only shared representations are exposed to decentralized aggregation.
C. Personalized and Global Optimization Objectives: Client-level empirical loss is evaluated on the dataset corresponding to each local task and aggregated using predefined task-importance weights.The local task portfolio is represented by Λ_i = {λ_i,t | t ∈ T_i}.
C. Personalized and Global Optimization Objectives: Each client jointly updates its localized encoder pathway and task-specific decoder during local gradient descent, but only the shared encoder backbone enters network consensus.Task weights λ_i,t determine the relative importance of task t, and the criterion may be task-specific, such as segmentation cross-entropy.
C. Personalized and Global Optimization Objectives: The aggregate local objective optimizes the shared backbone while treating localized decoders as conditionally optimized internal variables.This formulation preserves personalization while defining the optimization target for shared parameters.
C. Personalized and Global Optimization Objectives: The global objective is defined as the uniform average of client-specific losses across the entire mesh network.The global parameter collections aggregate shared parameters, task weights, and local datasets across clients.
C. Personalized and Global Optimization Objectives: DFL jointly pursues localized personalization and network-level consensus, with their relationship regulated by within-round aggregation depth J.The paper characterizes this coupling in its subsequent convergence analysis.
III. Decentralized Training for the DSC Model: Every communication round alternates local gradient-based semantic updates with peer-parameter fusion through dynamic, similarity-aware weighting.Knowledge sharing occurs exclusively through peer-to-peer communication in the fully decentralized DSC network.

A. Local Model Training · B. Local Model Aggregation

Clients train personalized models locally through repeated SGD epochs, then exchange shared representations and task-specific policies through semantically weighted peer-to-peer aggregation. The finite aggregation depth balances variance reduction against structural overconsensus bias in heterogeneous multi-task networks.

A. Local Model Training: Clients iteratively update aggregatable parameters θ_i via stochastic gradient descent on local datasets across R communication rounds and E local epochs.Each round begins from the client’s locally aggregated model from the preceding round.
A. Local Model Training: The local-training output after E epochs provides each client’s round-r model for subsequent decentralized aggregation.The output parameter is denoted by the local training formulation after E epochs.
A. Local Model Training: An idealized synchronized reference trajectory evaluates gradients at the instantaneous network average to anchor consensus-error and aggregation-trade-off analysis.This reference replaces each client’s local state with the network average during gradient evaluation.
B. Local Model Aggregation: The communication-while-aggregation protocol sends task-specific policy vectors and shared encoder parameters to one-hop neighbors for J consecutive peer-to-peer aggregation steps.The gossip design is scalable, fault-tolerant, and avoids a central bottleneck.
B. Local Model Aggregation: Column-wise softmax converts raw task similarities into normalized aggregation coefficients, prioritizing semantically aligned updates while preserving the global parameter average.The self-similarity is defined as the maximum possible affinity, and outgoing weights sum to one.
B. Local Model Aggregation: Successive neighborhood aggregation incorporates multi-hop information without global connections, while poorly aligned neighbor updates can cause aggregation-induced negative transfer.Negative transfer arises from peer-to-peer model mixing under heterogeneous task portfolios and can degrade local task performance.
B. Local Model Aggregation: Repeated aggregation exponentially suppresses topology-dependent variance but, as J grows without bound, pulls personalized models toward a weighted global state and amplifies structural OCB.These opposing effects require an optimal finite aggregation depth J*.

IV. Theoretical Formulation and Convergence Analysis · A. Composite Objective for Global Generalization and Personalization · B. Convergence Analysis

The paper defines a composite Lyapunov objective that couples global optimization with client personalization through consensus error. Its convergence analysis shows that aggregation depth J produces a U-shaped trade-off between variance reduction and structural overconsensus bias, yielding an optimal depth choice.

A. Composite Objective for Global Generalization and Personalization: The composite objective couples loss at the round-mean iterate with a post-aggregation consensus-error term to jointly represent global learning and personalization.This objective supports convergence guarantees while characterizing how aggregation depth J affects steady-state performance.
A. Composite Objective for Global Generalization and Personalization: Because W is strictly column-stochastic, J intra-round topological mixings preserve the global arithmetic mean and retain the global mean state after local training.The network average after aggregation exactly coincides with the global mean immediately after the local training phase.
A. Composite Objective for Global Generalization and Personalization: The exact loss decomposes into global-mean loss and a personalization-induced variation term, whose deviation is dominated by network consensus variance.The consensus error provides a tractable measure of geometric disagreement across clients.
A. Composite Objective for Global Generalization and Personalization: Minimizing Φr,J drives the network mean toward a global stationary point while constraining personalized models within a bounded consensus neighborhood.The penalty weight α incorporates the effective smoothness constant Leff and Young’s inequality scaling factors.
B. Convergence Analysis: Structural heterogeneity makes deeper mixing potentially move local models away from preferred personalized optima, creating the persistent OCB term.Deeper mixing can reduce consensus variance while amplifying bias from disagreement among client-specific optima.
B. Convergence Analysis: The consensus-error recursion separates disagreement from local training, gossip contraction, gradient noise, statistical heterogeneity, and irreducible OCB.Substituting this evolution into a smoothness-based descent inequality yields a unified Lyapunov drift bound.
B. Convergence Analysis: Under the stated assumptions, the composite objective converges linearly to a steady-state error floor with a J-dependent convergence characterization.The Lyapunov drift couples global optimization progress with consensus-error contraction, and the PL condition yields the linear rate and steady-state floor.
B. Convergence Analysis: Deeper aggregation suppresses topology-dependent consensus variance but inflates structural OCB, producing a U-shaped error floor and requiring an optimal depth J∗.The theorem frames J∗ as a critical design choice for resource-constrained edge deployments.

C. Optimal Aggregation Depth via Upper Bound Minimization

The section formulates aggregation depth J as a constrained discrete optimization variable by isolating its variance-reduction and structural-bias effects. It derives a condition-dependent optimal depth and translates the continuous solution into an integer, hardware-capped deployment rule.

Constrained optimization: Hardware and bandwidth budgets restrict J to a discrete feasible domain capped by Jmax.The admissible depth must also satisfy the limits established in Lemma 2.
Bias–variance trade-off: The simplified objective captures a U-shaped trade-off: deeper mixing reduces variance but increases structural bias from overconsensus.The variance fraction A/(1 −ρ2J) decays as J grows, while αB(1 −ρ2J) increases.
Bias–variance trade-off: In highly heterogeneous environments, increasing gossip depth initially suppresses gradient noise but eventually causes performance to collapse when OCB dominates the error floor.The opposite limiting behavior occurs in homogeneous settings, where the bias penalty disappears and larger J remains beneficial.
Theorem 2: When αB > A, the relaxed continuous objective has a unique global minimizer; when αB ≤ A, Bound(J) decreases monotonically with no finite upper-bound optimum.The first condition corresponds to sufficiently strong structural heterogeneity, while the second corresponds to a highly homogeneous network.
Deployment rule: For deployment, the integer aggregation depth is selected by testing the two nearest integers to the continuous root J∗ and is strictly capped by Jmax.This rule maps the theoretical minimum to practical edge-bandwidth constraints.

V. Experiments

The experiments evaluate similarity-aware aggregation in personalized multi-task client networks against standard peer-to-peer baselines. They sweep aggregation depth to test convergence and the theoretically derived optimal mixing depth, then examine robustness to network-size variation and imperfect conditions.

Experimental objectives: The evaluation compares similarity-aware aggregation with standard peer-to-peer baselines in heavily personalized client networks.The stated goal is to verify multi-task performance gains.
Aggregation-depth study: The experiments sweep aggregation depth J to observe convergence behavior.This directly tests the aggregation-depth dependence of the method.
Aggregation-depth study: The observed convergence behavior is used to validate the theoretically derived optimal mixing depth.The target theoretical result is the optimal mixing depth derived in Section IV-C.
Robustness evaluation: Additional experiments examine whether the gains remain stable under network-size variation and imperfect conditions.The supplied passage introduces these robustness examinations but does not specify the imperfect condition.

A. Experimental Setup · B. Results and Analysis

Experiments use NYU-v2 as the main testbed and Taskonomy for supplementary multi-task and network-size evaluation. Results show that task-aware aggregation improves collaboration, but performance depends on aggregation depth, task composition, client data, network size, and wireless conditions.

A. Experimental Setup: NYU-v2 evaluates segmentation, depth estimation, and surface-normal prediction using 795 training and 654 test images with dense annotations.Taskonomy supplements these experiments with segmentation, surface normals, keypoints, and edge detection while varying clients over N ∈[2, 30].
A. Experimental Setup: The evaluation tracks global loss and task-specific metrics, while relative improvement is measured against a strict no-aggregation, no-transfer baseline.Negative relative improvement indicates that aggregation degrades the corresponding task or client.
B. Results and Analysis: Under shallow aggregation, decentralized FedAvg improves overall performance by 1.17%, heuristic max improves Seg by 2.42% and Sn by 1.99% but reduces Dep by 0.12%, and FedAMP reaches 1.77%.The proposed method achieves a more balanced overall improvement of 2.11%.
B. Results and Analysis: Relative improvement follows a U-shaped relationship with aggregation depth: it first increases as broader exchange reduces variance, then declines when excessive mixing amplifies structural overconsensus bias.The optimal depth shifts with task composition; high-affinity “sn dep” tolerates deeper mixing better than lower-affinity or higher-dimensional portfolios.
B. Results and Analysis: Clients with richer local datasets attain larger peak collaborative gains, whereas data-scarce clients achieve smaller peaks and degrade faster as aggregation depth increases.The results associate data scarcity with greater vulnerability to externally induced overconsensus bias.
B. Results and Analysis: On Taskonomy, the proposed method achieves the highest Δall across tested network sizes, with changes from N=10 to N=20 of 0.09%, 0.10%, 0.13%, and 0.14% for FedAvg, max, FedAMP, and the proposed aggregation, respectively.The results suggest limited sensitivity to network-size variation when local communication density is fixed.
B. Results and Analysis: The proposed method maintains approximately 44.71 MB communication volume, while routing-policy vectors add only 96 bytes for a client assigned three tasks.This yields improved task performance with essentially the same communication volume and marginal latency overhead.
B. Results and Analysis: Higher regional SNR lowers packet-loss probability, while latency depends on retransmissions, backbone size, link density, scheduling cost, and channel quality.At low SNR, larger backbones suffer from more packets; at moderate SNR, shorter reliable links can offset added scheduling cost.

VI. Conclusion

The paper identifies a U-shaped steady-state error trade-off in decentralized multi-task optimization and addresses it with similarity-aware routing that filters structurally mismatched updates. At the analytically derived optimal aggregation depth, experiments support the analysis and show improved decentralized collaboration, with future work targeting realistic dynamic edge settings.

VI. Conclusion: Increasing aggregation depth reduces topology-dependent variance but amplifies overconsensus bias (OCB), producing a U-shaped steady-state error trade-off.This trade-off motivates limiting excessive topological mixing.
VI. Conclusion: Similarity-aware routing selectively filters structurally mismatched updates, enabling more reliable and effective decentralized collaboration.The strategy is designed to address the effects of structural heterogeneity in decentralized multi-task optimization.
VI. Conclusion: 4.77% global relative improvement was achieved at the analytically derived optimal depth, while the framework consistently outperformed decentralized FedAvg, FedAMP, and heuristic baselines on NYU-v2.The NYU-v2 experiments also showed mitigation of negative transfer.
VI. Conclusion: Taskonomy results showed that the proposed aggregation strategy was only mildly affected by moderate changes in network size.This evaluates robustness to network-size variation.
VI. Conclusion: Future work will address time-varying wireless channels, dynamic mesh topologies, and adaptive online aggregation-depth control.These extensions target more realistic edge settings.

Appendix · A. Proof of Lemma 1

The proof derives a cross-round consensus-error recursion by combining bounds for local-training disagreement and local-aggregation disagreement. It separates expansion, geometric contraction, and an irreducible structural-bias floor caused by decentralized mixing.

A. Proof of Lemma 1: The proof begins by relating the next round’s consensus error to the current error through local-training and local-aggregation disagreement bounds.This establishes the cross-round recursion used throughout the lemma.
A. Proof of Lemma 1: During local training, client updates follow stochastic gradient descent, while the mean model evolves according to the average gradient.The deviation of each local model from the mean is tracked across local steps.
A. Proof of Lemma 1: Gradient noise and statistical heterogeneity generate disagreement before aggregation, and recursive application over local steps yields a bound using constants a1 and a2.The derivation uses Jensen’s inequality and Assumption 3 to bound stochastic noise and heterogeneity contributions.
A. Proof of Lemma 1: Local aggregation applies Ψr,J = W^JΨr,0, with strict column stochasticity preserving the arithmetic mean.Consensus error is measured after aggregation using the mean-centering projection.
A. Proof of Lemma 1: The mixing matrix contracts disagreement orthogonal to the average-consensus subspace at geometric rate ρ.The analysis decomposes the error relative to dispersed client optima to capture aggregation effects.
A. Proof of Lemma 1: Column-stochastic mixing can create a persistent structural shift when client optima are dispersed, producing an overconsensus-bias floor despite suppressing network disagreement.This residual can displace iterates from their respective empirical minimizers.
A. Proof of Lemma 1: The finalized recursion has a dual effect: mixing suppresses initial disagreement and gradient noise while introducing an irreducible bias determined by structural network heterogeneity.Substituting the local expansion bound into the gossip contraction bound completes the proof.

B. Proof of Lemma 2 · C. Proof of Theorem 1

Lemma 2 bounds the per-round Lyapunov drift by descent, variance, disagreement, and OCB terms under a monotonicity condition. Theorem 1 then establishes linear convergence to a steady-state error floor and isolates the dual effect of gossip depth J, enabling communication-depth optimization.

B. Proof of Lemma 2: The per-round drift is analyzed through the Lyapunov function Φr,J = L(¯θr,E) + αEcons,r,J over E local steps.The analysis applies Leff-smoothness to the objective and tracks expected change across local updates.
B. Proof of Lemma 2: The decentralized descent bound separates gradient norm, stochastic variance, and model disagreement contributions.These terms arise from standard decentralized optimization bounds.
B. Proof of Lemma 2: The expected disagreement is bounded by 2EEcons,r,J + 2a2η2E2(bΓ + σ2) before substitution into the total drift.The bound combines local-drift components with the consensus recursion from Lemma 1.
B. Proof of Lemma 2: Monotonicity requires the coefficient of EEcons,r,J to be strictly negative, with α chosen sufficiently large so cΦ,3 ∈(0, 1).The resulting drift bound consolidates noise, heterogeneity, disagreement contraction, and OCB contributions.
C. Proof of Theorem 1: Theorem 1 decouples intrinsic gradient noise from consensus-induced noise, whose residual disagreement component is proportional to the spectral contraction ρ2J.The constants cbase and ccons characterize these two noise sources.
C. Proof of Theorem 1: Under the admissible condition in (35), consensus error contracts at least as fast as the global objective, yielding a linear recurrence for the composite Lyapunov function.The proof uses the PL condition and merges the negative descent terms into the composite function.
C. Proof of Theorem 1: The steady-state error is decomposed into three components, while factoring out (1−ρ2J) exposes the spectral-gap dependence of consensus variance.The resulting constants Cbase, CΓ, and CΩ recover the exact upper bound in Theorem 1 and establish the basis for optimizing J.

D. Proof of Theorem 2

The proof minimizes the steady-state objective after the transient term vanishes, then establishes a unique interior stationary point through a convex reparameterization. Strict convexity and the monotonic mapping back to J confirm the closed-form J* as the unique global minimizer and mathematically establish the U-shaped trend.

Objective reduction: As r →∞, the transient term vanishes, reducing the analysis to minimizing the steady-state objective after excluding J-independent noise Cbase.This reduction follows from Theorem 1.
Convex reparameterization: Relaxing J and substituting y ≜ 1 − ρ^2J yields a convex optimization variable for locating the stationary point.The transformed domain is y ∈ (0, 1).
Stationary point: The stationary point is y* = A/(αB), feasible as an interior solution when αB > A, and maps back to the closed-form J*.The mapping uses 1 − ρ^2J* = y* and produces the stated closed-form expression for J*.
Global optimality: Strict convexity and the monotonic relationship between y and J make J* the unique global minimizer, confirming the U-shaped performance trend.The proof concludes that the closed-form solution is globally optimal.