Source-linked AI summary

From Task Mixtures to Specialized Experts

Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari, M. Hadi Amini

arXiv:2610.00580v1cs.LG

TL;DR

The paper studies client data that mixes latent tasks, a form of heterogeneity within as well as across clients. It analyzes task-expert recovery and input-dependent routing, then proposes FedSEE; experiments report improved performance and avoidance of negative transfer among evaluated baselines.

  • Problem

    Federated learning commonly models heterogeneity across clients, while clients may also contain varying mixtures of latent task distributions.

    Source: §1 · p. 1
  • Method

    FedSEE uses task-specific LoRA experts, supervised extraction from mixed client updates, and mixture-aware routing with client-specific calibration.

    Source: §1 · p. 2
  • Results

    FedSEE improves over the strongest baseline by 2.9 points overall and 3.7 points on the worst-served quartile, and avoids negative transfer across the five-point heterogeneity sweep.

    Source: §1 · p. 3
  • Takeaways & Limitations

    The results support explicitly modeling within-client mixture structure across heterogeneity regimes in the evaluated natural language understanding tasks.

    Source: §6 · p. 10

    The exact barycentric characterization holds under squared loss, and the framework assumes the number of underlying tasks is known.

    Source: §6 · p. 10

Abstract

from arXiv · show

In collaborative foundation model fine-tuning, client data is rarely homogeneous. Instead, clients typically possess unknown mixtures of distinct data distributions, or tasks. Conventional federated learning primarily addresses heterogeneity across clients without explicitly resolving latent task mixtures within each client. We study this setting as compound heterogeneity, where data is heterogeneous both across and within clients. We study adaptation over a common frozen representation and show that, when tasks share the same feature geometry, the optimal model for a client's task mixture under squared loss is a convex combination of the optimal models for its underlying tasks. Thus, a single locally trained model represents the client's overall task mixture, while individual inputs may be drawn from different underlying task distributions. This motivates routing inputs to specialized experts, and we show that, when the task optima form a simplex, task-aligned routing achieves lower risk than any single adapted model for genuinely mixed clients. With access to a small set of task-labeled public samples, we derive a convex program to recover task experts and match them to their corresponding tasks. Our routing analysis shows that effective specialization requires input-dependent expert selection aligned with each client's task mixture. Motivated by this analysis, we propose FedSEE. Across our experiments, FedSEE avoids the negative transfer observed in the evaluated baselines and improves performance by 2.9 points overall and 3.7 points for the worst-served quartile.

1 Introduction

Clients can contain different mixtures of latent tasks, so a single locally optimized model captures the mixture rather than each task-specific optimum. The paper motivates task-supervised expert recovery and mixture-aware input routing, and proposes FedSEE to implement them.

Geometry under task mixtures

  • Under shared features and squared loss, each client optimum is a mixture-weighted combination of task optima, so it represents the client's overall task mixture rather than individual task optima.When task optima form a non-degenerate simplex, ideal input routing to the corresponding task optimum achieves strictly lower risk for genuinely mixed clients; the risks coincide for pure clients.Source: §1 · pp. 1–2

Expert recovery and identifiability

  • Averaging mixed-client optima stays inside the simplex, while hard clustering assigns mixed clients to single clusters rather than recovering task-expert vertices.The simplex geometry identifies task experts with vertices and client optima with barycentric points.Source: §1 · p. 2, Figure 1 · p. 2
  • A small task-labeled public set supplies task-specific losses for a convex program that recovers experts from pooled client uploads and identifies their task correspondence.Without task-indexed information, the expert set may be recovered but its members cannot be matched to tasks.Source: §1 · p. 2

Mixture-aware routing

  • Specialization requires average expert usage to reflect each client's task mixture and input-dependent routing to select experts for individual inputs.Uniform average expert usage is incompatible with exact specialization when a client's task mixture is non-uniform.Source: §1 · p. 2

FedSEE

  • FedSEE combines task-specific LoRA experts, supervised extraction, and mixture-aware routing with client-specific calibration.Its extraction weights indicate each client's contribution to recovered task experts and serve as a proxy for its latent task mixture.Source: §1 · p. 2
  • FedSEE improves over the strongest baseline by 2.9 points overall and 3.7 points on the worst-served quartile, while avoiding negative transfer across the five-point heterogeneity sweep.It is the only evaluated method to avoid negative transfer across that sweep.Source: §1 · p. 3

2 Problem Formulation

The formulation models each client's data as an unobserved mixture of shared task distributions and studies personalization through a parameter block over a shared frozen representation. It defines client and task risks and optima, then frames learning as expert recovery and input-dependent routing, with task-labeled public data used for task-indexed extraction.

Client and task model

  • Each client has an unobserved mixture of shared task distributions, with mixture proportions that vary across clients.Personalization acts through a parameter block on a shared frozen representation.Source: §2 · p. 3

Risks and optima

  • The formulation defines task and client population risks for any non-negative measurable loss, then specializes exact geometry to squared loss.For smooth general losses, the optimum is described as an approximate barycenter whose deviation is controlled by task-curvature dispersion.Source: §2 · p. 3

Recovery objective

  • In the finite-sample recovery abstraction, clients upload locally trained estimates and the server seeks to recover the task experts from those uploads.The server also has access to public reference sets.Source: §2 · p. 3

Routing and supervision

  • The learning problem couples expert recovery with input-dependent routing, which combines expert parameters according to each input's router weights.A small labeled reference set for each task makes extraction task-indexed.Source: §2 · p. 4
  • Task-specific public losses associate recovered experts with tasks and guide expert extraction.The reference data consists of labeled samples drawn independently from each task distribution.Source: §2 · p. 4

3 A General Framework for Federated Mixture Personalization

The framework analyzes federated personalization when clients contain mixtures of tasks, first characterizing client-optimum geometry, then expert recovery and routing requirements. Under shared-feature conditions, task-indexed supervision supports expert recovery, while specialization requires input-dependent routing aligned with each client’s mixture.

3.1 Barycentric Geometry of Client Optima

  • Under shared-feature linear regression and squared loss, each client’s optimum is a unique barycentric combination of task optima, with weights given by its task mixture.The result assumes task realizability, a common feature second moment, and non-degenerate features.Source: §3.1 · p. 4

3.2 Supervised Expert Recovery and Identifiability

  • With one near-pure client anchor per task, task-loss minimization over convex combinations of client uploads recovers experts with error bounded by anchor impurity, geometry conditioning, and expert diameter.The population recovery error vanishes for pure anchors; estimated uploads and finite public data add errors that vanish as sample sizes grow.Source: §3.2 · p. 5
  • Task-indexed supervision identifies expert-to-task correspondence, while interior upload pools can be consistent with different expert sets, making sufficient boundary information necessary for upload-only vertex recovery.Near-pure client anchors provide a sufficient form of boundary information.Source: §3.2 · pp. 5–6

3.3 Routing Requirements for Specialization

  • Specialization requires mean expert usage to reflect the client mixture and input-dependent routing to send different tasks to their corresponding experts.Uniform mean usage is incompatible with exact specialization for clients whose task mixtures are non-uniform; a static mixture-weighted router cannot close the mixing gap.Source: §3.3 · p. 6
  • FedSEE uses a contribution-based proxy for the latent client mixture to calibrate mean routing, while task loss governs input-level specialization.The calibration guarantee depends on mixture-estimation and proxy errors.Source: §3.3 · p. 6

4 FedSEE: The LoRA Instantiation

FedSEE instantiates the framework with LoRA, extracting task experts from mixed client updates and then using client-specific mixture information for expert capacity and routing. Its training objective combines task loss with calibration of mean expert usage toward a client-specific proxy.

LoRA instantiation

  • At a single adapted projection, the LoRA formulation matches the framework’s model, while the composed transformer departs from this geometry under a local first-order approximation.The departure is bounded in Appendix D.8.Source: §4 · pp. 6–7

Shared expert

  • FedSEE also keeps an always-on shared LoRA expert outside the routed pool, absorbing shared adaptation into the predictor’s shared component.The shared expert is not routed.Source: §4 · p. 7

Supervised expert extraction

  • FedSEE first averages client LoRA down-projections into a fixed shared projection, then uses task-labeled public losses to extract each task expert from the current client-upload pool.The server jointly optimizes taskwise extraction programs.Source: §4 · p. 7

Mixture-aware capacity and routing

  • Client contribution scores rank recovered experts for inference capacity, while a selected active subset is used for local updates and normalized retained scores form the router-calibration proxy.Experts available at inference but inactive during local updates are detached during backpropagation.Source: §4 · p. 8
  • The local objective pairs task loss, which learns input-dependent assignments, with a KL term that calibrates mean usage toward the client-specific proxy rather than uniform usage.Routers are aggregated across clients after each round.Source: §4 · p. 8

5 Experiments

Experiments evaluate FedSEE on six language-understanding tasks and use synthetic validation to examine the framework’s theoretical predictions. In the compound-heterogeneity benchmark and across a heterogeneity sweep, FedSEE improves accuracy relative to evaluated baselines and local-only training.

5 Experiments

  • Synthetic experiments confirm the predicted linear scaling of expert-recovery error with anchor impurity, supervision’s role in resolving identity ambiguity, and routing’s advantage over uniform calibration.They also validate non-identifiability for sufficiently interior upload pools.Source: §5 · p. 8

5.1 Natural Language Understanding Benchmark

  • The benchmark covers six GLUE/SuperGLUE tasks, with client task proportions drawn from a symmetric Dirichlet distribution and disjoint client validation and test sets.The validation set is used for model selection, while the test set is held out for final personalized evaluation.Source: §5.1 · p. 8

5.2 Compound Heterogeneity at αDir = 0.1

  • 2.90 points: FedSEE raises overall personalized accuracy from 79.75% to 82.65% over the strongest baseline in the αDir = 0.1 benchmark.FedSEE also improves Worst-50% accuracy from 72.46% to 76.54% and Worst-25% accuracy from 69.10% to 72.75%; the strongest baseline differs by metric.Source: Table 1 · p. 9, §5.2 · p. 9

5.3 Where the Advantage Holds: A Heterogeneity Sweep

  • FedSEE achieves the highest federated accuracy in four of five mixture regimes and remains within 0.38 points of the strongest baseline near the uniform endpoint.The closest baseline changes across the sweep, while FedSEE keeps the same task-expert representation and client-dependent routing.Source: §5.3 · p. 9, Table 2 · p. 10
  • 0.65 to 5.33 points: FedSEE’s gains over local-only training increase from pure clients to near-uniform mixtures, while every competing federated method falls below local-only in at least one regime.Figure 3 uses symlog scaling on the gain axis.Source: §5.3 · p. 9, Figure 3 · p. 9

6 Conclusion and Future Work

FedSEE applies task-supervised expert recovery and mixture-aware, input-dependent routing to personalization under compound heterogeneity. The framework’s exact barycentric result is limited to shared-feature squared loss, and the number of tasks is assumed known.

6 Conclusion and Future Work

  • FedSEE extracts task-aligned experts from mixed client updates and uses client-specific mixture calibration to route among them.The conclusion reports improved average and tail personalization and competitive performance from pure to near-uniform client populations.Source: §6 · p. 10
  • Under the shared-feature quadratic model, client optima are mixture-weighted barycenters of task optima, with affine independence inducing simplex geometry.This motivates supervised expert recovery and mixture-aware, input-dependent routing.Source: §6 · p. 10
  • The exact barycentric characterization assumes squared loss, and the framework assumes the number of latent tasks is known.Extending the analysis to deep models trained with cross-entropy and jointly estimating the task count and experts are identified as future directions.Source: §6 · p. 10

A Related Work

Related federated methods model client mixtures, cluster clients, or improve LoRA aggregation, while FedSEE addresses recovery of task-indexed experts from already mixed adaptations. The appendix describes controlled synthetic validation, where client-optimum error decreases with sample size at rates close to n^-1/2.

A Related Work

  • FedSEE uses task-labeled public data to recover and identify experts from mixed client uploads, unlike methods that jointly learn latent components or assign clients to one cluster.Without task-indexed information, task correspondence is unavailable; under the anchor condition, Theorem 2 guarantees recovery of task-indexed experts.Source: A Related Work · pp. 15–16
  • Centralized adapter composition combines already-trained task models, whereas FedSEE extracts latent task experts from adaptations that already mix task information.The related-work discussion also distinguishes FedSEE from federated LoRA methods focused on accurate aggregation and from expert methods that construct clusters.Source: A Related Work · pp. 15–16

B.1.1 Synthetic Setup and Barycentric Validation

  • The synthetic experiments enforce the shared-feature LoRA model and vary sample size, anchor impurity, conditioning, and client mixtures to test theoretical predictions.The setup uses six tasks and 12 clients by default, with least-squares client solutions and simplex quadratic programs for server recovery.Source: §B.1.1 · p. 16, §B.1 · p. 16
  • 0.801 to 0.036: mean client-optimum error falls as local sample size rises from 2 × 10^2 to 10^5, with successive ratios close to n^-1/2.The reported ratios are 2.19, 2.21, 1.99, and 2.29, compared with predictions of 2.24, 2.24, 2.00, and 2.24.Source: §B.1.1 · p. 16

B.1.2 Expert Recovery and Finite-Sample Behavior

Synthetic experiments examine expert recovery under anchor impurity and finite sampling, identifiability with and without anchors, and the risk of constant routing policies. They report that anchors reduce recovery error, while mixture-aware average expert usage avoids the excess risk of uniform balancing for non-uniform mixtures.

B.1.2 Expert Recovery and Finite-Sample Behavior

  • 0.087 to 0.039: recovery error falls as public-set size grows from 10^2 to 1.6 × 10^4; with local data increased from 250 to 1.6 × 10^4, it falls from 0.352 to 0.043.Under exact uploads, error is linear in anchor impurity and vanishes at zero impurity; the finite-sample error halves for each fourfold increase in sample size.Source: §B.1.2 · p. 17

B.1.3 Identifiability and Scaling Limits

  • 3.754 to 3.816: no-anchor recovery error remains nearly unchanged as local sample size rises from 500 to 3.2 × 10^4, while exact-anchor error falls from 0.251 to 0.033.The no-anchor error floor also remains nearly unchanged as federation size grows from 12 to 60.Source: §B.1.3 · p. 17, Figure 5 · p. 18

B.1.4 Mixture-Aware Routing

  • 0 to 26.324: excess risk of a balanced constant router rises as a client mixture moves from uniform toward a vertex; the increase is quadratic in its distance from uniform.In the full pipeline, mixture-aware calibration yields excess risk 0.031, compared with 7.526 under load balancing, over 24 non-anchor clients.Source: §B.1.4 · p. 18

B.2 Geometry and Routing Beyond the Exact Model

In deep-model experiments, client updates retain a task-simplex organization, recovered mixture coordinates track realized proportions, and mixed clients benefit from input-dependent routing. Mixture-aware calibration aligns average expert usage with client composition, while the experimental setup spans multiple tasks, client mixtures, methods, and protocols.

B.2 Geometry and Routing Beyond the Exact Model

  • Pure, edge, and interior client updates occupy task vertices, faces, and the interior of the expert triangle across adapted layers.The organization persists despite changes in triangle shape across layers.Source: §B.2 · p. 19, Figure 6 · p. 19
  • Recovered simplex coordinates correlate 0.814 ± 0.040 with realized task proportions and have mean absolute error 0.157 ± 0.017 across seeds.For clients with exactly two tasks, the recovered mass assigned to the absent task is 0.080 ± 0.021.Source: §B.2 · p. 20, Figure 7 · p. 21
  • Input-dependent routing lowers cross-entropy from 0.625 to 0.497 for edge clients and from 0.679 to 0.554 for interior clients, compared with static true-mixture weights.For pure clients, the true-mixture combination and input-dependent router are nearly identical.Source: §B.2 · pp. 20–21, Table 6 · p. 21
  • 0.791 to 0.253: mixture-aware calibration reduces overall mean-usage mismatch, while validation task cross-entropy remains nearly unchanged at 0.509 and 0.511.The diagnostic freezes task experts and optimizes only the router, isolating mean-usage calibration from its effect on expert training.Source: Figure 8 · p. 21, §B.2 · p. 22

B.3.1 Tasks, Public Data, and Evaluation

  • The main experiments use six GLUE and SuperGLUE classification tasks, task-labeled public reference examples, and disjoint client validation and test sets.Test data is reserved for evaluation after configurations are fixed; accuracy is also reported for the worst-served 50% and 25% of clients.Source: §B.3.1 · p. 22

B.3.2 Compound-Client Construction

  • Client populations range from a deterministic pure-client endpoint to Dirichlet mixtures, with larger concentration parameters producing more similar, near-uniform mixtures.Sampled task weights below 0.05 are zeroed before renormalization, and supply constraints can leave training-data amounts dependent on the sampled mixtures.Source: §B.3.2 · p. 23

B.3.3 Baseline Implementations

  • The comparison includes six federated LoRA baselines spanning global aggregation, personalization, and clustering, using shared data partitions and training budgets where applicable.The baselines include FedIT, FFA-LoRA, FedSA-LoRA, FedDPA-T, IFCA+LoRA, and FedLEASE.Source: §B.3.3 · pp. 23–24

B.3.4 FedSEE Configuration

  • FedSEE fixes a shared LoRA projection, extracts six experts, and by default updates at most one expert per client while making all six available at inference.The appendix also specifies router calibration and shared-expert settings, plus headline and heterogeneity-sweep protocols with distinct backbones and client counts.Source: §B.3.4 · p. 24, §B.3.5 · p. 24, §B.3.6 · p. 26

B.3.7 Ablation Protocol

The ablation protocol isolates design choices by changing one component at a time while holding the remaining settings fixed. Transformer experiments used H100 and A100 GPUs in a single-process federated simulation.

B.3.7 Ablation Protocol

  • Ablations use the headline compound-heterogeneity setup unless a component is directly changed: RoBERTa-large, αDir = 0.1, 10 clients, 10 rounds, and 5 local epochs.Training-capacity experiments separately use distilroberta-base, varying Li = 1 versus Li = 6 while keeping Ui = 6.Source: §B.3.7 · p. 26
  • The public-supervision ablation varies examples per task; other ablations test router calibration, expert count, and removal of the shared expert.Expert counts are M = 3, 6, or 9; the shared-expert ablation removes (As, Bs) while retaining routed experts.Source: §B.3.7 · p. 26

B.3.8 Computational Environment

  • Transformer experiments ran on NVIDIA H100 and A100 GPUs using a single-process federated simulation loop.Source: §B.3.8 · p. 26

B.4 Detailed Results under Compound Heterogeneity

FedSEE leads on three tasks and remains competitive on the others in the RoBERTa-large compound-heterogeneity experiment. The per-task breakdown complements the client-level overall and tail metrics.

  • FedSEE has the highest accuracy on QNLI, WiC, and BoolQ, and remains competitive on SST-2, CoLA, and QQP.The table reports per-task and personalized accuracy under compound heterogeneity (αDir = 0.1) with RoBERTa-large, averaged over three seeds.Source: §B.4 · p. 26
  • The per-task results complement client-level overall and tail metrics by showing where aggregate gains arise.Source: §B.4 · p. 26
  • The reported per-task breakdown is for the RoBERTa-large experiment in Section 5.2.Source: §B.4 · p. 26

B.5 Training-Capacity Analysis

The analysis compares training-capacity settings across heterogeneity regimes while holding the expert pool, inference capacity, and routing architecture constant. Li = 1 performs best for pure clients, while Li = 6 has small gains in mixed regimes.

B.5 Training-Capacity Analysis

  • FedSEE (L1) improves over full-capacity training by 0.89 points at αDir = 0.Both settings retain all six experts at inference, but L1 updates at most one task-specific expert per client while L6 updates six.Source: §B.5 · p. 27, Table 10 · p. 27
  • L6 gains 0.25, 0.92, 0.05, and 0.31 points at αDir = 0.1, 0.5, 1.0, and the near-uniform regime, respectively.Across every regime, the two configurations remain within one percentage point.Source: §B.5 · p. 27, Table 10 · p. 27
  • The authors use Li = 1 as the default because it substantially reduces task-specific training and communication cost while retaining the full six-expert inference pool.Source: §B.5 · p. 27

B.6 Component Ablations

Component ablations assess public supervision, router calibration, expert count, and the shared expert; a separate analysis tests whether extraction contributions track clients’ known task proportions. Results indicate that calibration targets matter and mixture estimates retain information about task composition.

B.6 Component Ablations

  • With 100 public examples per task, accuracy is 81.27%, 1.52 points above FedSA-LoRA at 79.75%; the 1,000-example default reaches 82.65%.Using 500 examples per task reaches 82.49%, only 0.16 points below the default.Source: §B.6 · pp. 27–28, Table 11 · p. 28
  • Mixture-aware calibration improves over no calibration by 1.30 points; uniform calibration reaches 80.85%, 0.50 points below no calibration.The default mixture-aware target has 82.65% accuracy; the no-calibration setting has 81.35%.Source: Table 11 · p. 28, §B.6 · p. 28
  • The six-expert default reaches 82.65%, versus 82.01% with M = 3 and 82.31% with M = 9.The modest reductions indicate robustness to incomplete and extraneous task coverage in the public reference set.Source: Table 11 · p. 28, §B.6 · p. 28
  • Removing the shared expert reduces accuracy from 82.65% to 82.03%, a decrease of 0.62 percentage points.Performance remains close to the full model; the authors report that routed task-specific experts account for most of the adaptation benefit.Source: Table 11 · p. 28, §B.6 · p. 28

B.7 Client-Mixture Estimation

  • At αDir = 0, the dominant task is correctly identified for every client, with MAE 0.059; mixed-regime MAEs are 0.135, 0.132, and 0.121 at αDir = 0.1, 0.5, and 1.0.Contribution-based estimates remain positively associated with underlying task proportions across regimes.Source: Figure 9 · p. 29, §B.7 · p. 29

C FedSEE Algorithm and Implementation Details

FedSEE combines supervised expert extraction with contribution-aware capacity assignment and input-dependent routing, while the accompanying analysis establishes the geometry and limits of specialization.

C FedSEE Algorithm and Implementation Details

C.1 Projection-Level Reduction

  • At a routed frozen-feature projection, the fixed global A lets the shared LoRA term enter the baseline, making the reduction to Eq. (5) exact; the full-transformer extension is analyzed separately.Source: §C.1 · p. 29

C.2 Joint Extraction with Balanced Transport

  • Joint extraction uses Sinkhorn-enforced balanced contributions, with each task’s weights summing to one and total expert contribution spread uniformly across clients.Source: §C.2 · p. 31

C.3 The Mixture Proxy and Capacity Assignment

  • Client contribution profiles proxy task-mixture relevance, and inference and training budgets select available experts and limit which receive updates.Source: §C.3 · pp. 31–32

C.4 Router Parameterization and Aggregation

  • A lightweight linear router masks unavailable experts, while the shared router is averaged across clients retaining distinct inference sets and calibration targets.Source: §C.4 · p. 32

D Theoretical Results and Proofs

  • The theoretical appendices establish squared-loss geometry and recovery and routing guarantees, then analyze extensions beyond exact quadratic settings and FedSEE approximations.Source: D Theoretical Results and Proofs · p. 32

D.1 Barycentric Geometry and the Mixing Gap

  • With common task feature covariances, a client’s optimum is barycentric, and its mixing gap vanishes exactly for pure clients.Source: §D.1 · pp. 34–35

D.2 Supervised Expert Recovery and Finite-Sample Guarantees

This section analyzes supervised expert recovery and its finite-sample guarantees, including sources of recovery error and identifiability limits relative to upload-only recovery. It also describes how task-labeled public loss identifies experts’ task correspondence, while barycentric estimates recover client mixtures and static adapters when upload and expert-recovery errors vanish.

D.2 Supervised Expert Recovery and Finite-Sample Guarantees

  • When ε = 0, supervised recovery exactly recovers every task expert.Source: §D.2 · p. 35
  • Recovery error separates into anchor impurity, local estimation error, and finite public-sample error, which vanish as ε → 0, δloc → 0, and nm → ∞ for fixed upload geometry.The public-sample term vanishes at the parametric nm^-1/2 rate for fixed upload geometry.Source: §D.2 · pp. 36, 40
  • For ε > 0, local estimation error couples with anchor impurity through a √ε δloc term, reflecting Hölder-1/2 dependence on the moving feasible set; consistency remains as ε, δloc → 0 and nm → ∞.Source: §D.2 · p. 40

D.3 Identifiability Limits

  • Task-indexed expert recovery is impossible in general without task labels: permuting task identities leaves the label-free observables unchanged and yields error at least diamF(Θ)/2 on one equivalent instance.The result relies on exchangeable abstract tasks; known task-distinguishing structure breaks the permutation symmetry.Source: §D.3 · p. 41
  • Upload-only recovery can face distinct vertex sets with identical upload pools, while supervised public losses distinguish them; the supervised program still needs near-pure anchors because its convex-hull constraint may exclude true task vertices.Under γ-interiority, the best achievable anchor impurity is at least (M − 1)γ/M.Source: §D.3 · pp. 41–42
  • With candidates sufficiently close to the true vertices, supervised assignment is exact with population losses and has margin at least 1/4 σmin(Φ) sep^2; the empirical assignment is simultaneously correct for all tasks when the sample-size condition holds.Source: §D.3 · p. 42
  • The pool-only vertex-recovery lower bound is linear in γ under the γ-interior condition when γ ≤ sep/(2 diam_F(Θ)), matching the anchor term’s dependence for ε = Θ(γ).Every γ-interior population has anchor impurity at least (M − 1)γ/M; this concerns pool-only recovery, unlike FedSEE’s supervised identification using labeled public loss.Source: §D.3 · p. 43
  • Task-labeled public loss establishes the true vertex–task correspondence by separating each task’s correct expert from alternatives by at least 1/4 σ_min(Φ) sep^2.Boundary information may recover the unordered vertex set, but cannot determine task correspondence; the task-indexed label-blind floor remains fixed even as anchor clients approach purity.Source: §D.3 · p. 43

D.4 Mixture Estimation and Static Reconstruction

  • Barycentric mixture estimation error is controlled by local upload error, expert-recovery error, feature conditioning, and simplex geometry, and vanishes when both errors vanish.With exact uploads and recovered experts, the estimated mixture equals the true mixture; greater affine separation among experts improves conditioning.Source: §D.4 · pp. 43–44
  • The reconstructed static adapter converges to the optimal adapter in parameter error and population excess risk when mixture-estimation and expert-recovery errors vanish.The reconstruction bound propagates both errors to recovery of the client’s optimal static adapter.Source: §D.4 · pp. 44–45

D.5 Routing and Task-Aligned Specialization

The section studies task-aligned expert routing and recovery, including extensions to general losses and model classes and conditions for balanced-transport extraction and temporal smoothing. It also examines calibration and router-representation limits, and distinguishes contribution-matrix stability from recovery of true client mixtures.

D.5 Routing and Task-Aligned Specialization

  • Specialization is input-level: a mixed client can route inputs from different tasks to different experts rather than assigning the whole client to one expert.Task-indexed correspondence requires task-labeled recovery; label-free observations alone cannot determine that correspondence.Source: §D.5 · p. 46
  • A static adapter incurs strictly positive additional risk for genuinely mixed clients, while input-dependent routing can move risk from that static floor in either direction.Mean routing must be calibrated to the client’s mixture, and routing must remain sufficiently input-dependent to specialize across tasks.Source: §D.5 · pp. 46, 53
  • As expert-recovery error and task-level routing-concentration error vanish, the routed predictor’s risk approaches the oracle risk.The guarantee assumes the stated model conditions and routing concentration for each task in the client’s support.Source: §D.5 · p. 48
  • Weak task information in router inputs can make different client mixtures induce similar mean-routing statistics, though a shared router can satisfy task-conditioned specialization without necessarily incurring excess risk.The routing-sensitivity bound motivates obtaining mixture information through supervised expert recovery rather than router statistics alone.Source: §D.5 · pp. 51, 53
  • Uniform mean-usage targets conflict with exact specialization for non-uniform mixtures, whereas mixture-aware KL calibration bounds mean-routing mismatch through calibration, mixture-estimation, and proxy errors.The section distinguishes the uniform target from FedSEE’s task-indexed calibration target.Source: §D.5 · pp. 50, 52, 53

D.6 Extensions Beyond the Exact Squared-Loss Setting

  • For general losses, the client risk remains an exact mixture of task risks, while the client optimum is a perturbed barycenter controlled by curvature dispersion; the analysis propagates this perturbation to expert recovery.The extension specializes the analysis to softmax cross-entropy over a frozen representation and discusses implications for mixture estimation, routing, and identifiability.Source: §D.6 · p. 53

D.6.1 Setup and Assumptions

  • The extension retains a frozen representation and uses a general loss, with the expert parameters as the sole trainable block and task risks defined over the true task distributions.The setup defines the true experts as the set of task-optimal parameters for M ≥ 2 tasks.Source: §D.6.1 · p. 54
  • Task-risk analysis assumes interior unique minimizers and smooth, strongly convex risks; under squared loss, curvature dispersion vanishes when tasks share input covariance.With heterogeneous covariances, κ equals the covariance-dispersion bias ρΦ.Source: §D.6.1 · p. 54

D.6.2 General-Loss Geometry and Expert Recovery

  • General losses preserve mixture-risk decomposition, but the client optimum may depart from the task-optimum barycenter by an amount controlled by curvature dispersion κ.Exact barycentricity holds under a directional condition; the general-loss expert-recovery guarantee also accounts for anchor impurity and curvature-induced deviation.Source: §D.6.2 · pp. 54–56

D.6.3 Softmax Cross-Entropy over a Frozen Representation

  • For softmax cross-entropy, the Hessian-Lipschitz constant is LH ≤ 2Q^3, measured in spectral norm against Frobenius parameter displacement; highly confident models can have loose condition-number and perturbation bounds.The analysis is most informative in a bounded-logit regime, where softmax cross-entropy satisfies the local strong-convexity, smoothness, and Hessian-regularity requirements.Source: §D.6.3 · p. 57

D.7 Additional Model-Class Instantiations

  • Linear probing and fixed random-feature or kernel representations inherit the framework only when task realizability and shared feature-covariance conditions hold.Approximately shared covariances admit a perturbed interpretation; the guarantees are conditional on the induced representation and task geometry.Source: §D.7 · p. 58

D.8 Deep-Model Extension for FedSEE

  • For deep models, a Jacobian-Lipschitz local first-order approximation supports approximate barycentric geometry when the stated neighborhood and strong-convexity conditions hold.At a single adapted projection, the reduction is exact conditional on the incoming representation; subsequent-layer representation and attention changes create deviations in the full transformer.Source: §D.8 · pp. 58–59

D.9 Recovery under Balanced Transport

  • When the coupling excess is zero, balanced transport retains the independent recovery guarantee; transport-compatible pure anchors yield exact population recovery.The balanced constraint narrows the joint feasible set and can increase taskwise extraction loss.Source: §D.9 · p. 60

D.10 Stability of the Deployed Contribution Matrix

  • The deployed contribution EMA preserves balanced-transport feasibility and tracks changing raw extraction matrices; stationary raw matrices admit a geometric tracking bound.When raw-matrix drift tends to zero, tracking is exact.Source: §D.10 · pp. 61–62
  • Contribution-matrix smoothing does not guarantee recovery of true client mixtures; normalized contributions are only a proxy, and recovery conditions apply to the upload pool.Those upload-pool conditions are not guaranteed by FedSEE training.Source: §D.10 · p. 63
Loading 2610.00580v1…