PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a Statistical Understanding of Mixture-of-Experts

Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang

arXiv:2609.03501v1stat.MLcs.LGmath.STstat.TH

TL;DR

Existing theory leaves a gap in understanding how MoE design choices shape approximation. This paper frames MoE as input-dependent expert aggregation and finds that sparse Top-K routing can retain localized benefits while reducing per-input expert computation.

  • Problem

    Existing theory leaves a gap in understanding MoE's structural design choices, including how they shape approximation.

    Source: §1 · p. 3

  • Method

    The paper develops a unified statistical framework for MoE and interprets gating through input-space regions where different experts perform well.

    Source: §1 · pp. 3–4

  • Results

    Sparse Top-K routing admits an oracle inequality with router-learning cost of the same order as dense softmax gating, up to a boundary-stability logarithmic term, while reducing per-input expert computation.

    Source: §7.1 · p. 52

  • Takeaways & Limitations

    The statistical role of MoE is tied to how gating combines experts across inputs.

    Source: §7.1 · p. 51

    The modular per-expert/router analysis leaves open how these mechanisms interact with gradient-based training and representation learning.

    Source: §7.2 · p. 53

Abstract

from arXiv · show

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choices, especially routing, sparse activation, and shared experts, remain only partially understood, as existing theory has largely focused on parametric or correctly specified MoE models. In this paper, we view MoE as a form of localized aggregation and show how this localization reshapes the approximation-estimation-computation tradeoff. We derive oracle risk bounds for learning dense and sparse routing with evolving experts, separating approximation, expert-learning, and router-estimation errors, and characterize how sparse Top-K routing can retain the benefits of localized aggregation while controlling per-input computation. We also interpret gating through the geometry of input space, relating routing performance to regions of local expert advantage, and show how shared experts, as adopted in architectures such as DeepSeekMoE, can extract common predictive structure so that routed experts focus on residual local variation. Together, these results provide a unified statistical framework for understanding MoE through input-dependent expert aggregation, in which expert specialization and computational tradeoffs are governed by local predictive structure.

1 Introduction

This section frames and formalizes MoE as input-dependent aggregation over routed experts, examining how dense softmax and Top-K routing, sparse activation, and always-active shared experts shape localization, statistical risk, and computation. It presents a unified framework for analyzing these design choices with evolving experts.

1 Introduction

  • Existing theory often assumes finite-dimensional, correctly specified MoE models and does not fully disentangle how modern architectural choices shape approximation, estimation, and computation.The paper identifies overparameterization, misspecification, and non-convex end-to-end optimization as limits of that perspective.Source: §1 · p. 3
  • To study router learning without characterizing full end-to-end dynamics, the analysis uses a predictable expert-training trajectory in which experts are plug-in predictors whose quality may evolve.Source: §1 · p. 3
  • Input-dependent routing creates localized aggregation that can be more expressive than global averaging, with dense gating smoothing expert outputs and sparse Top-K gating selecting experts locally.The paper characterizes when either form of localization can improve on global averaging.Source: §1 · p. 4
  • Oracle risk bounds for dense and sparse routing along evolving expert trajectories separate approximation risk, expert-learning error, and router-estimation cost.A discretization-based scheme avoids requiring global optimization of the non-convex MoE objective and highlights the estimation price of routing flexibility.Source: §1 · p. 4
  • Routing functions induce soft or hard input-space partitions, and their performance depends on alignment with regions where different experts perform well.Linear, quadratic, and kernel-based gates offer different geometric flexibility, motivating data-adaptive selection among routing classes.Source: §1 · p. 4
  • Shared experts extract common structure that routed experts would otherwise relearn, leaving them to model residual local variation and potentially reducing effective routed-learning complexity.Source: §1 · p. 4

1.1 Mathematical formulation of MoE architecture

  • The baseline formulation has routed candidate predictors and a router that maps each input to gating weights for combining or selecting experts.The analysis treats experts as candidate predictors; transformer experts are typically feed-forward networks.Source: §1.1 · p. 5
  • Linear routing scores induce affine decision boundaries, while quadratic scores allow curved boundaries, illustrating how gating geometry shapes the input-space partition.The paper uses these score classes to examine how the router partitions the input space.Source: §1.1 · p. 6
  • Dense softmax averages across all routed experts, whereas Top-K activates only the K highest-scoring experts and changes the active set across input-space regions.These are two forms of covariate-dependent aggregation: smooth averaging versus a piecewise-defined rule.Source: §1.1 · pp. 6–7
  • Motivated by DeepSeekMoE (Dai et al., 2024), the formulation combines always-active shared experts with routed experts selected by dense or sparse gating.The shared component is separated from the input-dependent routed component.Source: §1.1 · p. 7

2 Localized averaging in MoE

This section examines how input-dependent routing localizes expert aggregation across the input space, unlike fixed-weight global averaging. It considers the implications for approximation and function-class complexity, including sparse Top-K activation and settings where localized routing can outperform global convex combinations.

2.1 Constant experts: global versus localized averaging

  • With constant experts, linear-score softmax gating yields a class dense in C(X), allowing uniform approximation of any continuous function.Source: §2.1 · p. 8
  • For every fixed K, bounded-constant Top-K routing remains dense in C([0, 1]^d) as M grows, retaining universal approximation while activating only K experts per input.Increasing M lets the router generate finer local regions even with score parameters in a fixed box.Source: §2.1 · pp. 9–10

2.2 Approximation and metric entropy with a fixed number of experts

  • With fixed M, Top-K localized averaging improves worst-case approximation over global averaging: global averaging has non-vanishing error for general Lipschitz functions, while smaller K sharpens the Top-K bound.Source: §2.2 · pp. 10–11
  • For fixed M, dense routing adds finite-dimensional complexity, whereas Top-K active-set changes cause infinite small-scale uniform-norm entropy; under L2(PX), regularity or margin conditions can control boundary effects.Source: §2.2 · p. 12

2.3 Beyond constant experts

  • For a generic expert class, the subsection defines global and localized softmax aggregation classes to examine whether localization retains an expressive advantage beyond constant experts.Source: §2.3 · p. 12
  • Localized gating creates multiplicative structure that can represent σ(x1)^2, outside the finite input-independent aggregation class of the same expert family.The example shows that MoE expressivity comes from the interaction between router and experts, not only from the experts themselves.Source: §2.3 · p. 13

2.4 Two specialized experts: the advantage of localized routing

  • The localized Top-1 rule has strictly smaller population L2(PX) risk than every global convex combination of the same two fitted experts.Proposition 2.6 establishes this advantage under its stated condition on the good-region errors, cross-region errors, and γ.Source: §2.4 · p. 14
  • A single global weight compromises between regions, while localized routing preserves each expert’s good-region performance when cross-region errors greatly exceed good-region errors.In the symmetric case, γ = [(δ − ϵ)/2]+.Source: §2.4 · p. 14
  • The router turns fixed expert outputs into locally adaptive predictions, making dense routing, sparse routing, and global averaging genuinely different rules.The subsection frames this as a source of MoE expressive power beyond the complexity of individual experts.Source: §2.4 · p. 14

3 Risk bounds for dense MoE

This section develops oracle risk bounds for dense softmax gating, including bounds along an online training trajectory that separate approximation, expert-learning, and router-estimation errors. It also considers routed-expert growth and shared experts, a tractable fixed-expert implementation, and the connection between input-dependent gating aggregation and adaptive regression by mixing.

3 Risk bounds for dense MoE

  • Dense softmax routing risk is decomposed into approximation error, cumulative expert-learning error, and the statistical cost of learning the router.The bounds concern router learning along an online training trajectory.Source: §3 · p. 15

3.1 Basic setting and training procedure

  • Experts are treated as fixed plug-in components conditional on past data, while their quality can improve over the online training trajectory.After each prediction loss, aggregation weights are updated and the experts may also be updated for subsequent predictions.Source: §3.1 · pp. 15–16
  • A finite net of candidate gates is likelihood-aggregated to provide router risk guarantees without requiring global optimization of a non-convex empirical objective.The procedure is presented as a theoretical benchmark for router learning along the training trajectory, not as a practical substitute for end-to-end MoE training.Source: §3.1 · p. 16

3.2 Oracle inequalities for dense softmax gating

  • The oracle analysis studies dense softmax gating with linear scores and makes explicit that the number of routed experts may increase with sample size.The evolving predictors are indexed in a triangular-array setting.Source: §3.2 · pp. 18–19
  • Theorem 3.8 bounds dense softmax risk by oracle approximation plus learning cost, including average evolving-expert error and a gate-estimation term of order MRd log(MRdn)/n.The approximation term depends on population experts and gate expressiveness; the learning term also includes the expert trajectory’s average quality across calibration and aggregation.Source: §3.2 · p. 21
  • Increasing MR can improve localization and reduce approximation risk, but raises router-estimation cost; with fixed MR, consistency requires exact representation or a sufficiently rich expert collection.The bound allows MR to grow with sample size, balancing finer localization against a more complex gating class.Source: §3.2 · pp. 22–23
  • In Example 3.9, a shared expert capturing the oscillatory component leaves routed affine experts to approximate the smoother x² residual, removing dependence on L from approximation error.Without the shared expert, the oracle approximation risk is O(L^4/log^2 MR), an inflation of order L^4 in this example.Source: §3.2 · p. 22
  • When routed-expert complexity is moderate relative to sample size, aggregation provides a sharper oracle benchmark than direct ERM, with gate-estimation cost plus evolving-expert error.The gate-estimation term is of order MRd log(MRdn)/n.Source: §3.2 · p. 23
  • For fixed MR, Algorithm 1’s population projection can be replaced by a data-driven surrogate that projects aggregated weights onto the dense softmax class using time-expanded cross-entropy.Source: §3.2 · p. 23
  • For fixed M_R, the projection step is convex multinomial logistic regression with soft labels and avoids additional non-convex optimization.This tractable implementation uses a linear-score softmax parametrization.Source: §3.2 · p. 23
  • Under mild regularity conditions, the dense softmax estimator satisfies an oracle inequality involving the softmax oracle approximation risk A_soft(M_S, M_R).The bound also includes a constant C_MR depending on M_R.Source: §3.2 · p. 23
  • Unlike classical adaptive regression by mixing with input-independent weights, this aggregation mixes candidate gating functions to produce covariate-dependent expert weights.This extension matters because relative expert performance can vary across the input space.Source: §3.2 · p. 24

4 Risk bounds for sparse MoE

This section examines risk guarantees and computational tradeoffs for sparse Top-K routing, with bounds that separate approximation and learning errors and account for boundary stability. It also relates routing sparsity to local predictive structure, distinguishing regions with a dominant specialist from those requiring mixtures.

4 Risk bounds for sparse MoE

  • Top-K routing activates only K experts per input, addressing the computational cost of dense gating when the routed expert count is large.The section frames this as a statistical comparison between hard selection and local multi-expert aggregation.Source: §4 · p. 24

4.1 Oracle inequalities for Top-K gating

  • The Top-K gate assigns zero weight to unselected experts and normalizes transformed scores over the K selected experts.The exponential transformation yields a Top-K softmax gate; the analysis allows other regular score transformations.Source: §4.1 · p. 25
  • Under Assumption 4.4, the Top-K gating map has a one-sided local Hölder modulus of order 1/2 from Euclidean parameter distance to the L2(PX) metric.The result applies to regular score transformations, with a constant depending on K, transformation regularity, and C4.Source: §4.1 · p. 27
  • Theorem 4.7 introduces the Top-K risk bound under Assumptions 3.4–3.7 and 4.4, separating an oracle approximation term from a statistical learning term.The supplied passage introduces the theorem and its assumptions but does not include the bound itself.Source: §4.1 · p. 28
  • Top-K oracle risk separates approximation, routed-expert learning, and router-estimation errors, with an additional cost for controlling boundary-switch events.The boundary-stability parameter r_MR trades a potentially smaller approximation term against less stable active sets and higher learning cost.Source: §4.1 · pp. 28–29
  • For Top-K softmax, router-learning cost matches dense softmax up to log(1/r_MR), while evaluating only K routed experts reduces per-input expert-evaluation cost.These claims hold under the stated regularity conditions.Source: §4.1 · p. 29

4.2 Regionwise performance: hard selection versus soft aggregation

  • Top-1 is attractive when each input region has a dominant expert, whereas soft or multi-expert aggregation is needed when the best local predictor is a mixture.Under linear separability, Top-1 linear routing can realize the oracle region assignment.Source: §4.2 · pp. 29–30

4.2.1 Regionwise existence of a strong expert

  • With linearly separable regions and a strong local specialist in each, Top-1 achieves regionwise oracle risk, while dense softmax cannot exactly represent hard region assignments.The Top-1 bound combines regionwise approximation error with sparse-routing learning cost; dense softmax's approximation error is bounded by O(M0/log MR) under additional conditions.Source: §4.2.1 · pp. 30–31

4.2.2 Improved performance via multi-expert aggregation

  • When each region's best two-expert mixture improves on its best single expert, Top-1 incurs irreducible approximation error relative to that mixture.This limitation is intrinsic to the Top-1 function class and adds to estimation error.Source: §4.2.2 · p. 32, Algorithm 1 with Top-1 routing. Then · p. 32

Algorithm 1 with Top-1 routing. Then

  • Dense softmax and sparse Top-K routing with K ≥2 can assign nontrivial weights to multiple experts and approximate regionwise mixtures.Under suitable conditions, the dense-softmax upper bound includes the best regionwise two-expert mixture's oracle error and a statistical learning cost.Source: Algorithm 1 with Top-1 routing. Then · pp. 32–33
  • The risk bound's final term measures the error from approximating regionwise selection.Source: Algorithm 1 with Top-1 routing. Then · p. 33
  • When mixtures are strictly superior, Top-1 routing's excess term P_r δ_r remains at the oracle approximation level.Source: Algorithm 1 with Top-1 routing. Then · p. 33
  • Top-1 is sufficient when each region has a dominant specialist, but locally compositional prediction gives multiple-expert routing a role in representing mixtures that single-expert assignments cannot capture.The choice of K therefore depends on the local structure of prediction.Source: Algorithm 1 with Top-1 routing. Then · p. 33

5 Adaptation of the gating function

This section studies gating as an approximation to regions of expert performance, using assignment losses and geometry-matched or Gaussian-kernel gates for regionwise expert mixtures. It also examines adaptive selection among gating classes, separating local expert approximation, gating approximation, and the logarithmic cost of adaptation.

5.1 Performance-induced regions and gating approximation losses

  • Oracle regions defined by pointwise expert performance benchmark how well a gating class represents expert specialization across the input space.The global loss evaluates assignment alignment including boundaries, while the interior loss evaluates assignments away from a ρ-neighborhood of ambiguous boundaries.Source: §5.1 · pp. 33–35

5.2 Linear and quadratic gating: examples of geometric alignment

  • Top-1 linear gating exactly implements a threshold assignment, whereas bounded dense linear-softmax incurs error from smoothing the sharp boundary.The example separates global boundary-smoothing loss from interior assignment accuracy after removing a fixed boundary neighborhood.Source: §5.2 · pp. 35–36
  • For a disk partitioned into an inner ball and outer annulus, linear-softmax remains geometrically mismatched, while quadratic-softmax can capture radial boundaries.The interior loss therefore reveals mismatch between gating geometry and oracle partitions; linear gating suits affine partitions, while quadratic gating can improve approximation for curved ones.Source: §5.2 · p. 36

5.3 Gaussian-kernel-based gating function

  • Gaussian-kernel gating offers a flexible construction for approximating regionwise hard assignments and soft mixtures without prespecifying boundary geometry.Each region may target an arbitrary convex combination of routed experts rather than only a single expert.Source: §5.3 · pp. 36–37
  • A fixed data-independent Gaussian-kernel grid uses mesh width h to control approximation power and, through its number of centers, gating-class complexity.The grid has order h^-d centers.Source: §5.3 · p. 37
  • Gaussian-kernel gating approximates arbitrary regionwise target weights on partitions with controlled boundary layers, extending regionwise expert selection to expert mixing.The approximation error has an interior grid component and a boundary-smoothing component governed by Γ_MR.Source: §5.3 · pp. 38–39
  • A finer kernel grid can drive oracle approximation error to zero, but adds kernel centers and parameters, increasing statistical learning cost.Choosing h therefore requires balancing approximation error against the cost of learning the larger gating class.Source: §5.3 · p. 39
  • Away from region boundaries, the remaining Gaussian-kernel approximation error decays exponentially in h^-2, up to a multiplicative factor MR.The interior bound removes the boundary-layer contribution present in the global loss.Source: §5.3 · pp. 39–40
  • In simulations, linear, quadratic, and Gaussian-kernel gating perform best under linear, quadratic, and nonlinear region-boundary designs, respectively.The comparison reports ℓ1 gating-weight error and ℓ2 squared Euclidean error; the best-performing class aligns with each partition’s geometry.Source: §5.3 · p. 40, Table 1 · p. 40

5.4 Selection procedure for different classes of gating functions

  • The selection procedure adapts to unknown gating structure, nearly matching the best candidate class with an additive (log L)/n adaptation cost.The leading risk term balances each class’s approximation power against its statistical learning cost.Source: §5.4 · pp. 41, 43
  • The regionwise inequality separates local expert approximation error, best candidate-class gating approximation error, and the logarithmic cost of selecting a class from data.These terms describe the approximation of population experts on locally optimal regions, gating approximation among candidate classes, and data-driven class selection.Source: §5.4 · p. 43
  • The procedure adapts to unknown expert-performance geometry by choosing a gating class according to its approximation complexity and statistical learning cost.The selected estimator is defined over regions as in Definition 5.1.Source: §5.4 · p. 43

6 The role of shared experts

This section examines how a shared component changes routed learning by comparing shared-routed and pure-routed setups with routing fixed. It focuses on how extracting common structure can leave routed experts simpler residual targets, with the benefit depending on how those targets fit the regional learning rules.

6 The role of shared experts

  • The analysis isolates the statistical effect of a shared component by fixing the routing partition and regional learning rules, rather than modeling joint router and expert training.Shared experts are treated as components that capture common structure across regions; routed experts handle local variation.Source: §6 · p. 44, §6.1 · p. 46

6.2 A learner-specific comparison

  • The shared expert’s benefit is learner-specific: it can reduce the effective complexity of routed targets when the common component better matches the shared learning rule than the routed rules.The comparison assumes the routed learners’ hard branch targets are compatible with the shared representation.Source: §6 · p. 44, §6.2 · p. 48

6.1 Model and learning rules

  • At the approximation level, pure-routed learners can match the shared representation if their regional classes contain full branch functions, but they must estimate those functions rather than residuals.With a shared expert, each routed learner estimates only its region-specific residual; without one, it estimates the common component plus that residual.Source: §6.1 · p. 45

6.2 A learner-specific comparison

  • A sequential shared-then-routed fit need not recover the common component if regional residuals project onto the shared class; the stated shared-risk bound requires joint estimation or added conditions such as orthogonality or residualization.Joint penalized least squares over the additive shared-residual class is one suggested approach.Source: §6.2 · p. 47
  • Under the same regional learning rules, shared-routed learning can have asymptotically smaller worst-case risk for a fixed routed learner by estimating residual rather than full branch targets.Pure-routed targets may be harder to learn or less well approximated, though rates can match when the branch class is finite-dimensional and well matched to the routed learner.Source: §6.2 · p. 48

6.3 An illustrative example

  • After the shared expert extracts the common linear trend, routed experts estimate finite Fourier residuals at the parametric rate n^-1; pure-routed Fourier learners are slower because periodic approximation is limited by the nonperiodic linear component.Source: §6.3 · p. 50
  • When the shared component is complex but smooth and routed residuals are piecewise constant, shared learning leaves routed experts only the regional offsets, while pure routing must approximate the shared function on each region.If the shared function is poorly represented by the routed learner, pure routing may incur substantial approximation bias even when the offsets are easy to learn.Source: §6.3 · pp. 50–51
  • The shared-routed estimator has lower prediction MSE across displayed sample sizes in both illustrative settings, with gains arising from slower estimation or approximation bias for pure-routed learning.Which mechanism applies depends on how the induced targets interact with the local learning rules.Source: Figure 1 · p. 51, §6.3 · p. 51

7 Conclusions and outlook

The conclusion frames MoE as input-dependent expert aggregation that allocates capacity locally and shapes the approximation–estimation–computation tradeoff, while noting limits of the modular theory. Its outlook considers adapting sharing and exploration to local predictive structure, including multiscale reuse of common structure and exploration near unstable routing boundaries while preserving sparse inference.

7.1 Main conclusions

  • MoE's statistical role comes from input-dependent gating that allocates modeling capacity locally and changes the approximation-estimation-computation tradeoff.The framework treats dense routing, sparse routing, adaptive gating, and shared experts as mechanisms for matching capacity to local target complexity.Source: §7.1 · pp. 51–52
  • Sparse Top-K routing reduces per-input expert evaluation while retaining an oracle router-learning cost of the same order as dense soft-max gating, up to a boundary-stability logarithmic term.Gating classes also encode different assumptions about the geometry of the expert-induced input partition.Source: §7.1 · p. 52
  • Shared experts can turn routed learners' full branch-learning tasks into residual-learning tasks when the shared learning rule better matches the common component.They are useful when that component recurs across regions, avoiding redundant learning of global structure by routed experts.Source: §7.1 · p. 52

7.2 Limitations and open questions

  • The theory relies on high-level expert-error control and boundedness, uses partitions not directly observed in practice, and abstracts away key large-scale system and training interactions.Its results also condition some approximation and shared-expert analyses on population experts, oracle partitions, or fixed learning rules.Source: §7.2 · p. 53
  • Open problems include validating local expert advantages, explaining end-to-end expert-router co-adaptation, selecting expert granularity under budgets, and extending theory to deep MoE.The framework also leaves open how training produces experts that are diverse, stable, and locally useful, and how to incorporate computational constraints and high-dimensional inputs.Source: §7.2 · pp. 53–54

7.3 A statistical outlook for future MoE architectures

  • Future routers could adapt the number of active experts to local predictive structure, using fewer for dominant specialists and more when prediction is compositional.Such activation should preserve meaningful specialization while coexisting with system load, capacity, and communication constraints.Source: §7.3 · pp. 54–55
  • Shared–routed architectures could reuse predictive structure globally, within groups of related inputs, and locally through routed experts.The scope of shared components could adapt to the inputs over which they capture common structure.Source: §7.3 · p. 55
  • The paper frames specialization as local predictive advantage, but standard sparse training sees only executed routes and may reinforce early routing decisions without enough information about alternatives.This motivates exploring counterfactual routes, particularly near boundaries where small input perturbations can change the active expert set.Source: §7.3 · p. 55
  • Boundary-aware exploration could focus a limited exploration budget on inputs with unstable expert assignments, reducing specialization driven by early routing noise without activating more experts at inference.Routing boundaries are regions where small perturbations may change the active set.Source: §7.3 · pp. 55–56
  • Future MoE design may adapt specialization, expert granularity, activation, sharing, and exploration to local prediction structure while respecting sparse-computation constraints.These design choices are presented as directions for future work.Source: §7.3 · p. 56

Supplementary Materials for “Towards a Statistical Understanding of Mixture-of-Experts”

The supplementary file supplies proofs for the main article’s results, additional technical results, and simulation details. It also records notation used in the supplement.

Supplementary Materials for “Towards a Statistical Understanding of Mixture-of-Experts”

A.1 Proof of Proposition 2.2

The proofs construct nearest-grid-center experts to establish approximation properties of localized MoE representations. The second proof contrasts local approximation with a global constant predictor.

A.1 Proof of Proposition 2.2

  • A.1 partitions [0, 1]^d into q^d cubes and chooses expert scores whose maximizers correspond to the nearest grid centers.The proof uses n = q^d and M = Kn.Source: §A.1 · pp. 67–68
  • A.1 assigns K identical constant experts to each grid center, so Top-K routing selects copies attached to the nearest center.The resulting output is a convex combination of values of h at nearest centers and approximates h uniformly within ε.Source: §A.1 · p. 68

A.2 Proof of Proposition 2.3

  • A.2 uses a nearest-center construction for local experts and compares its approximation with a global constant predictor.The proof also treats the local upper bound using a cube partition and cube centers.Source: §A.2 · pp. 68–70

A.3 Proof of Theorem 2.4

The proof controls the complexity of softmax-gated functions through parameter coverings and separately constructs a separated family for sparse Top-K gating.

A.3 Proof of Theorem 2.4

  • A separate construction bounds the constant global class using the covering number of the interval [−R, R].The text identifies this as the covering number under the uniform metric for constant functions.Source: §A.3 · pp. 70–71
  • The softmax-class covering argument uses one expert as a reference and bounds changes in the logits through relative slope and intercept differences.The proof combines parameter nets to obtain the stated covering-number order.Source: §A.3 · pp. 72–74
  • The proof constructs an uncountable family of Top-K-gated functions that are separated by 2R/K, yielding a lower bound on covering numbers at small radii.The separation argument applies for 0 < ε ≤ R/K.Source: §A.3 · pp. 74–75

A.4 Proof of Example 2.5

The proofs show a local MoE representation of σ(x1)^2 that finite linear combinations of logistic functions cannot reproduce. They also derive a lower bound by balancing regional errors.

A.4 Proof of Example 2.5

  • A.4 represents σ(x1)^2 as a local average of two experts, while ruling out its representation as a finite linear combination of logistic functions.The non-representation argument uses the difference between simple and second-order poles.Source: §A.4 · pp. 76–77

A.5 Proof of Proposition 2.6

  • A.5 balances two opposing affine terms in w to obtain a lower bound from the regional errors.The resulting bound uses the positive part of (δ1δ2 − ε1ε2)/(δ1 + δ2 + ε1 + ε2).Source: §A.5 · pp. 77–78

B.1 Proof of Theorem 3.8

The proof establishes Theorem 3.8’s oracle inequality for covariate-dependent forecast aggregation with AFTER, covering its burn-in, calibration, and aggregation setup. It controls calibration-scale error and evolving comparators, and bounds the routing Jacobian using parameter-space discretization and a projection comparison.

B.1 Proof of Theorem 3.8

  • Under Assumption 3.7 and Assumptions B.1–B.2, generalized AFTER satisfies a risk bound for aggregating candidate forecasts.The proof uses likelihood control and Hellinger bounds to relate the aggregate to candidate prediction error.Source: §B.1 · pp. 79, 82, 83
  • For Theorem 3.8, the proof applies Proposition B.3 after burn-in and calibration, using aggregation length n3 and a uniform prior over candidates.It then applies Lemma B.4 to the candidate forecasts.Source: §B.1 · pp. 83, 86, 87
  • Lemma B.4 controls calibration-scale error for predictable candidate forecasts under Assumption 3.7.Its constant depends on B, σ, σ̄, and the standardized error distribution’s fourth moment.Source: §B.1 · pp. 83–84
  • Lemma B.5 reduces an evolving comparator to a population comparator, with the final term bounded by CMRa^2 when MS is fixed and MR ≥ 2.Source: §B.1 · p. 85
  • The routing Jacobian satisfies ∥J(z)∥∞→1 ≤ 1.The bound follows from ∥J(z)x∥1 ≤ 4p(1 − p) ≤ 1.Source: §B.1 · p. 88
  • Under the normalization θMR = 0, the effective parameter dimension is D = (MR − 1)(d + 1), and ΘMR lies in a D-dimensional box of radius C1 log MR.The proof uses a Euclidean η-net for this parameter set.Source: §B.1 · p. 89
  • The proof compares the time-averaged estimator ˜f0 with the projected estimator ˆf0, then uses the constructed grid comparator to prove the desired oracle inequality.The comparison uses the triangle inequality and (a + b)2 ≤ 2a2 + 2b2.Source: §B.1 · pp. 89–90

B.2 Proof of Example 3.9

The proof constructs softmax gating on a one-dimensional grid and bounds its behavior with and without shared experts. It chooses normalized parameters within the logarithmic parameter box.

B.2 Proof of Example 3.9

  • The construction sets τ proportional to 1/⌊√log M_R⌋ and normalizes logits by a shift that leaves softmax weights unchanged.The resulting parameter coordinates are O(log M_R), placing the construction in Θ_M_R under the chosen c_Θ.Source: §B.2 · p. 91
  • With shared experts, the proof bounds softmax-weight second moments using grid centers near each input and counts of centers at increasing distances.The grid argument applies uniformly over x in [0, 1], including boundary points, when τ ≥ 2h and M_R is sufficiently large.Source: §B.2 · pp. 91–92
  • Without shared experts, the proof assumes routed experts contain local affine approximations and obtains an error of order O(L^4/log^2 M_R) when L is large.The bound's constants retain their dependence on the curvature of f_0.Source: §B.2 · pp. 93–94

C Proof of Section 4

This appendix proves the Section 4 oracle results, establishing regularity and routing bounds, deriving the oracle decomposition and a lower bound, and analyzing gating choices through two examples of their effects on losses.

C Proof of Section 4

  • Lemma C.1 states that a continuously differentiable, strictly increasing ϕ satisfying the bounded-relative-derivative condition is regular with lϕ = 0.Its proof bounds the nonzero Jacobian block using the simplex weights and Bϕ.Source: C Proof of Section 4 · pp. 94–95

C.1 Proof of Proposition 4.5 and related remarks

  • The proof of Proposition 4.5 separates inputs with unchanged Top-K sets, where regularity controls gate variation, from set changes, controlled using score crossings and slope separation.The related covering argument uses effective dimension (MR − 1)(d + 1) and a parameter box with radius proportional to log MR.Source: §C.1 · pp. 96–98

C.2 Proof of Theorem 4.7

  • When n2 ≍ n3 ≍ n, both the discretization term and calibration term 1/n2 are absorbed, and combining these bounds with the preceding online inequality proves (4.2).The grid has effective dimension D = (MR −1)(d + 1).Source: §C.2 · p. 101

C.3 Proof of Corollary 4.9

  • For Corollary 4.9, a separated linear score system assigns Top-1 routing to region-wise specialists, while unused experts are filled without changing assignments.Applying Theorem 4.7 yields the stated sparse oracle rate, which also holds for the online average.Source: §C.3 · pp. 103, 105

C.4 Proof of Proposition 4.11

  • The proof of Proposition 4.11 uses the fixed partition and mixture advantage assumption to establish an oracle lower bound for Top-1 routing.It also uses the projection step and Jensen’s inequality to establish the second lower-bound claim.Source: §C.4 · pp. 105–107

D.1 Proof of the oracle decomposition for gating approximation

  • The proof establishes the stated oracle decomposition by applying Assumption 3.6 and bounding errors over the partition regions.It uses a squared-sum inequality before integrating over the regions.Source: §D.1 · p. 108

D.2 Proof of Example 5.2

  • The proof establishes softmax-gating bounds for L(1) and L(2), while a Top-1 score choice realizes the oracle partition exactly, except at x = 0.The boundary point x = 0 has probability zero under the example’s uniform distribution.Source: §D.2 · pp. 110, 112

D.3 Proof of Example 5.3

  • For the disk example, the proof combines a linear-softmax lower bound with a quadratic-softmax upper bound for L(2).Source: §D.3 · p. 115

D.4 Proof of Theorem 5.5

This section develops proof tools for Theorem 5.5 and related oracle arguments, including a Gaussian grid-tail bound and an additive-sieve least-squares inequality for the known-partition shared model. The shared-model bound combines approximation errors for the common component and partition-specific residuals.

D.4 Proof of Theorem 5.5

  • For σ=κh with κ∈[1/2,1], Lemma D.1 bounds Gaussian tail mass from uniform-grid centers at distance at least r by a constant depending only on d and κ.Theorem 5.5's proof applies the lemma when ρ_h≥2σ and restricts to sufficiently small h so ρ_h≤ρ_0.Source: §D.4 · pp. 115, 118

D.5 Proof of Corollary 5.7

  • With h<ρ/(2κ), the Corollary 5.7 proof applies Lemma D.1 and the interior-grid-point argument to obtain a bound with constant independent of h and M_R.Source: §D.5 · pp. 120–121

D.6 Proof of Theorem 5.9

  • Theorem 5.9's aggregation-projection proof uses candidate-specific oracle inequalities and sample splitting; with comparable split sizes, the selection-stage cost is of order log L/n.Source: §D.6 · pp. 121, 123

D.7 Proof of Corollary 5.10

  • The Corollary 5.10 proof substitutes regional bounds into Theorem 5.9, using the oracle region assignment on each region.Source: §D.7 · pp. 123–124

E.1 Proof of Proposition 6.2

  • For Proposition 6.2, conditioning on regional counts makes each regional rule a fixed-sample-size learner, and the branch with the larger weighted lower bound yields the pure-routed lower-bound argument.Source: §E.1 · pp. 125–126

E.2 A Sieve Justification for the Shared Upper Bound

  • The shared-upper-bound subsection gives a sufficient-condition approach under sieve least squares, assuming sieves for the shared and residual classes and defining their sieve oracle rates.Source: §E.2 · pp. 126–127
  • The sieve least-squares tradeoff has an approximation-error term and a variance cost for estimating k coefficients from m observations, with additional logarithmic factors absorbable into ϱS and ϱj.This is the standard sieve oracle inequality used to motivate the compatible additive-complexity condition.Source: §E.2 · p. 127
  • Under Assumption E.1, Proposition E.2 establishes the joint shared upper bound used in Assumption 6.1 for the known-partition model.The proposition gives a joint estimator obtained by least squares over a suitable additive sieve.Source: §E.2 · pp. 127–128
  • The approximation contribution is bounded by C times the common-sieve error plus the partition-weighted residual-sieve errors, AS(kS) + p1A1(k1) + p2A2(k2).The proof approximates the shared and residual functions separately and combines them in an additive sieve.Source: §E.2 · pp. 127–128

E.3 Proof of Example 6.4

The proof establishes an upper bound for a shared finite-dimensional class and contrasts it with a pure-routed Fourier-sieve lower bound, using local approximation and regional sampling arguments.

E.3 Proof of Example 6.4

  • With dK = 2K + 1, the variance lower bound combines with the approximation term to establish a risk lower bound for the Fourier-series learner.The proof states the resulting oracle risk for the Fourier orthogonal-series learner on Hj.Source: §E.3 · pp. 129, 130, 132
  • The true regression function belongs to the finite-dimensional shared class, whose dimension is fixed.The proof then verifies a shared upper bound for this class.Source: §E.3 · pp. 130–131
  • Linear independence of the shared-class basis under uniform design makes its population Gram matrix positive definite with smallest eigenvalue bounded away from zero.The proof uses endpoint periodicity of finite trigonometric polynomials to rule out coincidence with a nonconstant affine function.Source: §E.3 · p. 131
  • For K ≥ K0, the ramp component contributes a Fourier approximation-error lower bound of order K^-1 when |γ| ≥ γ0.The same lower bound extends to the finitely many K < K0 after changing the constant.Source: §E.3 · p. 132
  • The pure-routed estimator has disjoint regional loss, and conditioning on regional sample counts yields a lower bound when p1 = p2 = 1/2.The proof combines this lower bound with the shared upper bound to establish the stated procedure-level result.Source: §E.3 · p. 133

F.1 Order of Empirical Risk Minimization under Sub-Gaussian Noise

This section studies online ERM and cross-entropy approaches to softmax routing under sub-Gaussian noise, connecting routing-weight estimation to function prediction and giving dense-softmax approximation guarantees for partition-based targets. It also examines how gating-class geometry and alignment with the true partition affect routing approximation, including simulations that hold expert functions fixed.

F.1 Order of Empirical Risk Minimization under Sub-Gaussian Noise

  • The direct online ERM benchmark minimizes one-step-ahead squared loss, but requires global optimization of a non-convex softmax objective and is not advocated computationally.It serves as a theoretical comparison for gating estimation.Source: §F.1 · pp. 133–134
  • Under independent conditionally sub-Gaussian noise and the stated assumptions, Proposition F.1 bounds the ERM benchmark conditional on pretrained experts and extends the bound to population experts and time-averaged output.The supplied passages state these extensions but do not include the bound's formula or rate.Source: §F.1 · p. 135

F.2 Details for Empirical Parameter Estimation Using Cross-Entropy

  • The online aggregate generally cannot be represented by one averaged gate because both its aggregation weights and experts vary over time, so the method fits one softmax gate to the sequence of aggregated gates.Source: §F.2 · p. 140
  • The empirical cross-entropy objective is convex in the softmax parameters, making the parameter-estimation optimization numerically feasible.Source: §F.2 · pp. 141–142
  • The implementation uses the first three data subsamples for burn-in, calibration, and aggregation, and the final subsample for empirical parameter estimation.Source: §F.2 · p. 142
  • An additional fixed-MR compatibility condition ensures that gate discrepancies remain detectable through the evolving routed experts, linking weight approximation to function approximation.The text characterizes the condition as a positive minimum singular-value requirement for the relevant operator.Source: §F.2 · p. 143
  • When softmax probabilities are bounded away from zero, cross-entropy and squared weight distance are comparable, enabling the weight-error analysis.The passages establish the comparison for probability vectors and say it also holds after empirical averaging or integration over the input distribution.Source: §F.2 · pp. 144–145
  • Lemma F.6 establishes time-averaged weight estimation via cross-entropy, including an empirical L2 oracle inequality and its population counterpart.Its statement conditions on prior data and algorithmic randomness and assumes fixed MR with comparable sample-split sizes.Source: §F.2 · pp. 145–147
  • Under the stated assumptions and comparable sample splits, Proposition F.7 gives a function-estimation bound for the cross-entropy weight estimator.The proof transfers weight approximation to prediction error and invokes the ARM oracle bound; the bound’s formula is not present in these passages.Source: §F.2 · pp. 148, 150
  • The cross-entropy adjustment adds only a 1/n term, which can be absorbed into the original ARM oracle bound for the online aggregated sequence.Source: §F.2 · p. 151

F.3.1 Dense Softmax Upper Bound after Corollary 4.9

  • For fixed M0, dense softmax approximates hard Top-1 region assignments with error O(1/log MR) under margin regularity.The bounded margin density controls probability mass near region interfaces; the general approximation order is O(M0/log MR).Source: §F.3.1 · pp. 152, 155

F.3.2 Dense Softmax Upper Bound after Proposition 4.11

  • For fixed M0, dense softmax approximates regionwise two-expert mixtures with error O(1/log MR) when mixture weights are bounded away from zero.The stated general order is O(M0/log MR); the two target softmax weights preserve the mixture proportions while suppressing contributions from other experts.Source: §F.3.2 · pp. 157–159

G.1 Comparison of different gating classes

  • Monte Carlo simulations examine how gating-class performance depends on routing-region geometry, rather than provide a comprehensive benchmark comparison.Source: §G.1 · p. 161
  • A gating class performs best when its geometric structure aligns with the true partition.Source: §G.1 · p. 161
  • The comparison uses linear, quadratic, and Gaussian-kernel gates across DGPs with linear, quadratic, and highly nonlinear partitions.Source: §G.1 · p. 161
  • With expert functions fixed and known, differences across methods primarily reflect how well each gate approximates the true routing structure.Source: §G.1 · p. 161
  • Visualizations compare true and estimated expert-dominance regions to illustrate how the gating classes adapt to partition geometry.Source: §G.1 · p. 164

G.2 MSE Comparison With and Without a Shared Expert

Two simulations compare shared-routed and pure-routed estimators to examine prediction accuracy when the routing threshold is unknown. They cover complementary cases where shared and routed components have opposite levels of complexity.

G.2 MSE Comparison With and Without a Shared Expert

  • Two simulations compare shared-routed and pure-routed estimators for prediction accuracy when the routing threshold is unknown.The threshold is estimated rather than treated as known.Source: §G.2 · pp. 164–165
  • The designs cover complementary regimes: a simple shared component with complex routed components, and a complex shared component with simple routed components.Both studies use the Fourier-sieve illustration in Example 6.4.Source: §G.2 · p. 164
  • In the first design, the shared component is linear and the routed components are order-3 Fourier series; the second uses a Fourier shared component and simple linear routed experts.The second design uses a shared Fourier component of order Q = 3 and routed experts in span{1, t}.Source: §G.2 · pp. 165–166
  • For each design, the unknown threshold is selected by profile least squares over a 200-point grid, with coefficients fit by ordinary least squares.Each sample size is repeated 200 times, and prediction accuracy is evaluated by integrated squared error approximated on 5,000 grid points.Source: §G.2 · p. 166

Related papers

Loading 2609.03501v1…