Source-linked AI summary
More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Models
Chandresh Pandey
TL;DR
The paper examines whether operator-level MoE can improve spectral state-space modeling of heterogeneous, regime-switching dynamics. In a controlled three-regime prediction study with extensive ablations, it finds no improvement over a single expert and shows that lower chaotic-regime error can reflect attractor-destroying smoothing. The results identify a limitation of the studied operator interpolation and motivate geometry-aware evaluation.
Problem
The paper addresses limited evidence for whether mixing stable operators and learned routing improves modeling of heterogeneous, regime-switching time series while preserving chaotic dynamics.
Method
The study evaluates GMS, which convexly mixes stable spectral state-space operators using soft routing, on sequences containing chaotic, oscillatory, and noise-dominated regimes.
Results
Operator-level mixtures consistently failed to outperform a single-expert baseline; increasing capacity caused inverse scaling, routing collapse, and degradation that oracle supervision did not resolve.
Takeaways & Limitations
For the studied convex spectral-operator parameterization, naive expert interpolation can suppress regime-specific structure, so dynamical evaluation should include attractor geometry rather than error metrics alone.
Takeaways & Limitations
The conclusions concern the convex spectral-operator parameterization and training protocol studied, under the paper’s controlled synthetic setting.
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) architectures are commonly motivated as a way to increase expressivity by decomposing complex systems into simpler local dynamics. This intuition has recently been extended to spectral state-space models, where mixing stable operators is assumed to enable adaptation to heterogeneous or regime-switching time series. We critically evaluate this assumption in a controlled synthetic setting designed to isolate dynamical rather than representational challenges. We study a next-step prediction task on sequences composed of three regimes: chaotic dynamics generated by the Mackey-Glass system, a stable oscillatory regime, and a noise-dominated autoregressive regime. Across extensive ablations including capacity scaling, oracle routing, frozen-expert variants, and comparisons to output-level MoE baselines, operator-level mixture models consistently fail to outperform a single-expert baseline. Increasing the number of experts leads to inverse scaling, routing collapses or fails to induce meaningful specialization, and even perfect regime supervision does not prevent degradation in global performance. Furthermore, we show that apparent improvements in mean squared error on chaotic trajectories can be misleading. Phase-space analysis reveals that lower error often arises from temporal smoothing that destroys the geometry of the underlying attractor rather than from faithful modeling of the dynamics. These results identify a likely limitation of operator interpolation under the studied parameterization and training protocol, and underscore the need for geometry-aware evaluation when assessing regime-switching dynamical systems.
1 Setting and Hypothesis
The paper tests whether operator-level MoE improves prediction for heterogeneous, regime-switching dynamics. It evaluates the hypothesis that more experts should improve accuracy, specialization, and chaotic-attractor geometry.
- Real-world time series can alternate among chaotic, periodic, and stochastic regimes, making qualitative dynamical structure important alongside predictive accuracy.
- The study uses controlled sequences combining Mackey–Glass chaos, stable oscillation, and noise-dominated autoregression.Evaluation uses global and per-regime NMSE plus delay-embedded phase-space analysis.
- The setting is designed to isolate dynamical rather than representational challenges in regime-switching prediction.
- The tested MoE hypothesis predicts that additional experts will reduce NMSE, specialize by regime, and preserve chaotic-attractor geometry.
2 Proposed Solution
Gated Multi-Stability (GMS) extends linear spectral state-space models with multiple stable operators and learned soft routing. Its formulation assumes that interpolating local operators can represent heterogeneous dynamics while preserving stability.
- GMS combines multiple stable state-space operators through a learned routing mechanism for regime-switching dynamics.
- Each expert is a spectral-domain linear state-space model whose transition eigenvalues lie strictly inside the unit circle.The constraint enforces stability and prevents unbounded growth.
- At each timestep, the router produces soft expert weights, and GMS forms the effective transition operator as a convex combination.
- The approach assumes that interpolating a small set of stable local operators and using soft routing can induce regime specialization.It is evaluated against a single expert using oracle routing, frozen experts, and output-level MoE comparisons.
3 Observed Outcome
Increasing operator-level MoE capacity failed to improve performance, while routing collapsed toward one expert. Phase-space analysis showed that lower chaotic-regime error could reflect smoothing and attractor distortion rather than faithful dynamics.
- 3.1 Global Performance and Capacity Scaling: Beyond occasional modest gains at K = 2, higher-capacity models underperformed the single-expert baseline and showed increased variance.Both K = 4 and K = 8 had higher mean NMSE and substantially greater variance across seeds.
- 3.1 Global Performance and Capacity Scaling: The single-expert model was more reliable, while multi-expert models were more sensitive to initialization and optimization noise.The primary K = 4 configuration performed worse on average than the baseline despite greater representational capacity.
- 3.1 Global Performance and Capacity Scaling: Operator-level mixtures did not benefit from increased capacity; additional experts introduced instability and degraded generalization in this setting.
- 3.2 Routing Behavior and Regime Specialization: Under supervised warmup, the K = 4 router used only 1/3 experts across chaotic, oscillatory, and noise-dominated regimes.The collapse persisted after supervision was annealed to zero.
- 3.2 Routing Behavior and Regime Specialization: Even with explicit regime boundaries, routing overwhelmingly favored one expert, preventing meaningful specialization.The figure shows only transient deviations during regime transitions.
- 3.3 Phase-Space Analysis: Geometry versus MSE: NMSE can be misleading for chaotic systems because variability suppression may lower pointwise error while failing to preserve qualitative dynamics.
- 3.3 Phase-Space Analysis: Geometry versus MSE: Despite a 61% reduction in MSE, GMS collapsed the Mackey–Glass attractor toward a smooth, low-variance manifold and destroyed fine-scale chaotic structure.The analysis uses delay-coordinate embedding with τ = 17.
4 Conclusion
The study finds that operator-level MoE does not outperform a single expert for the tested regime-switching spectral state-space setting. It attributes the failure pattern to contractive operator interpolation and emphasizes geometry-aware evaluation.
- Across extensive ablations, operator-level mixtures consistently failed to outperform a single-expert baseline.Increasing capacity produced inverse scaling, routing collapsed or failed to specialize, and oracle supervision did not resolve degradation.
- Lower chaotic-regime MSE often arose from temporal smoothing that destroyed attractor geometry rather than faithful dynamical reproduction.
- For the studied convex spectral-operator parameterization, interpolation biases learned dynamics toward averaged, contractive behavior.The paper therefore highlights geometry-aware evaluation for spectral state-space models.
A Training Details and Optimization
The experiments use stable complex spectral state-space experts, input-and-state-based routing, identical optimization settings, and five-seed reproducibility checks on a three-regime synthetic dataset.
- Model architecture: All models use complex-valued linear spectral state-space operators with eigenvalues strictly inside the unit circle for stability.Mixture models allocate state dimension per expert, so total latent dimensionality and parameter budgets increase with K.
- Routing: For K > 1, a feedforward router maps the current input and latent-state magnitude to soft expert weights.The effective transition operator is formed from these routing weights.
- Dataset: The dataset concatenates 500 Mackey–Glass chaotic steps, 500 stable oscillatory steps, and 300 AR(1) noise-dominated steps.Signals are normalized to zero mean and unit variance.
- Optimization and reproducibility: All experiments use identical optimization settings, full-batch AdamW with gradient clipping, and five deterministic random seeds.A runnable repository, synthetic data generator, exact seeds, and reproduction scripts are planned for release.
- Training procedure: GMS training begins with supervised regime-label routing warmup, anneals supervision over 15 epochs, then optimizes reconstruction alone.The full objective combines MSE, routing, entropy, load, and diversity terms.
B Theoretical Analysis: Spectral Contraction under Operator Mixing
The analysis shows that convexly mixing distinct stable spectral operators makes the effective dynamics more contractive, explaining smoothing and attractor collapse.
- Spectral contraction: Convex combinations of distinct stable spectral operators are inherently contractive, providing a mechanism for observed temporal smoothing and attractor collapse.The result concerns operators with different phases and stability-preserving spectral parameterization.
- Derivation: For two diagonal operators with common radius r < 1 and unequal phases, interpolation produces an effective operator whose eigenvalue magnitude is bounded by r.The bound follows from the triangle inequality.
- Derivation: Equality in the eigenvalue-magnitude bound occurs only when the phases match; differing phases make the contraction strict.Thus, phase diversity prevents the mixture from retaining the original spectral radius.
- Dynamical consequence: Repeated mixing dissipates energy and drives the effective system toward a spiral sink, producing temporal smoothing and attractor collapse.This links the theoretical contraction result to the empirical dynamics reported for Figure 3.
C Seed Sensitivity and Variance
Across five seeds, operator mixing is less reliable than the single-expert baseline: most runs favor the baseline, while GMS has worse mean NMSE and much higher variance.
- Experimental design: Five independent seeds evaluate K = 1 and K = 4 under identical training conditions to measure sensitivity to initialization.The reported seeds are 42, 100, 1337, 2024, and 999.
- Per-seed results: The majority of seeds favor the single-expert baseline, although individual GMS runs occasionally achieve competitive performance.Seeds 2024 and 999 are identified as instances where GMS outperforms the baseline.
- Aggregated results: The operator-mixing model has substantially higher variance and worse mean NMSE than the single-expert baseline across seeds.Table 6 reports the corresponding aggregated statistics.
- Variance: 0.00694 vs. 0.00069: GMS has a standard deviation an order of magnitude larger than the baseline.Despite this instability, GMS can outperform the baseline in specific favorable seeds.
- Regime-wise behavior: Experts differ mainly in contraction strength, while regime-wise errors trade improved chaotic prediction against degraded performance on simpler regimes.This comparison is shown for the K = 4 GMS model versus the K = 1 baseline.
D Oracle Routing and Frozen Expert Ablations
Routing quality and expert learning do not resolve the failure of operator-level mixing. Oracle routing improves chaotic-regime error but worsens global performance, while frozen experts show routing alone cannot compensate for poor operators.
- Oracle Routing: Oracle routing uses ground-truth regime labels to force timesteps onto their corresponding experts, removing routing ambiguity.The router receives full supervision throughout training.
- Oracle Routing: Despite correct assignments and active use of multiple experts, oracle routing worsens global performance relative to the single-expert baseline.Chaotic-regime gains are offset by regressions in the oscillatory and noise regimes.
- Oracle Routing: Oracle routing improves chaotic-regime performance while degrading the oscillatory and noise regimes, creating a structural trade-off rather than uniform specialization.The net effect is a regression in global NMSE.
- Frozen Experts: Frozen-expert ablations train only the router while fixing all spectral parameters at initialization.The fixed parameters include λ_k, B_k, C_k, and D_k.
- Frozen Experts: NMSE increases by nearly two orders of magnitude when experts are frozen, showing that routing alone cannot compensate for mismatched or suboptimal operators.Together, the ablations show that improved routing and decoupled expert learning do not resolve operator-mixing failure.
- Interpretation: Routing collapse is interpreted as an optimization response to an unfavorable inductive bias rather than the root cause of failure.The optimizer prefers a single stable operator over interpolating incompatible operators.
E Operator Mixing versus Output Mixing
The comparison holds architecture, routing, and optimization constant while changing only whether experts are mixed in operator space or output space. Both mixtures underperform a single expert, but output mixing avoids the severe degradation of operator interpolation.
- Experimental Setup: The study compares operator-level and output-level MoE models under matched expert architectures, routing networks, and optimization settings.The models use identical diagonal spectral state-space parameterizations and the same router architecture.
- Experimental Setup: Operator mixing forms an effective transition operator from router-produced convex weights over expert-specific state transitions.Output mixing instead combines predictions from experts that retain separate internal dynamics.
- Results: Both mixture models underperform the single-expert baseline, but output mixing is consistently more stable and substantially less degraded than operator mixing.Because routing, training, and capacity are shared, the performance gap isolates interpolation in operator space.
- Interpretation: Convex combinations of stable transition operators can produce averaged systems that suppress regime-specific structure, whereas output mixing preserves each expert’s internal dynamics.The proposed mechanism is destructive interference during system evolution under operator interpolation.
- Interpretation: Output mixing does not outperform the single-expert baseline, but avoids the catastrophic degradation observed with operator interpolation.The negative result is characterized as a structural limitation of operator-level mixing, not of MoE architectures generally.
F LLM Usage Statement
The authors state that large language models assisted with language editing and stylistic refinement, while the scientific work was produced by the authors.
- LLM Usage: Large language models were used for language editing and stylistic refinement of the manuscript.The statement attributes scientific content, experimental design, implementation, results, and analysis to the authors.