PaperlayerView exact revision on arXiv

Source-linked AI summary

Output Dilution: Redundant but Fragile Representations in MoE Models

Orion Reblitz-Richardson

arXiv:2608.25231v1cs.LGcs.AIcs.CL

TL;DR

The paper asks whether MoE models concentrate moral features in specialized experts and tests this with expert-level probing, perturbation, and dense-model comparison. In OLMoE-1B-7B, moral information is broadly encoded but 4.2× more fragile than in a matched dense model, which the paper attributes to 74× output dilution.

  • Problem

    The paper asks whether MoE architectures create expert-level moral specialization that could provide discrete alignment intervention points.

  • Method

    The paper applies moral probing and fragility testing to OLMoE-1B-7B and compares it with dense OLMo-2 1B using identical inputs and methodology.

  • Results

    MoE moral encoding matches dense probing accuracy but is 4.2× more fragile because the MoE feedforward contribution is 74× smaller than the dense MLP’s.

  • Takeaways & Limitations

    In sparse architectures, redundant moral encoding does not imply robust encoding, so fragility testing complements probing accuracy as an alignment metric.

  • Takeaways & Limitations

    The study examines only OLMoE, so generalization to other MoE architectures and the prediction that higher sparsity increases fragility remain untested.

Abstract

from arXiv · show

Mixture-of-Experts (MoE) models appear to encode moral content as robustly as dense models, yet prove far more fragile in their encoding. In OLMoE-1B-7B, linear probes recover moral valence from nearly every expert-layer combination, with mean peak-layer accuracy above 90%. But these representations collapse under levels of activation noise that a dense model of matched size easily tolerates, with a 4.2-fold difference in robustness. We trace this to output dilution. Because the MoE block averages across active experts before contributing to the residual stream, the feedforward signal reaching downstream layers is nearly two orders of magnitude smaller than in a dense MLP. Moral information, our interest, survives aggregation intact but at a scale trivially overwhelmed by perturbation. Routing itself remains stable under noise while the vulnerability originates entirely in the diluted aggregate. Checkpoint trajectories confirm this is architectural, not learned. Experts never specialize and accuracy saturates within the first few thousand steps. In sparse architectures, redundant encoding does not imply robust encoding.

1 Introduction

The paper finds that moral content is broadly and accurately encoded across MoE experts, but the encoding is substantially more fragile than in a matched dense model because MoE outputs are diluted before entering the residual stream.

  • Findings: MoEs do not create expert moral specialization: 1,020 of 1,024 per-expert probes exceed 75% accuracy, and every expert exceeds 84% at its peak layer.Expert accuracy has Gini coefficient below 0.03 at all layers, while the router’s maximum moral-content preference is 1.8%.
  • Findings: OLMoE matches dense OLMo-2 1B at 99.0% peak probing accuracy, yet its moral encoding is 4.2× more fragile.Mean critical σ∗ is 0.92 for OLMoE versus 3.81 for the dense model; individual expert representations and routing remain robust in isolation.
  • Findings: The MoE block’s aggregated output is 74× smaller than the dense MLP output, so identical absolute noise can overwhelm the MoE moral signal while leaving the dense signal intact.The aggregate is a top-8 weighted average of 64 expert outputs, measured through feedforward output scale.
  • Findings: Specialization never emerges during training: per-expert accuracy remains broadly distributed from step 5K through step 1.2M.Peak-layer accuracy is 92.1% at step 5K and 93.7% at step 1.2M, while the Gini coefficient stays between 0.012 and 0.018.
  • Implications: The results show that probing accuracy can reveal present information without measuring how securely that information is encoded.The paper therefore treats fragility testing as a complement to probing accuracy, especially across architectures with different internal signal scales.
  • Contributions: The paper contributes expert-level moral probing, quantification of MoE output dilution, and a controlled dense-versus-MoE comparison using identical probing methodology.Experiments use base, non-instruct models.

2 Related Work

Prior work established MoE expert specialization in linguistic, domain, and multilingual features, while this paper extends moral probing and activation-robustness methods to an open MoE model.

  • Expert specialization: Prior MoE studies examined specialization in syntax, part of speech, domain, language, and token-type features.The cited work includes linguistic, domain, multilingual, and utilization-pattern analyses.
  • Expert specialization: Before this paper, MoE expert specialization had not been examined for moral or ethical features.This motivates testing whether moral content concentrates in discrete experts.
  • Moral probing: Moral probing uses lightweight classifiers on internal representations to test whether moral reasoning features are encoded.The methodology is grounded in Moral Foundations Theory and extends companion dense-model work.
  • Representational robustness: Activation perturbation studies motivate Gaussian noise injection as a way to assess representation stability and identify causally relevant features.The paper’s fragility protocol defines critical noise as a quantitative robustness metric for alignment-relevant features.
  • Dense-model comparison: Companion dense-model work found early, broad moral encoding and a fragility gradient that continues after probing accuracy saturates.Prior suppression experiments also motivated testing whether MoE structure reduces feature redundancy.
  • OLMo ecosystem: OLMo and OLMoE’s open checkpoints and infrastructure enable controlled architectural comparison, output-scale measurement, and training-trajectory analysis.OLMoE’s unusually extensive checkpoint record is central to the trajectory analysis.

3 Methodology

The methodology compares matched dense and sparse OLMo models using identical moral data, probes, perturbation procedures, output-scale measurements, and selected training checkpoints.

  • Models and comparison design: OLMoE-1B-7B is a 16-layer model with 64 experts per layer, top-8 routing, 6.9B total parameters, and 1.3B active parameters per token.Each expert is a gated MLP, and routing uses a learned projection with softmax and top-k selection.
  • Models and comparison design: OLMo-2 1B provides the dense architectural control, sharing the lab, training philosophy, layer count, hidden dimension, and comparable active parameter count.Both models are base, non-instruct checkpoints.
  • Experimental controls: Both models use the same 240-pair moral dataset, probe architecture, and fragility protocol, making architecture the independent variable.Identical inputs also support the output-scale comparison without data-driven differences.
  • Activation collection: Per-expert probing bypasses the router and computes all 64 expert outputs from mean-pooled pre-MoE hidden states.The computation is batched across experts, while router analysis separately uses router logits.
  • Activation collection: Clean aggregated outputs use softmax router weights, top-8 selection, normalized weights, and a weighted sum of selected expert outputs.This produces the MoE block output used for downstream probing.
  • Fragility protocol: Fragility testing injects Gaussian noise at σ ∈ {0.1, 0.3, 1.0, 3.0, 10.0}, averages 10 seeds, and defines σ∗ as the first level below 0.6 accuracy.Layers that never cross the threshold are censored at the grid maximum when aggregating.
  • Fragility protocol: MoE perturbation targets are router logits, individual expert outputs, or the final aggregated output, with clean-trained probes evaluated on perturbed test outputs.Results are averaged over 10 random seeds per noise level.
  • Output-scale measurement: Output-scale analysis measures component and feedforward-output standard deviations across the same test texts for both architectures.The measurement is used to interpret component fragility results.

4 Results

OLMoE matches dense OLMo-2 in moral probing accuracy, but its encoding is substantially more fragile. The evidence points to uniformly distributed expert representations and a much smaller aggregated residual-stream signal as the key architectural difference.

  • 4.1 Dense vs. MoE: Same Accuracy, Different Robustness: 99.0% peak probing accuracy is achieved by both OLMoE and dense OLMo-2, with moral content decodable from every layer.OLMoE peaks at layer 13 and OLMo-2 at layer 12; OLMoE is weaker only in early layers.
  • 4.1 Dense vs. MoE: Same Accuracy, Different Robustness: 4.2× greater fragility separates OLMoE from dense OLMo-2 despite identical peak probing accuracy.Mean critical noise is σ∗=0.92 for OLMoE versus 3.81 for OLMo-2.
  • 4.2 No Expert Moral Specialization: 1,020 of 1,024 per-expert probes exceed 75% accuracy, showing that moral encoding is uniformly distributed rather than concentrated in a sparse expert subset.At the peak layer, all 64 experts exceed 84% accuracy and the Gini coefficient ranges from 0.016 to 0.023.
  • 4.3 Router Preference: 1.8% is the router’s maximum moral routing preference, indicating that routing does not segregate moral inputs to specific experts.The 95th percentile preference across expert-layer combinations is below 2%.
  • 4.4 Output Dilution Explains MoE Fragility: 74× smaller feedforward output scale makes the aggregated MoE signal easier for noise to overwhelm than the dense MLP signal.The aggregated output is the most fragile component, while the router is most robust; the smaller residual-stream contribution operates on a fundamentally different scale.
  • 4.5 Specialization Never Emerges During Training: 92.1% peak-layer accuracy is already present at step 5K and remains within a 92–94% band through step 1.2M without expert concentration.Across 11 checkpoints, peak-layer Gini remains between 0.012 and 0.018, while expert identity changes near randomly between adjacent checkpoints.

5 Discussion

The discussion argues that MoE architecture does not make moral representations easier to target, while output dilution creates a major robustness concern. It also identifies scope and methodological limits that constrain how broadly the findings should be interpreted.

  • Output Dilution as an Architectural Property: 74× smaller feedforward output makes MoE moral signals more vulnerable to perturbation than dense signals.The gap is attributed to sparse expert aggregation and routing-weight attenuation; the measured scale concerns the feedforward contribution to the residual stream.
  • Interpretability and Intervention: MoE models provide no moral experts to prune, fine-tune, or monitor because moral encoding is distributed across modules.This distribution leaves MoE no more tractable for targeted moral intervention than dense models.
  • Probing Accuracy as an Alignment Metric: The same high probing accuracy can conceal a large robustness difference: MoE encoding is 4.2× more fragile despite near-perfect accuracy and broad layer coverage.Probing measures whether information is extractable, whereas fragility testing measures how much perturbation the encoding withstands.
  • Probing Accuracy as an Alignment Metric: OLMoE’s early layers reach 79–86% per-expert accuracy versus 94–97% for OLMo-2, and tightened datasets widen the diagnostic concern.The discussion links this pattern to diluted feedforward contributions and greater reliance on shallow features when dataset artifacts are available.
  • Limitations: The findings are limited to one MoE family, mean-pooled representations, linear probes, a non-identical dense comparison, and English Moral Foundations Theory data.Generalization to other architectures, token-level routing effects, nonlinear probes, fully matched training conditions, and other languages or moral frameworks remains unresolved.

6 Conclusion

The study finds no expert-level moral specialization in OLMoE, but identifies a structural fragility caused by output dilution. This fragility persists through training and motivates testing alternative aggregation mechanisms.

  • 6 Conclusion: All 64 experts encode moral content comparably, while the router shows no preference for routing moral inputs to specific experts.The reported Gini coefficient is below 0.03, and maximum router preference is 1.8%.
  • 6 Conclusion: MoE moral encoding is 4.2× more fragile than dense encoding despite matching probing accuracy, because sparse aggregation attenuates the residual-stream signal.The MoE contribution is 74× smaller in scale than the dense MLP’s, leaving the moral signal easily overwhelmed by noise.
  • 6 Conclusion: The absence of specialization is already present at the earliest checkpoint and remains throughout training, indicating that training does not create or remove it.At step 5K, moral encoding is already observed, while the Gini coefficient remains below 0.03 throughout training.
  • 6 Conclusion: The architectural gap between probing accuracy and fragility is permanent under the studied aggregation mechanism rather than merely a temporal training phenomenon.The conclusion attributes this persistent gap to sparse aggregation and states that resolving it requires changing the aggregation mechanism.
  • 6 Conclusion: Higher sparsity, fine-tuning, and alternative expert-output aggregation are predicted to produce distinct fragility profiles.The proposed alternatives include concatenation, attention-based mixing, and denser routing.

A Probing dataset construction

The probing dataset is a deterministic, balanced subsample of a larger Moral Foundations Theory dataset, with matched moral and neutral sentences filtered for quality and neutrality.

  • A Probing dataset construction: The study uses a deterministic 240-pair subsample of a 1,200-pair parent dataset constructed with published quality guidelines and LLM-assisted filtering.The parent dataset and construction pipeline are described in companion work.
  • A Probing dataset construction: The parent dataset covers six Moral Foundations Theory foundations with 200 sentence pairs per foundation.The foundations are care/harm, fairness/cheating, loyalty/betrayal, authority/subversion, sanctity/degradation, and liberty/oppression.
  • A Probing dataset construction: Each pair matches a moral sentence with a neutral sentence preserving syntactic structure and topic domain while removing moral content.Neutral sentences are filtered for naturalness and moral neutrality.
  • A Probing dataset construction: Pairs pass length-ratio, keyword-filtering, and deduplication gates before release of the parent dataset.Neutral sentences contain no explicit moral keywords under the filtering procedure.
  • A Probing dataset construction: The 240 pairs are sampled with seed 42 and split into 192 training pairs and 48 test pairs while preserving foundation balance.The corresponding text counts are 384 training texts and 96 test texts.

A.4 Dataset identity across experiments

All experiments use the same balanced 240-pair dataset and shared protocols, isolating architecture as the independent variable. Per-expert probes show broadly uniform moral encoding rather than specialization.

  • A.4 Dataset identity across experiments: The dense-versus-MoE comparison, per-expert probing, routing analysis, and output-scale comparison all process the identical 240-pair subsample.This design ensures observed differences are architectural rather than data-driven.
  • A.4 Dataset identity across experiments: The per-expert probe table reports accuracy distributions across all 16 layers, including mean, standard deviation, minimum, maximum, Gini, and threshold counts.“Above 90%” and “Below 60%” count experts relative to those accuracy thresholds on the 96-text test set.
  • A.4 Dataset identity across experiments: 1,020 of 1,024 probes exceed 75% accuracy, no expert falls below 60%, and the Gini coefficient never exceeds 0.023.The probes span 64 experts across 16 layers.
  • A.4 Dataset identity across experiments: A Gini coefficient measures distributional inequality, with 0 representing perfect equality and 1 representing maximum inequality.The observed values are contrasted with hypothetical mild and strong specialization ranges.
  • A.4 Dataset identity across experiments: The observed Gini range is an order of magnitude below the hypothetical mild-specialization range, supporting the conclusion that moral features are not concentrated in specific experts.The observed no-specialization range is 0.016–0.023, versus 0.05–0.15 for the hypothetical mild case.

B.3 Router analysis details

Router moral preference is measured by comparing mean router logits for moral and neutral inputs across tokens, with routing differences reported across experts and layers.

  • B.3 Router analysis details: Router moral preference is the token-averaged difference in mean router logit between moral and neutral inputs.The maximum preference across all 64 experts and 16 layers is 1.8%.
  • B.3 Router analysis details: Top-8 expert selection frequencies for moral and neutral inputs differ by less than 0.5% at every layer.This frequency comparison complements the router-logit preference measure.

C Output scale measurement methodology

Feedforward output scale is measured by hooking each layer’s MLP before residual addition. The extraction differs by architecture: OLMoE uses the aggregated output from its returned tuple, while OLMo-2 exposes the MLP tensor directly.

  • Forward hooks capture each layer’s feedforward output before residual addition, isolating its contribution to the residual stream.
  • OLMoE analysis records the first element of the MLP return tuple, which contains the aggregated output.
  • OLMo-2 analysis records the MLP output tensor directly rather than selecting it from a tuple.

C.2 Scale metric

The scale metric is the across-input standard deviation of mean-pooled feedforward outputs. OLMo-2 outputs are much larger than OLMoE outputs, with ratios varying sharply by layer and averaging 74×.

  • The metric is feedforward-output standard deviation across 100 test texts drawn from the first 50 training pairs.Mean pooling averages across the sequence dimension before variability is measured.
  • 74× is the mean OLMo-2-to-OLMoE output-scale ratio across layers, ranging from 5× at layer 2 to 167× at layer 0.The low layer-2 ratio reflects an unusually large MoE output there, possibly from early-layer representational adjustments.
  • OLMo-2 MLP output grows from 0.28 at layer 2 to 7.99 at layer 15, spanning approximately 29×.
  • OLMoE aggregated output spans 0.003 at layer 0 to 0.089 at layer 15, while layerwise ratios reflect routing and aggregation dynamics.
  • The scale gap mechanistically explains the fragility gap because noise calibrated to the residual scale overwhelms MoE outputs that are 74× smaller.Dense MLP outputs are comparable to the residual, whereas the MoE aggregate is much smaller.

D Reproducibility

The experiments are designed for local, reproducible execution with fixed models, seeds, documented scripts, and released metadata. They run on a single MacBook Pro using Apple’s MPS backend, with approximately 2.5 hours of compute.

  • All experiments run on a single MacBook Pro M4 Pro with a 12-core CPU, 24 GB unified memory, and an M4 Pro GPU through PyTorch MPS.
  • Total runtime is approximately 2.5 hours of MPS compute time, excluding model downloads; each OLMoE checkpoint is approximately 14 GB.
  • Experiment runtimes range from approximately 3 minutes for Experiments 1+2 to approximately 1.5 hours for the 11-checkpoint trajectory analysis.
  • A minimal MPS and CPU fallback for torch.histc counts tokens per expert without affecting numerical results or gradient computation.
  • The study documents dataset and experiment seeds, model revisions, base-checkpoint precision, commands, dependencies, and pinned package versions.
  • Each experiment produces structured JSON metadata, and the code, scripts, and outputs are publicly released in the paper’s repository subdirectory.
Loading 2608.25231v1…