PaperlayerView exact revision on arXiv

Source-linked AI summary

From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts

Kang Chen, Sihan Zhao, Yixin Cao, Yu-Gang Jiang

arXiv:2609.06403v1cs.AI

TL;DR

Existing analyses largely describe reasoning cohorts through final-answer signals, leaving their internal reorganization over inference time without a standard label-free account. The paper introduces routing effective rank from cross-rollout MoE routing similarity and finds a reproducible low–high–low trajectory, with common-mode reallocation explaining most motion while residual structure remains informative.

  • Problem

    Existing cohort analyses emphasize final answers, while the internal organization of reasoning trajectories over inference time lacks a standard label-free account.

  • Method

    The paper constructs an activation-only cross-rollout routing-similarity graph and measures its spectral diversity with routing effective rank, including an exact common-mode/residual decomposition.

  • Results

    Across ten MoE configurations and five math/science benchmarks, routing effective rank shows a low–high–low trajectory with an interior maximum in 98.5% of 3,105 model–question cohorts.

  • Takeaways & Limitations

    Common-mode concentration predicts same-answer recoverability, while higher reasoning effort delays and broadens the high-rank regime across all four tested architectures.

  • Takeaways & Limitations

    The confirmatory behavioral analysis covers 318 unique questions, and cross-family results are interpreted as rank-based trajectory comparisons rather than capability claims.

Abstract

from arXiv · show

Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds. We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity. Across ten MoE configurations and five math/science benchmarks, deff exhibits a reproducible low-high-low trajectory, with a prominent interior maximum in 98.5% of 3,105 model-question cohorts: routing similarity is concentrated early, maximally differentiated at intermediate budgets, and reconcentrated later, and the timing of this maximum varies systematically with architecture and reasoning effort. An exact decomposition separates cohort-wide common-mode mass from residual spectral dimensionality: common-mode reallocation accounts for about two thirds of the trajectory, while the residual spectrum contributes about one quarter and retains substantial variation beyond the common mode. The decomposition further localizes behavior: among non-unanimous cohorts, increases in common-mode concentration strongly predict same-answer recoverability, and higher reasoning effort delays the maximum by 2.59 octaves (doublings of the token budget) and consistently expands the high-rank period across all four tested architectures, locating the effort effect in timing and duration rather than peak amplitude. Correctness comparisons separate structural monitoring from answer selection, positioning routing effective rank as a decomposable, label-free diagnostic of cohort organization - a principled spectral lens on how MoE reasoning cohorts differentiate and reconcentrate over inference time.

1 Introduction

The paper introduces routing effective rank as a label-free, decomposable measure of how MoE reasoning cohorts reorganize over inference time. Across broad evaluations, it finds a reproducible low–high–low trajectory whose timing depends on architecture and reasoning effort, with common-mode and residual spectral channels explaining distinct aspects of the pattern.

  • Motivation: Test-time scaling produces cohorts of candidate reasoning trajectories whose internal organization can be studied alongside their aggregated answers.These cohorts are used by majority voting, verifiers, reward models, and group-based training.
  • Motivation: Routing traces provide an online, activation-only signal of cross-rollout organization, represented as a similarity graph whose spectrum describes cohort geometry.The traces arrive with decoding and can be read at any prefix.
  • Method: Routing effective rank is the entropy-effective dimensionality of the routing-similarity spectrum and is analyzed as a decomposable trajectory rather than an isolated scalar.The analysis separates temporal regularity, spectral channels, and behavioral links to answer organization and reasoning effort.
  • Results: 98.5% of 3,105 model–question cohorts show a prominent interior maximum in a robust low–high–low trajectory across ten MoE configurations and five math/science benchmarks.Peak timing varies systematically with architecture and reasoning effort, while expert-identity permutations remove the prominent arc.
  • Results: The exact spectral decomposition attributes roughly two thirds of the trajectory to common-mode mass reallocation and about one quarter to residual spectral dimensionality.The leading mode aligns almost perfectly with cohort-wide agreement, while the residual spectrum retains substantial variation.
  • Implications: Common-mode concentration predicts same-answer recoverability, whereas higher reasoning effort delays and broadens the high-rank regime across four architectures.These channels position routing effective rank as a label-free diagnostic of cohort organization and temporal routing geometry.

2 Related Work

Related work studies reasoning cohorts, internal computation, MoE routing, and spectral diversity from several complementary perspectives. This paper combines these lines by analyzing the temporal behavior of effective rank on routing-similarity graphs and decomposing its common-mode and residual channels.

  • Test-time cohorts and internal signals: Test-time reasoning cohorts are aggregated through majority vote, verifiers, compute allocation, group-based training, adaptive stopping, and selective prediction.Other work uses internal computation for selectors and verifiers through neuron agreement, hidden-state probes, and temporal signals.
  • MoE routing as an internal signal: Prior MoE studies examine specialization, load balancing, routing stability, prefetching, rollout selection, and what routing encodes about expert or hidden-state organization.The paper’s question concerns cross-rollout routing organization rather than those existing uses alone.
  • Effective rank and spectral diversity: Effective rank is the exponential of spectral entropy and has been used to diagnose representation collapse, Vendi diversity, and order-one Hill diversity.Here it is applied temporally to routing-similarity graphs with an exact common-mode/residual decomposition.

3 Method: Routing Effective Rank and Its Spectral Anatomy

The method converts MoE routing traces from rollout cohorts into a weighted-similarity graph and summarizes its normalized spectrum with routing effective rank. An exact entropy decomposition separates leading common-mode mass, residual dimensionality, and their interaction.

  • Cohorts and routing graph: Each cohort contains N=64 rollouts evaluated at 16 logarithmic token-prefix budgets, with routing histograms accumulated over trailing 16-token windows.Histograms use router gate weights over layer–expert slots; answer annotations locate commit windows but do not enter similarity computation.
  • Cohorts and routing graph: Weighted-Jaccard similarities between activation-only routing histograms form a symmetric cohort graph W.The graph indexes expert slots and separates routing geometry from answer labels.
  • Routing effective rank: Routing effective rank d_eff is the exponential of spectral entropy, quantifying the effective dimensionality of the routing-similarity spectrum.Low d_eff means a few shared routing patterns dominate, whereas high d_eff requires more independent spectral directions.
  • Exact spectral decomposition: The exact identity log d_eff = h(m) + (1−m) log d_⊥ decomposes effective rank into common-mode entropy and residual spectral dimensionality.Here m is the normalized leading-eigenvalue mass, while d_⊥ is the effective dimensionality of the renormalized residual spectrum.
  • Exact spectral decomposition: Trajectory segments are attributed exactly to mixing, common-mode mass changes, and residual-dimensionality changes through a midpoint decomposition of Δ log d_eff.The leading eigenvector is interpreted as the cohort-wide agreement direction, motivating the common-mode label.
  • Statistical design: The analysis treats each model–question cohort as the unit of analysis and uses grouped cross-validation plus two-way clustered bootstrap intervals.Budget steps within cohorts are not treated as independent, and predictive splits keep questions entirely within train or test.

4 Experiments

Across MoE configurations and math/science benchmarks, routing effective rank shows a reproducible low–high–low trajectory within individual cohorts. The experiments attribute most motion to common-mode mass, identify residual dimensionality as a distinct structural axis, and connect the channels to answer recoverability and reasoning effort.

  • RQ1: Trajectory: 5.21→18.68→5.84: pooled d_eff rises at the earliest budget, peaks at b=512, and returns near its initial level at the largest budget.The corresponding commit value is 5.76, consistent with concentrated routing similarity at the endpoints and broader spectral spread in the middle.
  • RQ1: Trajectory: 98.5% of 3,105 model–question cohorts contain a prominent interior maximum, with the qualitative arc present across benchmarks, configurations, and difficulty strata.The 95% CI is [96.7, 99.8], and the median relative prominence is 0.68 of peak height.
  • RQ1: Timing: Peak timing depends on model, dataset, and definition, while dataset ordering is nearly fixed across configurations, with GPQA peaking 1.5–2 octaves before math benchmarks.Family-averaged and per-cohort timing can differ by up to 1.5–2 octaves, so per-cohort medians are used for within-family timing.
  • RQ1: Controls: Expert-identity permutation reduces the prominent-peak rate from 98.5% to 0.0% and flattens d_eff toward cohort size, while still-generating controls preserve the interior maximum in 99.8% of cells.These controls isolate aligned cross-rollout routing organization rather than marginal activation volume as the source of the arc.
  • RQ2: Spectral anatomy: 0.666: common-mode mass accounts for the largest share of absolute normalized motion in log d_eff, versus 0.243 for residual dimensionality and 0.092 for mixing.The common-mode share exceeds the residual share in both early and late budget halves and remains stable under alternative axis partitions.
  • RQ2: Spectral anatomy: 63% of residual log d_⊥ variation remains unexplained by common-mode mass, establishing residual dimensionality as a distinct coordinate of routing geometry.A grouped cross-validated fit explains 37% of out-of-sample variance overall and motivates separate behavioral localization.
  • RQ3: Answer organization and effort: Common-mode increases strongly predict same-answer recoverability from routing among non-unanimous cohorts, whereas routing remains complementary to answer-side statistics for answer selection.The decomposition therefore distinguishes a behaviorally aligned common-mode channel from residual structural variation and label-free cohort monitoring.
  • RQ3: Answer organization and effort: +2.59 octaves: higher reasoning effort delays the maximum and expands the high-rank regime across all four tested architectures.It raises high-rank occupancy by +0.102 and widens the d_eff ≥0.8d_max interval by +1.29 octaves, while pooled peak-height contrast spans zero.

5 Robustness, Scope, and Limitations

The diagnostic is robust across several controls, but its absolute values and some behavioral claims remain bounded by representation, sampling, and analysis scope. The authors therefore frame conclusions around within-architecture trajectory shape and timing rather than capability.

  • The claims target within-architecture trajectory shape and timing because absolute d_eff depends on routing representation, expert count, and cohort size.
  • The low–high–low trajectory survives still-generating-rollout restriction, answer masking, and transfer to 1,582 LiveCodeBench-v5 problems.
  • Lexical and formatting cues may remain entangled with same-answer recoverability, while sensitivity to sampling temperature remains uncharacterized.
  • 318 unique questions under two-way clustering limit the resolution of confirmatory effect-size intervals, and cross-family comparisons are rank-based rather than capability claims.

6 Conclusion

The paper establishes routing effective rank as a label-free, decomposable view of how MoE reasoning cohorts reorganize over inference time. Across broad configurations and benchmarks, common-mode allocation dominates trajectory motion while effort shifts the timing and duration of the high-rank regime.

  • Routing effective rank provides a label-free spectral view of same-question MoE cohort reorganization over inference time.
  • Common-mode mass allocation drives most trajectory motion, while residual dimensionality retains substantial independent structure.
  • Higher reasoning effort delays and broadens the high-rank regime across all four tested architectures, while common-mode concentration predicts same-answer recoverability among contested cohorts.
  • The diagnostic survives frozen transfer to held-out code generation and can be computed from sparse expert-activation indicators before answers are graded.
  • The framework tracks when MoE cohorts differentiate, sustain distributed routing structure, and reconcentrate as inference compute unfolds.

C Difficulty Correlations

Difficulty associations are analyzed at commit-aligned windows and interpreted primarily through common-mode structure. The reported relationships vary by model family, while residual dimensionality adds little beyond common-mode mass in the tested regression.

  • Difficulty correlations use commit-graph d_eff and compare pooled analyses with within-shard means to control for cross-model capability differences.
  • Early budgets support no difficulty-triage claim; commit-window alignment is required because long-reasoning models may still be on the rising branch at b=4096.
  • Adding residual dimensionality after common-mode mass yields ΔR^2_{r|m}=+0.022 [−0.001, +0.050], with an interval spanning zero.
  • The authors interpret the descriptive correlations as difficulty associations rather than evidence for an additional residual-specific mechanism.
  • At b=4096, Table A2 reports effective-rank levels that rise monotonically with reasoning effort within each architecture.

D Trajectory Controls

Controls indicate that the trajectory is temporal, carried by shared expert identities, and linked to model-specific generation-length scales rather than a universal token budget. Risk-set and masking analyses preserve the main arc while refining how late-stage timing and collapse are interpreted.

  • The low–high–low trajectory could arise from marginal activation statistics, rollout completion mixtures, or aggregation, motivating the expanded controls.
  • Permuting expert identities preserves activation counts, load, and sparsity but reduces the prominent-peak rate from 98.5% to 0.0%, implicating cross-rollout expert co-activation identities.
  • Dataset ordering of peak timing is nearly invariant across configurations: GPQA peaks 1.5–2 octaves earlier than math benchmarks, while HMMT’25 has the widest high-rank period.
  • Restricting analysis to still-generating rollouts leaves an interior maximum in 99.8% of cells, but risk-set and full-graph peak positions correlate only moderately at ρ=+0.632.
  • Shuffling frame order lowers the prominent-peak rate from 98.5% to 56.2%, showing that the effect depends on temporal sequence rather than the marginal frame collection.
  • Peak location tracks generation length with rank correlation +0.94, so cross-model readings use per-cohort peak units rather than fixed token budgets.

F Spectral Anatomy: Details, Controls, and Attribution Tests

The exact spectral decomposition is numerically validated and attributes most trajectory motion to common-mode mass, while residual dimensionality contributes independently and varies across segments and architectures.

  • Identity and validation: 49,549 of 49,680 frames satisfy the decomposition, with maximum per-frame residual 5.7 × 10−14.The 131 frames at m≈1 have undefined d⊥; 2,999 cells have all 16 frames defined.
  • Attribution: +0.158 is the matched-cell contrast showing that residual share is larger early than late.The contrast uses 2,604 matched cells and remains supported under the pre-registered Bonferroni correction.
  • Controls: 88%/97% of the attribution split is reproduced after permuting peak positions, indicating that the accounting follows the budget axis rather than estimated peaks.Fixed-cut-point sweeps likewise produce similar splits without using cohort peaks.
  • Interpretation: The common-mode term’s nats-scale dominance reflects a 5.89× multiplier ratio, not uniformly faster common-mode motion.Early headroom-normalized motion is statistically indistinguishable between common-mode and residual factors.
  • Architecture contrasts: Residual-share differences across architecture families concentrate on descending and late branches, while early-segment differences remain unresolved.The reported family contrasts are descriptive because family and routing architecture covary.

G Behavioral Localization: Preregistered Plan and Predictive Ladders

The behavioral audit uses frozen out-of-fold predictive ladders and preregistered endpoints to localize routing-spectrum associations. It finds recoverability linked primarily to common-mode changes, while effort effects appear in timing and duration rather than pooled peak height.

  • Preregistered plan: The frozen ladder compares controls, density, common-mode mass, common-mode plus residual dimensionality, and scalar log deff using grouped cross-validation by question.The specification uses cubic B-splines and model×dataset fixed effects, cohort size, and cohort-accuracy covariates.
  • Trajectory attribution: 0.666 of trajectory motion is attributed to common-mode mass and 0.243 to residual dimensionality under the midpoint split.Table A6 defines these as shares of absolute Δ log deff, pooled over each segment’s cells.
  • Residual structure: The residual axis is only partially predictable from the common mode, preserving a distinct structural coordinate.This supports treating residual dimensionality as complementary rather than redundant.
  • Effort localization: The preregistered peak-height effort contrast has pooled interval +0.104 [−0.079, +0.232], spanning zero.Peak time, occupancy, integrated masses, and width are secondary localization endpoints.
  • Behavioral localization: +0.522 is the coefficient for changes in common-mode mass predicting changes in same-answer recoverability, versus −0.047 for residual changes.A 1-SD common-mode step moves recoverability roughly four times as much as a residual step.
  • Measurement considerations: Peak-frame rank estimation needs substantially larger subcohorts than terminal-frame ordering, where n≈8 is usable.The peak is the frame with the greatest spectral spread.
  • Correctness boundary: Answer-side baselines remain stronger than standalone routing features for correctness selection.The comparison separates structural monitoring from answer-selection use cases.

I Frozen-Coordinate Transfer to LiveCodeBench-v5

A frozen transfer to LiveCodeBench-v5 tests whether trajectory readouts generalize beyond math and science while preserving all analysis settings. Gross routing geometry transfers, but text-based answer-selection signals do not.

  • Transfer design: The frozen evaluation applies the same settings to 1,582 LiveCodeBench-v5 problems across the same ten configurations.The code benchmark was excluded from every analysis decision.
  • Rank fidelity: 0.881 is the cross-problem rank fidelity at n=8 and b=4096 on LiveCodeBench-v5, versus 0.866 for the matched within-configuration math value.The corpus-pooled ≈0.98 value is not a valid anchor because between-configuration levels dominate its ranking.
  • Correctness transfer: The routing-side correctness readout remains 0.407 within its null band after transfer.This contrasts with the domain-sensitive text channel.
  • Answer-selection scope: String-match vote margins fall to AUC 0.433 on code, compared with ≈0.83 on math.Functionally similar programs rarely share identical strings, so execution-based verification is the appropriate substitute.
  • Predictive bounds: Residual incremental contributions remain bounded at +0.04 for answer effective count and +0.02 for AP lift on tested targets.These are 95% upper bounds and remain compatible with small effects.
  • Selection versus monitoring: The strongest routing feature reaches AUC 0.542, far below text-side baselines of 0.82–0.91 for correctness selection.The comparison supports routing as a structural-monitoring signal rather than a primary answer selector.

J Representation Robustness: Binary Expert-Set Jaccard

Replacing weighted routing similarity with binary expert-set Jaccard preserves the paper’s soft and geometric findings across the matched cohort set. The representation changes absolute scale but not the main ordering or structural conclusions.

  • Robustness result: Binary expert-set Jaccard leaves every soft and geometric result unchanged or slightly better on the same 3,105 cohorts.The comparison is matched because both pipelines retain identical problem sets.
  • Scale and ordering: Family ordering is preserved, although absolute deff scale compresses from 9.2/18.8/25.0 to 7.5/14.9/19.6.The representation affects magnitude more than relative ordering.
  • Spectral validity: The symmetrized weighted-Jaccard matrices are positive semidefinite to numerical precision across 52,785 cohort graphs.The minimum eigenvalue is at least −5.8 × 10−14, and no graph falls below −10−12.
  • Subcohort measurement: Terminal-frame deff ordering is reliable from n≈8, whereas peak-frame fidelity rises from 0.20 at n=2 to 0.92 at n=32.The peak frame’s greater spectral spread makes it more demanding to estimate from subcohorts.
  • Interpretive boundary: Routing spectra provide a label-free view of internal cohort geometry, while margins, execution signals, and verifiers remain better suited to answer selection.The robustness comparison reinforces the division between structural monitoring and selection.

L.1 Structural Analysis: Routing Geometry Is Diffuse Rather than Partition-Like

Routing effective rank is descriptive rather than prognostic: it characterizes current cohort organization but does not reliably predict unanimity or correctness. Across tested selection and frontier protocols, answer-side signals outperform routing-based alternatives, concentrating routing’s value in label-free geometric calibration.

  • d_eff predicts majority-vote correctness at AUC 0.506, versus 0.832 for vote margin, while same-answer-support coherence is nearly identical for correct and incorrect answers.
  • Prefix routing state predicts eventual unanimity poorly, with pooled AUCs of 0.45–0.56, so it describes present organization rather than destination.
  • Soft-geometry weighting adds little once hard answer labels are available: graph-weighted extrapolation matches plain estimation, and density-weighted voting matches equal-weight voting.
  • Oracle-seeded propagation remains limited by diffuse membership, with 16-seed subgraph purity of 0.55–0.68 and ground-truth-seeded propagation capped at 0.616 for prefix and 0.692 for commit.
  • None of six d_eff voting integrations improves its corresponding answer-side baseline, and diversity reweighting slightly reduces commit majority-vote performance by −0.0072.
  • Routing-based levers do not reach the token–accuracy frontier: plain self-consistency peaks at N≈24 with 0.7613 accuracy at 40.9% cost, while routing adds no measurable gain to early stopping or confidence decisions.
Loading 2609.06403v1…