Source-linked AI summary
How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
Pengxiang Zhao, Xing Li, Xianzhi Yu, Wei Guo, Zhenhua Dong
TL;DR
The paper asks how a trained multi-stream mHC model uses its available routing and mixing capacity. It analyzes DeepSeek-V4-Flash with depth-resolved diagnostics and inference-time interventions, finding concentrated, shifting routing, distinct stream representations, and limited measured benefit from late residual mixing. The results localize most functional dependence to early mixer structure and show that retaining three routing weights is usually sufficient on the evaluated metrics.
Problem
How trained mHC models use broad routing, residual mixing, and multiple stream representations remains unclear, particularly beyond smaller HC models with dominant-stream behavior.
Method
The paper measures effective routing width, residual mixing, and inter-stream cosine similarity across DeepSeek-V4-Flash, then tests their functional importance with inference-time interventions.
Results
Replacing late mixers with identity raises C4 perplexity by only 1.9% while preserving the six-task average score, whereas replacing early mixers raises perplexity by 41%; retaining three routing weights raises perplexity by at most 2.7%.
Takeaways & Limitations
The four-stream pathway shows structured under-utilization: blocks rarely require all four streams, and late residual mixing provides little measured benefit.
Takeaways & Limitations
The checkpoint analysis cannot distinguish weak functional demand from weak gradient transmission, and causal mechanisms require training-time routing and mixer trajectories.
Abstract
from arXiv · showhide
Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only 1.9% and preserves the six-task average score, whereas replacing the early mixers increases perplexity by 41%. Fixing each early mixer to its C4 diagnostic mean increases perplexity by only 0.2% and reduces the average score by 0.25 percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most 2.7% and changes the average score by at most 0.4 points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.
1 INTRODUCTION
This study examines how DeepSeek-V4-Flash uses the flexibility of four-stream mHC, finding concentrated but depth-varying routing, distinct stream representations, and functionally important early mixing but limited late mixing.
- Motivation: mHC permits broad routing and cross-stream mixing, motivating tests of how trained models actually allocate computation across streams.The analysis asks whether routing is broad or concentrated, whether residual states mix, and whether streams are distinct.
- Approach: The study measures effective read/write width, residual weights, and inter-stream similarity across tokens, sublayers, and depth, then applies inference-time interventions.These diagnostics distinguish globally dominant streams from layer-dependent stream use and connect structure to model quality.
- Routing and representations: About two of four streams are effectively used at a typical attention or FFN site, while the dominant stream changes across layers and representations remain directionally distinct.The results indicate concentrated, depth-varying routing rather than uniform use or complete stream collapse.
- Routing interventions: Retaining the three largest routing weights per token increases C4 perplexity by at most 2.7% and changes the average downstream score by at most 0.4 percentage points.By contrast, retaining only two weights increases perplexity by 12–14% and reduces the average downstream score by 2.9–6.6 percentage points.
- Residual mixing: Late residual mixers can be replaced by identity with a 1.9% C4 perplexity increase while preserving the six-task average score, whereas replacing early mixers increases perplexity by 41%.Fixing early mixers to their C4 diagnostic means changes perplexity by only 0.2% and the average score by 0.25 points, indicating limited measured value from token-wise variation.
2 BACKGROUND AND RELATED WORK
Hyper-Connections expand a single residual stream into n coupled streams using learned read, write, and residual maps. mHC constrains residual mixers to preserve stability while still allowing cross-stream exchange.
- Residual streams and Hyper-Connections: Hyper-Connections maintain n coupled states and use a read map, write map, and residual mixer at each residual sublayer.The read map forms the branch input, the write map distributes branch output, and the residual mixer recombines incoming stream states.
- Residual streams and Hyper-Connections: Each realized map combines learned static parameters with a token-dependent component, so routing can vary by token and sublayer.The resulting token- and sublayer-specific values are called realized maps.
- Manifold-constrained mixers: Residual mixers compose multiplicatively across successive sublayers, allowing unconstrained products to amplify signals with depth.mHC addresses this cumulative transformation by constraining each realized mixer to the Birkhoff polytope.
- Manifold-constrained mixers: Under the exact doubly stochastic constraint, mixers have spectral norm one and their cumulative product remains non-expansive.mHC uses Sinkhorn–Knopp normalization to implement the projection, while still learning mixers rather than fixing them to identity.
- Related work: Earlier HC studies found dominant-stream behavior and near-identity residual mixing in smaller nanoGPT models, but model scale and training regime may alter this organization.These findings motivate testing whether concentration and near-identity mixing generalize to trained mHC models.
3 MEASURING REALIZED MULTI-STREAM COMPUTATION
The paper measures realized multi-stream computation through routing breadth, residual mixing, and inter-stream representation similarity, then tests functional necessity with inference-time interventions.
- Measurement framework: The analysis characterizes read/write routing, residual mixing, and stream-representation similarity across tokens, sublayers, and depth.It combines diagnostic statistics with routing sparsification and residual-mixer replacement interventions.
- Read/write routing breadth: Winner consistency measures how often tokens share the same highest-weight stream within a sequence.A value of 1 means every token in every sequence shares the same winner; lower values indicate token-varying dominant streams.
- Read/write routing breadth: Load-effective stream count measures how many streams carry the aggregated normalized routing load.The statistic equals 1 for fully concentrated load and n for uniform load, and is averaged over diagnostic sequences.
- Inference-time interventions: Figure 3’s top-k routing intervention retains the largest realized routing weights and rescales them to preserve their original sum.The intervention changes routing support while preserving total routing weight and is applied separately to read and write maps.
- Cross-stream residual mixing: Identity deviation quantifies residual mixing: it is zero for independent identity skip paths and increases with off-diagonal cross-stream weight.The metric is averaged across tokens within each sublayer to produce a depth profile.
- Inter-stream representation similarity: Mean pairwise cosine similarity assesses whether stream representations are directionally aligned or separated.Higher values indicate closer alignment, while lower values indicate greater directional separation; the measure is averaged across tokens by sublayer.
4 EXPERIMENTS
Experiments on DeepSeek-V4-Flash show concentrated, depth-varying routing, distinct stream representations, and residual mixing concentrated in early layers. Interventions indicate that late mixing and the weakest routing paths contribute little measured benefit, while early mixer structure remains important.
- Experimental setup: The study instruments 86 attention and MoE-FFN sites in the 43-layer, four-stream mHC pathway of DeepSeek-V4-Flash.The checkpoint has 284B total parameters, 13B activated parameters, and was pretrained on more than 32T tokens.
- Routing organization: 1.998 read and 1.775 write effective streams show that typical routing spans about two of four streams.Winner consistency averages 0.871 for reads and 0.905 for writes.
- Routing organization: Dominant routing streams reorganize across depth: streams 0 and 1 dominate 32 of 44 early sites, while streams 2 and 3 dominate 36 of 42 late sites.Read and write winners differ at 38.4% of the 86 sites, so routing schedules are coupled but not identical.
- Representation geometry: Mean inter-stream cosine similarity is 0.404 beyond layer 0, with pair averages ranging from 0.288 to 0.560, showing directional separation without representation collapse.The streams start identical but rapidly diverge, and no pair remains uniformly aligned across depth.
- Residual mixing: Early residual mixers have identity deviation 0.046 versus 0.009 in layers 22–42, where near-diagonal maps perform little direct cross-stream exchange.Across all sites, identity deviation averages 0.028; 32 of 40 sites in layers 23–42 fall below 0.01.
- Functional interventions: Replacing late mixers with identity raises C4 perplexity by only 1.9%, whereas replacing early mixers raises it by 41.4%; retaining three routing weights raises perplexity by at most 2.7%.Fixing early mixers to site-specific C4 means raises perplexity by only 0.2% and changes the six-task average by −0.25 percentage points.
5 DISCUSSION AND IMPLICATIONS
The discussion interprets concentrated routing and near-identity late mixing as structured under-use of mHC capacity rather than complete stream collapse. It also identifies training-dynamics questions and proposes alternatives whose value requires matched training comparisons.
- Implications: Routing is concentrated within sublayers but reorganizes across depth, while residual mixing becomes nearly identity after mid-depth.These properties motivate further analysis of how training allocates multi-stream capacity.
- Routing mechanism: A possible feedback loop could reinforce routing preferences because read weights affect branch-mediated gradients and write maps select receiving streams.The proposed mechanism remains a candidate because its coefficients are coupled through shared token-dependent routing.
- Limitations: The checkpoint cannot distinguish weak functional demand for cross-stream exchange from weak gradient transmission to off-diagonal mixer entries.Resolving this distinction requires training-time routing and mixer trajectories.
- Design directions: A proposed alternative retains input-conditioned H_pre while allowing H_post and potentially H_res to condition additionally on branch output.Evaluating it requires matched training comparisons with full and simplified alternatives.
6 CONCLUSION
The conclusion finds structured under-utilization in the four-stream pathway: routing is locally narrow but depth-varying, stream states remain distinct, and residual exchange is mainly early. Interventions show that late mixing and weakest routing paths can be removed with little measured loss, unlike early mixer structure.
- Conclusion: DeepSeek-V4-Flash exhibits locally concentrated but depth-varying routing, directionally distinct stream representations, and residual exchange concentrated in early layers.These findings come from true-forward measurements across the trained mHC pathway.
- Conclusion: The weakest routing weight per token and late residual mixers can be suppressed with little measured loss, while remaining routing support and early mixer structure are functionally important.Fixing early mixers to C4 diagnostic means nearly preserves quality, indicating greater importance for site-specific structure than token-wise variation on evaluated metrics.
- Conclusion: The four-stream pathway shows structured under-utilization rather than uniform participation or complete stream collapse.Effective multi-stream capacity depends on component- and depth-specific organization learned during training.
A.1 DIAGNOSTICS BY DEPTH AND SUBLAYER TYPE
Depth-split diagnostics show that residual mixing declines consistently across attention and FFN sites, while routing breadth changes mainly in write maps, especially at FFN sites.
- Depth-split diagnostics: Residual mixing reduction is consistent across attention and FFN sites under the common depth split.Table 3 places routing, representation, and residual-mixing diagnostics under the same depth ranges.
- Depth-split diagnostics: Routing breadth changes are concentrated in write maps, particularly at FFN sites.Winner consistency and effective stream count are averaged over the indicated routing sites, while cosine similarity averages over six stream pairs.
A.2 IDENTITY OF THE LOWEST-LOAD ROUTE
The weakest routing stream changes across sites and depth rather than remaining globally fixed, so no single stream is consistently dispensable throughout the network.
- All four streams are the lowest-load stream somewhere across map and sublayer-site combinations.The locally weakest route is therefore not a single globally removable stream.
- The weakest stream changes across depth, favoring site-adaptive sparsification over removing one fixed stream.Its mean load is generally small but varies across depth.
B LAYERWISE RESIDUAL MIXERS
Residual mixing is concentrated at selected early and middle layers, then largely disappears after mid-depth, with attention and FFN sites sharing a late near-identity regime but differing in where early mixing occurs.
- Attention sites: Attention residual mixers show the strongest visible exchanges at layers 1–2, 9, 11, and 22, then remain near diagonal through layer 42.The off-diagonal structure is depth- and pair-specific before the late near-identity regime.
- FFN sites: FFN residual mixers have strongest off-diagonal weights at layers 0–1 and 13, while layers 23 onward remain close to identity.The early high-mixing locations and stream pairs differ from attention sites.
- Comparison: Attention and FFN sites share a late near-identity regime despite differing in which early layers and stream pairs mix.Figures 9 and 10 expose the stream pairs responsible for residual exchange at each site.
C LOCAL OPTIMIZATION ANALYSIS
The analysis represents routing and residual mixing through realized token- and sublayer-specific maps, then examines how local optimization pressure and Sinkhorn normalization can govern movement away from identity.
- Routing maps: The realized read map aggregates stream states into a branch input, while the write map distributes the branch output back across streams.For a token and sublayer, the branch computes a weighted input, applies F, and adds weighted outputs to persistent streams.
- Local optimization: Read preferences can reinforce themselves because larger realized read weights scale branch-mediated gradients through the selected upstream stream.The write map participates similarly by controlling which persistent states receive branch output and influence later computation.
- Residual mixing: The realized residual mixer is parameterized as H_res = S(A), with Sinkhorn normalization applied to pre-Sinkhorn logits.The local optimization analysis studies how gradient updates to logits affect movement away from identity.
- Residual mixing: Movement away from identity may remain small because cross-stream exchange receives little loss pressure or because the local Sinkhorn Jacobian weakly transmits that pressure.The final checkpoint cannot distinguish which factor governed the late-layer mixers or produced their observed regime.