Source-linked AI summary
mHC: Manifold-Constrained Hyper-Connections
Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Kuai Yu, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, Wenfeng Liang
TL;DR
Hyper-Connections improve architectural expressiveness by widening residual streams and diversifying connectivity, but their unconstrained mappings can disrupt identity-preserving signal propagation and hinder scalable training. The paper proposes mHC, which projects residual mappings onto a doubly stochastic manifold and combines this constraint with infrastructure optimizations. Experiments report improved stability, downstream performance, and large-scale efficiency, including 6.7% additional time overhead at expansion rate n=4.
Problem
HC’s unconstrained connections compromise identity mapping, causing signal divergence, training instability, scalability limitations, and memory-access overhead.
Method
mHC projects HC residual connection space onto a manifold of doubly stochastic matrices and uses infrastructure optimizations for efficient training.
Results
mHC demonstrates improved training stability and scalability, while outperforming HC on most downstream tasks and adding 6.7% time overhead at n=4.
Takeaways & Limitations
mHC provides a generalized HC extension that preserves signal-conservation properties while retaining HC’s performance advantages and supporting large-scale training.
Abstract
from arXiv · showhide
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
1. Introduction
Residual connections support stable signal propagation, but Hyper-Connections expand stream width and connectivity in ways that compromise identity mapping and practical scalability. mHC constrains HC residual mappings onto a doubly stochastic manifold and adds infrastructure optimizations for efficient large-scale training.
- Residual connections: Residual connections preserve direct signal flow across layers, supporting stability and efficiency during large-scale training.The identity component maps a shallower-layer signal directly to a deeper layer without modification.
- Hyper-Connections: HC expands the residual stream from C to n×C and uses learnable mappings to mix, aggregate, and remap stream features.Hres mixes features within the widened stream, while Hpre and Hpost connect the stream to layer inputs and outputs.
- Challenges: Unconstrained HC mappings fail to preserve the global feature mean across depth, allowing signal amplification or attenuation that destabilizes large-scale training.The resulting instability, together with unresolved memory-access costs for widened streams, restricts HC’s practical scalability.
- Manifold-Constrained Hyper-Connections: mHC projects HC residual connection matrices onto the Birkhoff polytope, whose doubly stochastic matrices restore identity-like conservation properties.Equal row and column sums make residual propagation a convex combination of input features, conserving feature means and regularizing signal norms.
- Efficiency and scalability: Kernel fusion, mixed-precision kernels, selective recomputing, and communication overlap are used to reduce mHC’s infrastructure and memory overhead.In-house large-scale training reports only 6.7% additional time overhead at expansion rate n=4.
2. Related Works
Macro-design determines how representations propagate across network blocks, and recent work has increased topological complexity through connectivity, aggregation, and wider residual streams. Hyper-Connections exemplify this shift by using learnable matrices to modulate feature connections across depths.
- Macro-design: Macro-design establishes the inter-block topology that governs how representations are propagated, routed, and merged across layers.This contrasts with micro-design, which specifies processing inside computational blocks.
- Modern architecture: Transformers established Attention and Feed-Forward Networks as fundamental building blocks, with Attention enabling global information propagation.Sparse Mixture-of-Experts variants extend FFNs toward parameter scaling without proportional computational costs.
- Topological complexity: Post-ResNet architectures increased topological complexity through dense connectivity, multi-path structures, and recursive feature aggregation.DenseNet, FractalNet, and Deep Layer Aggregation represent successive examples of this macro-design direction.
- Wider residual streams: Recent macro-design research has expanded residual-stream width and introduced learnable matrices that modulate connection strengths among features at varying depths.Hyper-Connections are identified as an example of this wider-stream, more complex-connectivity paradigm.
3. Preliminary
HC expands the residual stream and introduces learnable mappings, but its unconstrained residual mapping creates numerical instability and substantial system-level overhead. The analysis motivates manifold constraints to preserve identity-like signal propagation while improving practical scalability.
- Hyper-Connections: HC expands the residual stream by a factor n and uses three learnable linear mappings to manage read-out, write-in, and update operations.The mappings are Hpre, Hres, and Hpost; n-stream residuals decouple residual-stream capacity from the layer input dimension.
- Hyper-Connections: HC’s residual mapping Hres yields the most significant performance gain among the ablated components.The result highlights the importance of information exchange within the residual stream.
- Numerical Instability: Across layers, HC’s unconstrained composite mapping deviates from identity, allowing forward signals and backward gradients to explode or vanish.The resulting signal divergence destabilizes training in deeper or larger-scale models.
- Numerical Instability: HC exhibits a loss surge around the 12k step that is highly correlated with instability in the gradient norm.The observation is reported relative to mHC in large-scale experiments.
- System Overhead: n: HC increases residual-stream memory-access cost by an approximately proportional factor, while also increasing activation storage and pipeline-parallel communication.Without fused kernels, the additional I/O degrades throughput; larger communication costs create larger pipeline bubbles.
4. Method
mHC constrains HC’s residual mappings to the doubly stochastic Birkhoff polytope, preserving signal stability while allowing stream interaction. It combines this manifold projection with kernel, memory, and communication optimizations for efficient large-scale training.
- Manifold constraint: mHC projects residual mappings onto the Birkhoff polytope, whose doubly stochastic matrices preserve identity-like signal behavior while permitting information exchange.The manifold is the convex hull of permutation matrices, so residual mappings act as convex combinations of permutations.
- Manifold constraint: Doubly stochastic mappings have non-negative entries with unit row and column sums, making each output stream a convex combination of input features.This structure conserves feature means and regularizes signal norms during propagation.
- Theoretical properties: The constraint provides norm preservation and compositional closure: spectral norms are bounded by 1, and products remain doubly stochastic.These properties mitigate gradient explosion and preserve the constraint across multiple layers.
- Parameterization and projection: mHC parameterizes dynamic and static mappings from flattened hidden states, then applies Sigmoid-based processing and Sinkhorn-Knopp normalization to obtain constrained mappings.Sinkhorn-Knopp alternately rescales rows and columns after exponentiating the matrix; experiments use t_max = 20.
- Efficient infrastructure design: Infrastructure optimizations fuse kernels, use mixed precision, recompute intermediate activations, and extend DualPipe to overlap communication with computation.For expansion rate n = 4, mHC introduces a 6.7% additional training-time overhead.
5. Experiments
Experiments compare mHC with HC and a baseline across training stability, downstream benchmarks, propagation behavior, and scaling. mHC improves stability and performance while retaining scalability across model and token budgets.
- Training Stability: mHC reduces the final loss by 0.021 versus the baseline while maintaining a gradient-norm profile comparable to the baseline.In 27B training, mHC mitigates HC’s instability in both loss and gradient norm.
- Downstream Performance: mHC consistently outperforms the baseline and surpasses HC on the majority of eight downstream benchmarks.Compared with HC, gains include 2.1% on BBH and 2.3% on DROP.
- Scaling Experiments: mHC maintains a robust performance advantage over the baseline across compute budgets spanning 3B, 9B, and 27B parameters.The advantage shows only marginal attenuation at higher computational budgets.
- Stability Analysis: Representative mappings show that HC’s large maximum gains coincide with broadly significant gains across propagation paths.mHC’s constrained mappings are evaluated through forward signal gain and backward gradient gain.
- Stability Analysis: mHC’s propagation gain remains bounded near 1.6, compared with HC’s nearly 3000 maximum gain magnitude.The constrained mapping substantially reduces forward and backward propagation instability.
6. Conclusion and Outlook
The paper presents mHC as a manifold-constrained extension of HC that restores stable signal propagation through doubly stochastic residual mappings. It reports improved large-scale training and scalability, while positioning broader manifold choices and macro-architecture design as future directions.
- Conclusion: mHC projects HC’s residual connection space onto a manifold and uses doubly stochastic constraints to transform propagation into convex combinations of features.This design restores the identity mapping property and supports stable signal flow.
- Conclusion: Efficient infrastructure optimizations allow mHC to deliver its stability and scalability improvements with negligible computational overhead.The paper identifies infrastructure efficiency as part of the practical contribution.
- Outlook: Future work can explore manifold constraints tailored to learning objectives and study their trade-off between plasticity and stability.The authors also connect this direction to macro-architecture design and topological effects on optimization and representation learning.
A.1. Detailed Model Specifications and Hyper-parameters.
The experiments use DeepSeek-V3-based 3B, 9B, and 27B models with specified HC and mHC expansion, projection, optimization, and training configurations.
- Model Specifications: Table 5 specifies model architectures and hyper-parameters for 3B, 9B, and 27B DeepSeek-V3-based models.It includes residual-stream expansion, Sinkhorn-Knopp settings, optimization, and training protocols for HC and mHC.