Hyper-Connections

Defa Zhu; Hongzhi Huang; Zihao Huang; Yutao Zeng; Yunyao Mao; Banggu Wu; Qiyang Min; Xun Zhou

Source-linked AI summary

Hyper-Connections

Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, Xun Zhou

arXiv:2409.19606v3Sep 29, 2024cs.LGcs.CLcs.CVcs.NE

TL;DR

Residual-connection variants trade off gradient vanishing against representation collapse because their connection strengths are predefined. Hyper-connections learn depth- and width-connections, optionally conditioned on inputs, and experiments show improvements across language-model pre-training and vision tasks. In the reported OLMoE experiment, DHC converges 1.8 times faster and improves ARC-Challenge by 6 points after 500B tokens.

Problem
Pre-Norm and Post-Norm residual connections trade off gradient vanishing and representation collapse, while residual pathways predefine connection strength.
Method
Hyper-connections use trainable or input-predicted (n + 1) × (n + 1) connection matrices to adjust feature connections across depths and widths.
Results
1.8 times faster convergence and 6 points higher ARC-Challenge accuracy are reported for DHC than the baseline after 500B tokens, with benefits also reported across language and vision tasks.
Takeaways & Limitations
Hyper-connections are presented as an alternative to residual connections with reported benefits for language-model pre-training, image generation, and image classification.

Abstract

from arXiv · show

We present hyper-connections, a simple yet effective method that can serve as an alternative to residual connections. This approach specifically addresses common drawbacks observed in residual connection variants, such as the seesaw effect between gradient vanishing and representation collapse. Theoretically, hyper-connections allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers. We conduct experiments focusing on the pre-training of large language models, including dense and sparse models, where hyper-connections show significant performance improvements over residual connections. Additional experiments conducted on vision tasks also demonstrate similar improvements. We anticipate that this method will be broadly applicable and beneficial across a wide range of AI problems.

1 INTRODUCTION

Residual connections trade off gradient stability against representation collapse because they predefine connection strengths, while hyper-connections learn flexible depth- and width-connections. Experiments report faster convergence, improved benchmark performance, and lower adjacent-layer similarity with hyper-connections.

Motivation and contribution: Hyper-connections learn connection strengths and improve performance with negligible additional computation and parameters, while encompassing Pre-Norm and Post-Norm as non-trainable special cases.They are proposed to address the trade-off between vanishing gradients and representation collapse.
Core idea: Hyper-connections use learnable depth- and width-connections to integrate features vertically across depths and exchange information laterally between hidden vectors.The network expands its input into multiple copies, each with its own depth connection.
Empirical evidence: 1.8 times faster convergence and 6 points higher ARC-Challenge accuracy are reported for OLMoE-1B-7B-DHC×4 than the baseline after 500B tokens.The comparison also reports a significant advantage at 500B tokens and superior HellaSwag and ARC-Challenge accuracy curves.
Visualization analysis: Hyper-connections produce lower and more varied adjacent-layer feature similarity than the Pre-Norm baseline, suggesting that individual layers retain greater impact.The baseline tends toward representation collapse, characterized by high similarity between adjacent-layer features.

2 METHOD

The method represents hidden states as an expanded hyper-hidden matrix and applies connection matrices that mix information across depths and widths. Static connections use fixed weights, while dynamic hyper-connections predict input-dependent weights with stabilized transformations.

Static hyper-connections: Hyper-connections replicate the initial hidden vector n times into a hyper-hidden matrix, process it through each layer, then sum the final rows to form the network hidden vector.The resulting vector is passed through a final projector, such as normalization and unembedding in transformers.
Static hyper-connections: Connection matrices assign weights to current-layer outputs and prior hidden states, enabling depth-connections and width-connections to mix information across layers and hidden vectors.Depth-connections perform weighted sums involving layer outputs, while width-connections connect the hyper-hidden states.
Dynamic hyper-connections: Dynamic hyper-connections make connection-matrix entries depend on the input hyper-hidden matrix H rather than remaining fixed.The dynamic matrices are combined with static matrices in the practical DHC implementation.
Dynamic hyper-connections: Dynamic parameters are produced by linear transformations with normalization, tanh activation, and a small initial learnable scale to stabilize training.The paper reports that dynamic hyper-connections outperform static variants in language-modeling experiments.
Initialization: The initialization sets dynamic parameters to zero and chooses static matrices so hyper-connections initially match Pre-Norm residual connections.The layer index and modulo operation determine the stated initialization pattern.

3 WHY HYPER-CONNECTIONS

Hyper-connections reinterpret residual variants as non-trainable connection matrices and extend them with learnable weights that can rearrange layers between sequential and parallel configurations.

3.1 RESIDUAL CONNECTIONS AS NON-TRAINABLE HYPER-CONNECTIONS: Pre-Norm and Post-Norm residual connections are non-trainable hyper-connections represented by distinct matrices at expansion rate n = 1.Pre-Norm uses fixed entries, while Post-Norm weights depend on input-output statistics.
3.1 RESIDUAL CONNECTIONS AS NON-TRAINABLE HYPER-CONNECTIONS: Hyper-connections generalize residual connections with trainable or input-predicted (n + 1) × (n + 1) matrices.These matrices can assign connection weights and extend the fixed residual formulations.
3.2 SEQUENTIAL-PARALLEL DUALITY: The method learns to blend sequential and parallel layer arrangements rather than committing to either configuration.The figure illustrates sequential and parallel arrangements with n = 2.
3.2 SEQUENTIAL-PARALLEL DUALITY: With suitable hyper-connection matrices, layers can form sequential arrangements or parallel groups of two consecutive layers.The sequential case degenerates to a residual connection, while the parallel case resembles parallel transformer blocks.
3.2 SEQUENTIAL-PARALLEL DUALITY: Learnable hyper-connection matrices can create soft-mixture or dynamic layer arrangements beyond traditional sequential and parallel configurations.Static arrangements remain fixed after training, whereas dynamic arrangements can adapt for each token.

4 RESULTS

Experiments across dense and MoE language models show that dynamic hyper-connections improve performance, stability, and training efficiency over residual baselines when sufficiently expanded. Visualization analyses further show that hyper-connections learn flexible layer arrangements and connection patterns that combine Pre-Norm- and Post-Norm-like behavior.

4.1 ABLATION STUDY: 0.034 lower V2 Eval Loss and 0.029 lower V3 Eval Loss are achieved by OLMo-1B-DHC×8 without tanh compared with the baseline, while DHC with n ≥2 descends faster and shows no training-loss spikes.Performance is inferior at n = 1, best at n = 4, and gains from increasing to n = 8 are minimal overall.
4.1 ABLATION STUDY: All HC variants outperform the baseline, with DHC notably better than SHC at expansion rate 4, whereas their improvements are similar at expansion rate 2.The comparison is reported in the static-versus-dynamic hyper-connection ablation.
4.2 COMPARISON WITH RELATED WORKS: 0.021 higher V2 loss and 0.017 higher V3 loss result when WC is not trainable, making trainability of both WC and B important for performance.Not training B has a less pronounced effect than not training WC.
4.3 7B MODELS: 0.710 average downstream score exceeds the OLMo-7B baseline’s 0.701, while DHC×4 also improves V2 loss by 0.022 and PPL by 0.293.The 7B DHC×4 model significantly outperforms the baseline across all reported average metrics.
4.4 MOE MODELS: 6 points on ARC-Challenge and 1.2 points on MMLU Var. are among improvements from DHC×4 over residual connections, while many metrics reach baseline performance with half the training tokens.The same comparison reports reductions of approximately 0.027 in training loss and 0.028 on C4-en validation loss.
4.5 VISUALIZATION ANALYSIS: Hyper-connections learn layer arrangements beyond fixed residual paths, including parallelizable layer pairs, fewer long-term attention connections, and a Λ-shaped pattern mixing Pre-Norm- and Post-Norm-like behavior.The analyses also find that input word embeddings contribute to most layers but are eliminated from the final model output.

5 RELATED WORK

Hyper-connections are presented as an alternative to residual connections in transformers, whose variants trade off gradient vanishing against representation collapse. The approach aims to provide stable training and consistent improvements across language and vision tasks.

Residual connections facilitate deep-model training, but their limitations remain unresolved in transformers and CNNs.
Hyper-connections replace residual connections while providing stable training and consistent improvements in natural language processing and computer vision.
Pre-Norm mitigates gradient vanishing but can cause deep representation collapse, whereas Post-Norm reintroduces vanishing-gradient problems.

6 CONCLUSION

The paper concludes that hyper-connections are an effective alternative to residual connections and can dynamically adjust network architecture. Experiments report benefits across language-model pre-training, image generation, and image classification.

Hyper-connections are introduced as an effective alternative to residual connections in transformers.
Hyper-connections enable dynamic adjustments in network architecture and show benefits across language-model pre-training, image generation, and image classification.

B PARAMETERS, COMPUTATION AND MEMORY FOOTPRINT ANALYSIS

The analysis characterizes the parameter, computation, and memory costs of static and dynamic hyper-connections. It reports negligible parameter and computational overhead, with a minor activation-memory increase.

Static Hyper-Connections: Static hyper-connections use n · (n + 2) parameters per hyper-connection module, with two modules per layer.The modules correspond to self-attention and feedforward components.
Static Hyper-Connections: For OLMo-1B-SHC×4, the reported extra-parameter count is 768.
Dynamic Hyper-Connections: Dynamic hyper-connections have parameter count |θnorm| + dmodel × (n + 2) + n × (n + 2) + 2.In OLMo models, |θnorm| = 0; in OLMoE, |θnorm| = dmodel.
Computation Analysis: Hyper-connections introduce minimal additional parameters and computational overhead for both static and dynamic variants.The paper provides parameter and FLOPs comparisons in Tables 7 and 8.
Computation Analysis: The additional hyper-connection computation is O(dmodel × n × (n + 1)), which is negligible relative to attention projection and feedforward computation.
Memory Footprint: Hyper-connections add a minor activation-memory overhead, quantified separately in the measured-memory analysis.The corresponding measurements are reported in Table 9.

C MOE 1B/7B MODEL EXPERIMENTS

The supplied passages identify validation-loss and downstream-accuracy curves for OLMo and OLMoE models with and without DHC×4. They do not state the numerical or directional outcome of those comparisons.

Figure 9 compares V3 validation loss and downstream-task accuracy for OLMoE-1B7B and OLMoE-1B7B-DHC×4.
Figure 10 compares V3 validation loss and downstream-task accuracy for OLMo-7B and OLMo-7B-DHC×4.

E VISION EXPERIMENTS

Vision experiments show that hyper-connections improve image-generation efficiency and ImageNet classification accuracy, with dynamic variants especially strong in larger models.

E.1 IMAGE GENERATION: The image-generation study trains DiT models for 1400 epochs on ImageNet using FP16 precision, flash attention, and QK-Norm.These choices are described as cost-saving and training-stabilization measures.
E.1 IMAGE GENERATION: DiT models with hyper-connections achieve metrics comparable to DiT models with 50% more parameters, without increasing model size.The experiments use the DiT framework on ImageNet image generation.
E.2 IMAGE CLASSIFICATION: ViT classification uses 224 × 224 images for 300 epochs, replacing residual connections with static or dynamic hyper-connections.The experiments evaluate ViT/16-Base and ViT/16-Large models.
E.2 IMAGE CLASSIFICATION: 79.94% accuracy is achieved by the 307M-parameter ViT/16-Large-DHC model, versus 77.25% for the re-implemented baseline.The corresponding SHC model reaches 78.38%, while DHC gives the highest reported performance.
E.2 IMAGE CLASSIFICATION: 77.60% and 77.26% accuracy are obtained by Base-model SHC and DHC, respectively, versus 76.38% for the 85M baseline.These correspond to relative increases of 1.22% and 0.88%.

F MORE VISUALIZATION AND ANALYSIS

Visualization analyses show that hyper-connections preserve and reorganize intermediate representations through diverse connection patterns, while the single-copy variant can waste layers and weaken long-range connections.

F MORE VISUALIZATION AND ANALYSIS: The unfolded connection matrices are derived for the initial and subsequent hyper hiddens to reveal how layer outputs propagate through the network.The visualization procedure computes connection matrices recursively from the hyper-connection formulation.
F MORE VISUALIZATION AND ANALYSIS: Hyper hiddens exhibit different connection patterns, preserving FFN outputs longer than attention outputs and storing some long-term connections in opposing-sign pairs.The paired connections can cancel during sum-pooling before unembedding.
F MORE VISUALIZATION AND ANALYSIS: SHC and DHC share the same connection patterns, while SHC contains more PTB-like blocks that can be physically computed in parallel.SHC’s token-independent relations permit this reorganization.
F MORE VISUALIZATION AND ANALYSIS: HC×1 can waste layer 17 and lacks the Λ-shaped connection pattern supported by HC×2 and HC×4.The authors associate missing early-to-final connections with possible gradient vanishing and performance degeneration.

G DERIVATION OF NON-TRAINABLE HYPER-CONNECTION MATRIX FOR RESIDUAL CONNECTIONS

The derivation shows that residual-connection variants and several layer arrangements can be represented as non-trainable hyper-connection matrices, while the implementation maintains expanded hidden states and produces the final output by row summation.

G.1 RESIDUAL CONNECTIONS AS HYPER-CONNECTIONS: Pre-Norm and Post-Norm residual connections can each be represented by corresponding hyper-connection matrices.The Post-Norm equivalence assumes LayerNorm, with the RMSNorm analysis described as almost identical.
G.2 SEQUENTIAL ARRANGEMENT: A specified hyper-connection matrix produces n identical networks arranged sequentially with residual connections between them.The construction uses an n × n identity matrix, its column vectors, and a row vector of ones.
H.2 HYPER-CONNECTION MATRIX OF PARALLEL ARRANGEMENT: Another matrix construction arranges every n adjacent layers in parallel, with the resulting groups executed sequentially.The matrix sums outputs from the previous group and stores combined input-output values in hidden-vector slots.
IMPLEMENTATION: The implementation initializes an expanded hidden state, applies width and depth connections at each layer, then sums the final hidden-state rows before normalization and output projection.Dynamic parameters are computed from normalized hidden states, passed through tanh, and scaled by small learnable factors.
IMPLEMENTATION: The PyTorch-style transformer implementation applies hyper-connections separately around attention and FFN blocks.Each block computes width mixing, normalization, the block operation, and depth connection updates.

L 1B MODEL EXPERIMENTS

The 1B-model experiments compare training-loss behavior and validation or downstream metrics across related methods and DHC variants, with separate analyses for tanh and non-tanh configurations.

L 1B MODEL EXPERIMENTS: Training-loss curves are compared across related works, DHC with tanh, DHC without tanh, and parallel transformer blocks.The curves are smoothed using EMA with a decay rate of 0.99.
L 1B MODEL EXPERIMENTS: The 1B-model evaluation includes downstream benchmarks, validation losses, and validation perplexities across V2 and V3 validation sets.The benchmark configuration excludes grey benchmarks because their performance indicators are unstable.