PaperlayerView exact revision on arXiv

Source-linked AI summary

xHC: Expanded Hyper-Connections

Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Xiaoming Shi, Huaijin Wu, Yebin Yang, Zhuo Xia, Shaofeng Zhang, Lin Yao, Yuliang Liu, Yu Cheng, Junchi Yan

arXiv:2607.14530v1cs.LGcs.CL

TL;DR

Existing HC-family methods typically stop at N=4 because larger mHC expansions offer diminishing returns while increasing cost. xHC addresses this with temporal feature augmentation and sparse residual updates, improving downstream performance and making large-N expansion more cost-effective and practical.

  • Problem

    Existing HC-family methods typically stop at N=4 because scaling mHC further yields diminishing loss improvements while training FLOPs increase substantially.

  • Method

    xHC combines temporal feature augmentation for richer write-back with sparse residual updates that activate k=4 of N=16 streams while retaining dense access.

  • Results

    48.8 average downstream score versus 44.8 with mHC, while xHC adds only 4.1% training FLOPs relative to the vanilla baseline at 18B MoE scale.

  • Takeaways & Limitations

    xHC makes residual-stream expansion beyond N=4 more cost-effective, while xHC-Flash brings per-sublayer memory traffic close to mHC at N=4.

  • Takeaways & Limitations

    Directly scaling mHC beyond N=4 remains limited by insufficient write-back information and O(N^3C) residual-mapping generation cost.

Abstract

from arXiv · show

Hyper-Connections (HC) expand the residual stream of Transformers into $N$ parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from $N{=}1$ to $N{=}4$ suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at $N{=}4$. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with $N$. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond $N{=}4$. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only $k=4$ of the $N=16$ streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC, while adding only modest training FLOPs over the vanilla baseline. Scaling-law experiments show that the vanilla and mHC require $1.50\times$ and $1.19\times$ the compute of xHC, respectively, to reach the same loss. Practical large-$N$ training also requires controlling memory traffic from the expanded residual state. We therefore introduce xHC-Flash, which reduces the per-sublayer memory traffic from $73.5C$ to $40C$, comparable to the $34C$ required by mHC at $N{=}4$, while retaining the gains of full xHC. Together, xHC and xHC-Flash make large-$N$ residual-stream expansion effective and practical for LLM pre-training.

1 Introduction

xHC makes residual-stream expansion beyond N=4 effective by enriching write-back information and sparsifying residual updates, overcoming mHC’s information and cubic-cost bottlenecks. It improves loss, downstream performance, compute efficiency, and large-N memory traffic, while xHC-Flash makes the approach practical.

  • Expansion efficiency: At N=16 versus N=4, xHC cuts loss by 0.012 with 4% extra FLOPs, whereas mHC cuts loss by 0.006 with 32% extra FLOPs.This comparison is reported on a 2.5B MoE model.
  • Empirical results: 1.758 final loss and 48.8 average downstream score: xHC surpasses mHC and vanilla while adding only 4.1% training FLOPs over vanilla at 18B MoE scale.mHC reaches 1.776 loss and 44.8 average downstream score, while vanilla reaches 1.799 loss.
  • Scaling efficiency: To reach the same loss, vanilla requires 1.50× xHC’s compute and mHC requires 1.19×, demonstrating systematic compute efficiency across model scales.Large-scale validation covers 18B and 28B MoE models with consistent downstream improvements.
  • Efficient deployment: xHC-Flash reduces per-sublayer memory traffic from 73.5C to 40C, near mHC’s 34C at N=4, while retaining most full-xHC performance gains.It shares routing and dense-read computation across sublayers and uses N=16, k=4.
  • Bottleneck diagnosis: mHC’s diminishing returns arise from limited write-back information and the cubic cost of dense residual mixing as N grows.Each layer supplies only one write-back signal despite the need for diverse stream histories.
  • Method: xHC enables meaningful expansion beyond N=4 through temporal feature augmentation for richer write-back and sparse residual updates for affordable mixing.The sparse architecture updates only k=4 of N=16 streams while retaining dense access to the full residual state.

2 Related Work

Prior work primarily carries representations through Transformer layers with a single residual stream, while sparsity has been used to reduce computation through expert routing or token-restricted attention. xHC instead applies sparsity along a different residual-related dimension.

  • Cross-Layer Information Flow: Most LLMs retain a single residual stream, with related methods improving its optimization, weighting, cross-layer access, or feature reuse.Residual connections carry token representations across Transformer layers, while gates, learned scaling, depth-dependent rescaling, and activation reuse modify information flow.
  • Sparse Computation in Language Models: Sparsity commonly scales language models through expert routing over computation or restricted attention over tokens.Mixture-of-Experts decouples total parameters from per-token FLOPs, while sparse attention reduces long-context attention cost.
  • Sparse Computation in Language Models: xHC applies sparsity on a different dimension: the residual stream.

3 Method

xHC makes large-N residual-stream expansion practical by enriching write-back signals and sparsely updating only k=4 of N=16 streams while retaining dense access to the full state. It targets mHC’s diminishing returns beyond N=4, caused by insufficient write-back diversity and O(N^3C) residual-mixing costs.

  • 3.1 Hyper-Connections: mHC stabilizes multi-stream training by projecting residual mappings onto the Birkhoff polytope, preserving stable information propagation across depth at N=4.Sinkhorn normalization enforces row and column sums of one for the residual mapping.
  • 3.2 Scaling Saturation: Increasing mHC from N=4 to N=16 reduces loss by only 0.006 while increasing training FLOPs by 32%.This diminishing benefit-cost tradeoff motivates a different approach to scaling residual-stream expansion.
  • 3.2 Scaling Saturation: mHC saturates because every stream receives only one write-back component, making additional streams increasingly redundant as N grows.Larger expansion requires more diverse write-back components to form non-redundant histories.
  • 3.3.2 Sparse Residual-Stream Architecture: mHC’s residual-mixing projection costs O(N^3C), whereas xHC restricts generation to the k active streams and costs O(k^3C).At N=16, mHC adds roughly 32% FLOPs over N=4 for limited loss reduction on a 2.5B MoE model.
  • 3.3 xHC: xHC combines temporal feature augmentation with sparse residual mixing, reading all N=16 streams but updating only k=4 active streams.This reduces residual-mapping generation from O(N^3C) to O(k^3C) while preserving dense connectivity to the full residual state.
  • 3.3.1 Temporal Feature Augmentation: Temporal augmentation expands the write-back basis with causal depthwise convolutions at kernel sizes 4, 8, and 12, producing Kr=4 components.It is applied only after MLP layers because attention already mixes positions and post-attention augmentation destabilized training empirically.

4 Experiments

Across 18B and 28B MoE models, xHC outperforms mHC and the vanilla residual baseline with modest training-cost overhead. Scaling-law, expansion-rate, ablation, and optimizer experiments show that xHC improves efficiency, benefits from larger N, addresses information and cost bottlenecks, and remains effective with Muon.

  • Scaling Laws: 1.1% lower loss than mHC and 2.4% lower than vanilla at the largest compute point, with xHC maintaining the lowest fitted loss curve across the measured range.Figure 4 compares matched scaling-law curves trained from 1.7 × 10^19 to 4.0 × 10^20 FLOPs.
  • Downstream Results: At 18B, xHC raises the average score from 44.8 with mHC to 48.8, while at 28B it raises the score from 50.5 to 53.6 (+3.1).xHC outperforms both mHC and the vanilla baseline at both scales; at 28B, it adds 3.0% training FLOPs over vanilla.
  • Expansion-Rate Sweep: Loss decreases consistently as xHC expands from N=2 to N=16, with each doubling producing clear improvement at small additional FLOPs.Unlike mHC, xHC enlarges residual-memory capacity while keeping sparse update cost controlled by k.
  • Ablations: Temporal feature augmentation increasingly improves loss as N grows, while Dense Read and fixed streams preserve information connectivity for sparse residual updates.Without Dense Read and fixed streams, loss degrades to 1.997; Dense Read keeps every stream accessible despite routing.
  • Optimizer Robustness: xHC maintains substantial gains over the Muon baseline across benchmarks, showing compatibility with Muon beyond AdamW.The Muon evaluation applies Muon to backbone parameters while retaining AdamW for xHC parameters and removes Gram–Schmidt orthogonalization from temporal feature augmentation.

5 Practical Deployment

xHC adds modest compute but incurs substantial memory traffic from its expanded residual state. xHC-Flash and fused kernels reduce this overhead while preserving xHC’s performance advantage over mHC.

  • Overhead analysis: At the default N=16, k=4 configuration, xHC adds 3.0% training FLOPs at the 28B scale, while memory traffic dominates runtime overhead.Expanded residual states require additional I/O for normalization, projection, and residual mixing at every sublayer.
  • xHC-Flash: xHC-Flash amortizes full-state operations across consecutive sublayers, reducing traffic from 73.5C to 51C and then 40C with its four-sublayer extension.The design uses shared routing, separate sublayer-specific pre-mappings, dense-read reuse, and deferred residual mixing.
  • xHC-Flash: xHC-Flash-4sub reduces memory traffic to 40C, near mHC’s 34C at N=4, while retaining a loss advantage over mHC (1.984 vs. 2.004).Full xHC uses 73.5C per sublayer, whereas xHC-Flash matches its validation loss of 1.983 at 51C.
  • Kernel fusion: Fused implementation stages combine operations with shared inputs to reduce memory traffic and kernel launches in the memory-bound N=16, k=4 workload.Residual states and projection operands use bfloat16, while normalization statistics, routing, mapping coefficients, and Sinkhorn iterations use float32.

6 Conclusion

xHC makes residual-stream expansion beyond the common N=4 setting more effective and affordable by addressing mHC’s write-back and cubic residual-mapping bottlenecks. It combines temporal feature augmentation with sparse updates that preserve dense read access.

  • xHC targets effective, affordable residual-stream expansion in HC-family models beyond the common N=4 setting.
  • Directly scaling mHC is limited by insufficient write-back information and the cubic cost of residual-mapping generation.
  • xHC combines temporal feature augmentation with sparse residual-stream updates that preserve dense read access while updating only k active streams.

A Experimental Details and Hyperparameters

The experiments use matched MoE architectures, optimization recipes, and data budgets while configuring xHC with 16 streams and 4 active streams per layer. Training stability is improved through Sinkhorn row-sum clamping, and multi-stream states are reduced to one vector before prediction.

  • Model Architectures: The MoE backbones use a DeepSeekMoE-style design with one leading dense layer, routed and shared experts, top-8 sigmoid routing, GQA, and QK Norm.Detailed model configurations are provided in Table 6.
  • xHC Configuration: xHC uses N=16 streams with k=4 active per layer, including 2 fixed streams and 2 dynamically selected streams.A linear router scores streams from the LayerNorm-normalized flattened residual state, while temporal feature augmentation uses r=3 causal depthwise-convolution branches.
  • xHC Configuration: Row-sum clamping after Sinkhorn normalization stabilizes xHC training without degrading performance.The procedure rescales only rows whose post-normalization sums exceed one, addressing rare extreme activations that can amplify the forward signal.
  • Training Hyperparameters: xHC, mHC, and vanilla baselines use matched optimization recipes and data budgets, including AdamW with gradient clipping and a warmup–stable–decay schedule.AdamW uses β1=0.9, β2=0.95, ϵ=10^-15, weight decay 0.1, and clipping at 1.0; the schedule has 500 warmup steps and exponential decay.
  • Final Stream Reduction: Before the language-model head, both mHC and xHC sum the N streams within each token and apply the resulting C-dimensional vector to final RMSNorm and unembedding.This reduction restores the multi-stream hidden state to a single hidden representation for prediction.

B Scaling Law Experiment Configuration

The scaling-law experiment compares vanilla, mHC, and xHC across four model sizes trained with matched recipes and compute ranging from approximately 1.7 × 10^19 to 4.0 × 10^20 FLOPs. It fits a shifted power law to each method’s final language-model loss using a fixed irreducible-loss estimate of E = 0.72.

  • Scaling-law experiment setup: Four model sizes per method span training compute from approximately 1.7 × 10^19 to 4.0 × 10^20 FLOPs under matched recipes.The compared methods are the vanilla baseline, mHC, and xHC.
  • Scaling-law experiment setup: All models share a DeepSeekMoE-style template with 144 routed experts, top-8 routing, one shared expert, and one leading dense layer, while width, depth, and expert FFN size vary.The configurations also use context length 8192, RoPE with θ=50000, 500 warmup steps, and the same data mixture.
  • Scaling-law fitting: The fits use L(C) = AC^-α + E on each method’s final language-model loss, with C as training FLOPs and E = 0.72 as the estimated irreducible loss.Linear regression is performed on log2(L − E) against log10 C, then converted back to the shifted power-law form; fitted parameters are reported in Table 9.

C Training FLOPs and Parameter Overhead Analysis

This section analyzes the per-layer parameter overhead and training FLOPs introduced by xHC and mHC.

  • The analysis derives the per-layer parameter overhead introduced by xHC and mHC.
  • The section reports the resulting training FLOPs cost for xHC and mHC.
  • The overhead analysis covers both xHC and mHC at the per-layer level.

C.1 Per-Layer Parameter Overhead

xHC reduces per-layer parameter overhead at large expansion by activating only k streams while adding richer temporal write-back components. At N=16 and k=4, xHC uses 1256C overhead versus 9216C for dense mHC, while retaining N=16 residual-memory capacity.

  • xHC: xHC adds a stream router and pre-mapping generator scaling as N^2C, a residual-mapping generator scaling as k^3C, and rank-Kr post-mapping generation with convolution only in the MLP sublayer.Temporal augmentation is applied only after the MLP sublayer, while the attention sublayer uses rank-1 post-mapping generation.
  • mHC: Dense mHC activates all N streams, omits routing, temporal augmentation, and convolution, and its 2N^3C residual-mapping term drives cubic cost scaling.For the representative budgets, PxHC,N=4,k=2 is 124C versus PmHC,N=4 at 192C, while PxHC,N=16,k=4 is 1256C versus PmHC,N=16 at 9216C.
  • Comparison: 1256C versus 9216C: xHC at N=16, k=4 uses 7.3× fewer per-layer parameters than dense mHC at N=16 while retaining the same residual-memory capacity.Under the representative setting Kr=4 and kernel sizes {4, 8, 12}, PxHC,N=16,k=4 has 1256C overhead, compared with PmHC,N=16 at 9216C.

C.2 Training FLOPs Overhead

At N=16, xHC adds only modest training FLOPs, unlike dense mHC’s approximately 19% overhead at 18B. Its overhead decreases with model scale because backbone cost grows with width while PxHC scales linearly in C.

  • Training FLOPs Overhead: The 6× factor accounts for forward and backward passes through parametric projections, while non-parametric operations are analyzed separately.Separate analyses cover Sinkhorn normalization, residual mixing, multi-stream LayerNorm, and Gram–Schmidt projections.
  • Training FLOPs Overhead: The overhead percentage is computed as Δ% = FHC/Fbackbone × 100, with Fbackbone including linear-layer, sequence-dependent attention, and LM-head costs.The backbone uses the standard 6× approximation for linear layers, attention cost 6SnqdhL, and LM-head cost 6V C.
  • Training FLOPs Overhead: At 18B, dense mHC at N=16 requires approximately 19% training-FLOPs overhead, whereas xHC incurs only modest overhead across the main model scales.Table 10 reports parameter and training-FLOPs overhead across methods and scales.
  • Training FLOPs Overhead: xHC’s training-FLOPs overhead decreases with model scale because Fbackbone grows with width while PxHC scales only linearly in C.The parameter overhead follows the same trend, supporting the modest cost of large-N expansion.

D Additional Ablation Studies

Additional ablations examine where temporal feature augmentation is applied, the role of Gram–Schmidt orthogonalization, and the benefit of multiple temporal convolution branches. Under dense mHC at N=16, adding one branch improves validation loss from 1.998 to 1.989, while three branches further improve it to 1.984.

  • Experimental setup: The additional ablations use the same 10B MoE setting as Section 4.5 and report validation loss on the Pile test set unless otherwise specified.The multi-scale branch ablation is separately conducted with dense mHC at N=16 without sparse residual updates.
  • Attention-side Temporal Feature Augmentation: xHC applies temporal feature augmentation after MLP layers, not attention layers, because MLPs process tokens independently and offer a cleaner site for lightweight local context injection.Attention already performs content-dependent token mixing, whereas the MLP sublayer provides a more localized setting for the augmentation.
  • Effect of Gram–Schmidt Orthogonalization: The study tests Gram–Schmidt orthogonalization to remove components collinear with the original layer output and keep augmented write-back components distinct.The motivation is to avoid amplifying the original write-back direction and to make write-back scaling less predictable.
  • Effect of multi-scale temporal components: 1.998 to 1.989 validation loss follows from adding one causal convolution branch, while three different-kernel branches further reduce loss to 1.984.The ablation uses dense mHC at N=16 without sparse residual updates, isolating the effect of enriching the write-back signal.

E Design Analysis of xHC-Flash … E.3 Exactness, Complexity, and Design Trade-offs

xHC-Flash exactly reconstructs later sublayer inputs from precomputed dense readouts and sparse active-stream updates without rereading all N streams. Its approximation concerns shared routing, entry-state pre-mappings, and deferred residual mixing, while xHC-Flash-4sub extends the correction to multi-component write-backs without an accumulated-delta buffer.

  • E Design Analysis of xHC-Flash: xHC-Flash exactly recovers later sublayer inputs from precomputed base readouts and sparse write-backs without rereading the full N-stream state.This removes Attention-side residual mixing and generalizes from the two-sublayer variant to xHC-Flash-4sub.
  • E.1 Two-Sublayer Dense-Read Reuse: The two-sublayer correction reduces to a scalar–vector multiply and addition after the base readout, because the preceding Attention write-back has one component.The scalar is assembled token-wise from dynamically generated mapping coefficients.
  • E.2 Four-Sublayer Extension: xHC-Flash-4sub forms four pre-mappings from the group-entry state while sharing one routing decision across the group.Later sublayers consume both Attention and MLP write-backs, requiring the multi-component correction described for four-sublayer groups.
  • E.2 Four-Sublayer Extension: For MLP write-backs with K_r=4 temporal components, xHC-Flash-4sub uses active-stream state plus a non-active base readout instead of storing accumulated deltas.This avoids an extra [S, B, k, C] buffer and still reconstructs the same input as applying preceding sparse write-backs to the group-entry readout.
  • E.3 Exactness, Complexity, and Design Trade-offs: The dense-read correction is exact when routing and pre-mappings are fixed at sharing-window entry, intermediate residual mixing is absent, and write-backs affect only active streams.Final-MLP residual mixing remains compatible because its input has already been formed.
  • E.3 Exactness, Complexity, and Design Trade-offs: Relative to full xHC, xHC-Flash approximates dynamic stream control by sharing routing, using entry-state pre-mappings, and deferring residual mixing to the final MLP.The approximation is in the control schedule, not in the dense-read correction itself.
  • E.3 Exactness, Complexity, and Design Trade-offs: The approximation is motivated by short sharing windows and retains dense read access, separate sublayer-specific pre-mappings, and sparse write-back to active streams.xHC-Flash uses one Transformer block per sharing window, while xHC-Flash-4sub uses two blocks.
Loading 2607.14530v1…