PaperlayerView exact revision on arXiv

Source-linked AI summary

Virtual Width Networks

Seed, Baisheng Li, Banggu Wu, Bole Ma, Bowen Xiao, Chaoyi Zhang, Cheng Li, Chengyi Wang, Chengyin Xu, Chi Zhang, Chong Hu, Daoguang Zan, Defa Zhu, Dongyu Xu, Du Li, Faming Wu, Fan Xia, Ge Zhang, Guang Shi, Haobin Chen, Hongyu Zhu, Hongzhi Huang, Huan Zhou, Huanzhang Dou, Jianhui Duan, Jianqiao Lu, Jianyu Jiang, Jiayi Xu, Jiecao Chen, Jin Chen, Jin Ma, Jing Su, Jingji Chen, Jun Wang, Jun Yuan, Juncai Liu, Jundong Zhou, Kai Hua, Kai Shen, Kai Xiang, Kaiyuan Chen, Kang Liu, Ke Shen, Liang Xiang, Lin Yan, Lishu Luo, Mengyao Zhang, Ming Ding, Mofan Zhang, Nianning Liang, Peng Li, Penghao Huang, Pengpeng Mu, Qi Huang, Qianli Ma, Qiyang Min, Qiying Yu, Renming Pang, Ru Zhang, Shen Yan, Shen Yan, Shixiong Zhao, Shuaishuai Cao, Shuang Wu, Siyan Chen, Siyu Li, Siyuan Qiao, Tao Sun, Tian Xin, Tiantian Fan, Ting Huang, Ting-Han Fan, Wei Jia, Wenqiang Zhang, Wenxuan Liu, Xiangzhong Wu, Xiaochen Zuo, Xiaoying Jia, Ximing Yang, Xin Liu, Xin Yu, Xingyan Bin, Xintong Hao, Xiongcai Luo, Xujing Li, Xun Zhou, Yanghua Peng, Yangrui Chen, Yi Lin, Yichong Leng, Yinghao Li, Yingshuan Song, Yiyuan Ma, Yong Shan, Yongan Xiang, Yonghui Wu, Yongtao Zhang, Yongzhen Yao, Yu Bao, Yuehang Yang, Yufeng Yuan, Yunshui Li, Yuqiao Xian, Yutao Zeng, Yuxuan Wang, Zehua Hong, Zehua Wang, Zengzhi Wang, Zeyu Yang, Zhengqiang Yin, Zhenyi Lu, Zhexi Zhang, Zhi Chen, Zhi Zhang, Zhiqi Lin, Zihao Huang, Zilin Xu, Ziyun Wei, Zuo Wang

arXiv:2511.11238v2cs.LGcs.AI

TL;DR

Wider representations can support richer functions, but increasing hidden dimensions incurs quadratic growth in parameters and compute. VWN expands token embeddings while keeping backbone hidden dimensions fixed; at 8× expansion, it reaches baseline next-token and next-2-token losses with 2.5× and 3.5× fewer tokens, respectively.

  • Problem

    Can wider representations be harnessed without the quadratic parameter and compute cost of naively increasing hidden dimensions?

    Source: §1 · p. 2

  • Method

    VWN expands token-embedding width while keeping Transformer backbone hidden dimensions fixed, decoupling representational width from backbone width.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    2.5× fewer tokens for baseline next-token loss and 3.5× fewer tokens for baseline next-2-token loss at 8× embedding-width expansion.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The approximately log-linear relation between virtual width and loss reduction suggests virtual width as a new, predictable dimension for scaling model efficiency.

    Source: §6 · p. 13

    Translating VWN’s algorithmic gains into production efficiency depends on systems realities.

    Source: §6 · p. 13

Abstract

from arXiv · show

We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.

1 Introduction

Virtual Width Networks (VWN) widen token embeddings while keeping Transformer backbone width fixed, aiming to increase representational capacity without the quadratic cost of naive width scaling. The introduction reports that 8× VWN reaches matched-baseline next-token and next-2-token losses with fewer tokens.

  • Naively increasing hidden dimensions grows parameters and compute quadratically, while MoE leaves the backbone hidden dimension fixed and its representational capacity bottlenecked.Directly widening hidden layers can close the performance gap but incurs quadratic increases in parameters and computation.Source: §1 · p. 2
  • Generalized Hyper-Connections (GHC) unify Hyper-Connections [36] and Frac-Connections [37], routing Over-Width Hidden States through backbone-width attention and feed-forward modules.GHC compresses the widened states before each module and expands module outputs back to the over-width representation.Source: Figure 2 · p. 2, §1 · p. 3
  • VWN paired with multi-token prediction (MTP) uses auxiliary n-gram supervision to exercise the expanded virtual space and yields consistent downstream accuracy improvements.The paper attributes the synergy to MTP's denser supervision and VWN's additional representational degrees of freedom for short-range compositional modeling.Source: §1 · p. 3
  • With 8× embedding width, VWN reaches baseline next-token loss with 2.5× fewer tokens and baseline next-2-token loss with 3.5× fewer tokens.The efficiency advantage increases as training progresses.Source: §1 · p. 3
  • VWN decouples embedding width from backbone width, enabling r× virtual widening with minimal additional compute and revealing a log-linear relation between virtual width and loss.The reported scaling experiments examine the loss and virtual-width factor under a fixed backbone.Source: §1 · pp. 2–3

2 Related Works

Related work scales model capacity through methods such as MoE, hidden-dimension connections, and expanded token embeddings. VWN decouples embedding width from backbone hidden size, while its GHC design combines expansion with structured sub-states.

  • MoE [9] [22] scales model size through conditional subnetwork activation, while VWN increases effective capacity by decoupling embedding width from backbone hidden size to avoid widening’s quadratic compute cost.Source: §2 · p. 3
  • HC [36] and AltUp [2] expand hidden dimensions through low-cost compositional links, but HC can underuse large expanded spaces when updates rely on few scalar weights.Source: §2 · p. 3
  • FC [37] partitions existing width into smaller segments, whereas GHC combines expansion with structured sub-states for finer capacity control and effective fractional expansion ratios.GHC’s expansion ratio need not be an integer multiple.Source: §2 · pp. 3–4
  • Vocabulary scaling [29] and hierarchical n-gram embeddings [17] can improve expressiveness and training efficiency with negligible overhead; Over-Tokenized Transformer uses Over-Encoding and Over-Decoding, with MTP [10] a practical OD instantiation.Source: §2 · p. 4

3 Method

VWN expands token embeddings while retaining the backbone hidden width, using generalized connections to integrate the wider representations and mechanisms to limit added compute and memory. Its multi-token prediction head also uses block-level mixing to keep costs comparable to the unexpanded case.

3.1 Rethinking the Model Width

  • VWN decouples embedding dimension from hidden width, expanding token representations while preserving nearly the original computational cost of intermediate-layer computation.The rationale is that embedding lookup is a minor fraction of total cost, while scaling Transformer width increases computational complexity quadratically.Source: §3.1 · p. 5

3.2 Over-Width Embedding

  • Over-Width Embedding expands input embeddings to D′ and projects final states back to D, applying normalization before reduction and GroupNorm for large expansions.At an 8× expansion, a 4096-dimensional hidden size yields a 32K-dimensional representation; GroupNorm uses groups of size D.Source: §3.2 · p. 5

3.3 Generalized Hyper-Connections

  • Generalized Hyper-Connections use light layer-wise transformations to relate backbone hidden-state segments to over-width embeddings, while DGHC conditions transformations on input representations.The implementation combines static and dynamic parameters, with dynamic parameters generated by a lightweight projection network [36] [37].Source: §3.3 · pp. 5–6

3.4 Multi-token Prediction

  • For multi-token prediction, VWN adds prediction layers and replaces dense hidden-state–embedding mixing with a shared block-level linear that keeps mixing cost comparable to r=1.Following DeepSeek-AI et al. [7], the head concatenates next-token and preceding-token embeddings; block-level mixing avoids the costly 2rD→rD dense projection.Source: §3.4 · p. 7

3.5 Cost Analysis

  • With m=2 and n=3, normalization, dynamic-parameter calculation, and width connection cost 42D FLOPs, while depth connection costs 6D FLOPs; these costs are described as minor.The operations are fused to reduce I/O overhead, and for small m the added cost is roughly n/m−1 times layer normalization and residual addition.Source: §3.5 · p. 8
  • With m=2, n=3, and η=0.5, VWN adds 3D bytes of activation memory per layer, approximately 8.8% of a vanilla layer’s footprint.Selective recomputation mitigates activation-storage overhead; inference memory overhead comes only from extra parameters and is described as negligible.Source: §3.5 · p. 8

4 A Connectivity Perspective

The section frames GHC-based VWN as learned, attention-like connectivity over depth, mediating how layers retain and reuse prior representations. This analogy concerns a depth-oriented KV cache, not standard self-attention.

Connectivity patterns

  • Plain stacks without residuals see one layer, residuals [11] see two, dense connectivity [16] [32] sees all prior layers, and GHC provides learned fixed-cost access between these patterns.The section likens these connectivity patterns to attention windows over a depth sequence, with layer indices as token positions and hidden states as a vertical KV cache.Source: §4 · p. 8

GHC mechanism

  • GHC linearly aggregates backbone-transformed features from earlier layers, using a learned carry operator to propagate slot information and writes to incorporate each layer’s backbone summary.This gives VWN a linear-attention-like mechanism over a compressed depth cache.Source: §4 · p. 8

Memory allocation

  • The memory budget r=n/m trades per-layer fidelity against depth-window length: m=1 retains fewer full-width layers, while m>1 stores more layers at D/m dimensions.For (m, n)=(8, 64), r=8 and the 64 slots each have width D/8.Source: §4 · pp. 8–9

Hard vs. soft depth windows

  • Hard routing yields a fixed depth window, while soft routing can retain attenuated contributions beyond the nominal window, extending the effective receptive field.The hard window is r layers for m=1 or n layers for m>1; soft routing can preserve earlier-layer contributions with decay.Source: §4 · p. 9

Scope of the analogy

  • The attention analogy is limited to a depth-oriented KV-cache perspective: GHC routes information with learned matrices, not similarity scores, pairwise correlations, or dot-product attention over layers.Routing may be static or input-conditioned, and carries, compresses, and writes information across layers at fixed cost.Source: §4 · p. 9

5 Experiments

Across MoE experiments, VWN improves training and downstream performance at both fractional and larger virtual widths. At 8× virtual widening, its optimization advantage grows during training, while loss reductions scale approximately log-linearly with virtual width.

5.1 VWN 1.5×

  • On 0.4B/4B MoE, VWN lowers NTP loss, while VWN+MTP gives augmented variants’ lowest loss (still 0.016 above baseline) and highest Collection A accuracy.MTP alone slightly increases NTP loss and is comparable to baseline on downstream evaluation.Source: §5.1 · pp. 9–10, Figure 4 · p. 10
  • On 2.5B/25B MoE, VWN and VWN+MTP each finish about 0.015 below baseline loss, and VWN+MTP yields the best average downstream accuracy across training.Adding MTP on top of VWN does not degrade optimization at this scale.Source: §5.1 · p. 10, Figure 5 · p. 10

5.2 Large Virtual Width

  • On MoE-A0.8B, increasing m improves convergence modestly at r=2, but partition-granularity benefits diminish beyond m≈4 at r=4 and r=8.The authors hypothesize that larger models may require higher m to maintain virtual capacity, while smaller models saturate at lower m.Source: §5.2 · p. 11, Figure 6 · p. 11

5.2.1 Scaling Law of the Virtual Width Factor.

  • The virtual-width factor’s loss reduction follows a log-linear relation: each doubling corresponds to an approximate 0.0069 loss reduction.The fitted relationship is y = −0.0069 · log2(x) + 1.6212, with R2 = 0.9986.Source: §5.2.1 · p. 12, Figure 8 · p. 12
  • At 500B tokens, VWN×8 reduces next-token loss by 0.035 and next-2-token loss by 0.058, and improves Collection B accuracy by +4.16 points over baseline.Gains rise monotonically from VWN×2 to VWN×8: next-token loss reductions are 0.020, 0.028, and 0.035; next-2-token reductions are 0.030, 0.045, and 0.058; accuracy gains are +3.2, +3.5, and +4.16 points, respectively.Source: Table 1 · p. 12, §5.2.1 · p. 12

5.2.2 VWN on Large Scale Model

  • At 3 T tokens, VWN’s next-token and next-2-token loss gaps reach ∆=0.032 and ∆=0.056, after matching baseline losses with 2.5× and 3.5× fewer tokens.The Collection B peak average accuracy is +2.16 points over baseline. The authors attribute the larger gain on the multi-token objective to richer over-width representations and GHC gradient transmission without increasing intermediate-layer width.Source: §5.2.2 · p. 12

6 Conclusion

Virtual Width Networks decouple representational width from the quadratic compute cost of widening. At 8× virtual width, optimization accelerates, and loss reduction scales approximately log-linearly with virtual width.

  • VWN decouples representational width from widening’s quadratic compute cost, integrates with standard Transformer stacks and training recipes, and shows consistent improvements at 1.5× expansion.Source: §6 · pp. 12–13
  • At 8× virtual width, next-token prediction loss converges more than 2× faster and multi-token prediction loss more than 3× faster than at baseline width.Source: §6 · pp. 12–13
  • Each doubling of virtual-width factor r corresponds to an average loss decrease of about 0.0069, indicating an approximately log-linear scaling relation.Source: §6 · p. 13
  • Communication and memory-access overheads constrain deployability; 1.5×–4× expansions are more feasible today, while larger ones may require software, memory-layout, and interconnect co-design.Source: §6 · p. 13

Appendix · A Detailed Downstream Results for MoE-A0.8B Models

On MoE-A0.8B downstream benchmarks, VWN×8 improves accuracy over the non-VWN baseline across diverse tasks and maintains better sample efficiency throughout training.

A Detailed Downstream Results for MoE-A0.8B Models

B Implementation of Generalized Hyper-Connections

The implementation applies Generalized Hyper-Connections to both attention and feed-forward blocks using width and depth connection steps. Width connections mix hidden representations and produce coefficients that the depth connections use to combine block outputs with the mixed representations.

B Implementation of Generalized Hyper-Connections

  • Width connections normalize reshaped hidden states, compute dynamic alpha and beta alongside static parameters, mix representations with alpha, and return beta.The dynamic coefficients are scaled and combined with static parameters before being split into alpha and beta.Source: B Implementation of Generalized… · p. 19
  • Depth connections combine the beta-weighted block output with the remaining mixed representations, then reshape the result into the hidden state.Source: B Implementation of Generalized… · p. 19
  • In both attention and FFN blocks, the transformer applies a width connection, normalizes and processes a reshaped slice, then uses a depth connection after dropout.The attention path uses self-attention; the FFN path uses the feed-forward network.Source: B Implementation of Generalized… · p. 20

C Downstream Benchmarks

The section presents downstream benchmarks from two collections, A and B.

Related papers

Loading 2511.11238v2…