PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding

StepFun, :, Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Ka Man Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxin Miao, Chang Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengyuan Yao, Daokuan Lv, Dapeng Shi, Deshan Sun, Ding Huang, Dingyuan Hu, Dongqing Pang, Enle Liu, Fajie Zhang, Fanqi Wan, Gulin Yan, Han Zhang, Han Zhou, Hanghao Wu, Hangyu Guo, Hanqi Chen, Hanshan Zhang, Hao Wu, Haocheng Zhang, Haolong Yan, Haoran Lv, Haoran Wei, Hebin Zhou, Heng Wang, Heng Wang, Hongxin Li, Hongyu Zhou, Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, Jialing Xie, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yan, Jie Yang, Jieyi Hou, Jinguang Zhang, Jinlan Cao, Jisheng Yin, Junfeng Liu, Junhao Huang, Junzhe Lin, Kaijun Tan, Kaixiang Li, Kang An, Kangheng Lin, Kenkun Liu, Lei Yang, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lin Zhang, Lina Chen, Liwen Huang, Liying Shi, Longlong Gu, Mei Chen, Mengqiang Ren, Ming Li, Mingzhe Chen, Na Wang, Nan Wu, Qi Han, Qian Zhao, Qiang Zhang, Qianni Liu, Qiaohui Chen, Qiling Wu, Qinglin He, Qinyuan Tan, Qiufeng Wang, Qiuping Wu, Qiuyan Liang, Quan Sun, Rui Li, Ruihang Miao, Ruosi Wan, Ruyan Guo, Shangwu Zhong, Shaoliang Pang, Shengjie Fan, Shijie Shang, Shilei Jiang, Shiliang Yang, Shiming Hao, Shuli Gao, Siming Huang, Siqi Liu, Tiancheng Cao, Tianhao Cheng, Tianhao Peng, Wang You, Wei Ji, Wen Sun, Wenjin Deng, Wenqing He, Wenzhen Zheng, Xi Chen, Xiangwen Kong, Xianzhen Luo, Xiaobo Yang, Xiaojia Liu, Xiaoxiao Ren, Xin Han, Xin Li, Xin Wu, Xu Zhao, Yanan Wei, Yang Li, Yangguang Li, Yangshijie Xu, Yanming Xu, Yaqiang Shi, Yeqing Shen, Yi Yang, Yifei Yang, Yifeng Gong, Yihan Chen, Yijing Yang, Yinmin Zhang, Yizhuang Zhou, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yuchu Luo, Yue Peng, Yufan Lu, Yuhang Deng, Yuhe Yin, Yujie Liu, Yukun Chen, Yuling Zhao, Yun Mou, Yunlong Li, Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, Yuyang Chen, Zejia Weng, Zhe Xie, Zheng Ge, Zheng Gong, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhirui Wang, Zidong Yang, Zili Wang, Ziqi Wang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Xiangyu Zhang

arXiv:2507.19427v1cs.LGcs.AI

TL;DR

LLM decoding is hardware-inefficient and particularly costly for long-context reasoning, motivating methods that reduce decoding cost without relying only on parameter counts. The paper presents Step-3, combining hardware-aware model-system co-design with MFA, MoE, and AFD. It reports lower decoding costs than representative models, with advantages widening at longer context, while identifying deployment and hardware constraints on sparsity and acceleration.

  • Problem

    LLM decoding has low hardware efficiency and high cost, especially when longer reasoning requires more generated tokens.

  • Method

    Step-3 combines hardware-aware model-system co-design with MFA attention, MoE FFNs, and AFD deployment that separates attention and FFN subsystems.

  • Results

    Step-3 achieves lower theoretical decoding costs than representative models, with its advantage widening at longer context.

  • Takeaways & Limitations

    Cost-effective decoding depends on aligning attention arithmetic intensity, MoE sparsity, and subsystem deployment with available hardware.

  • Takeaways & Limitations

    MTP can increase FFN costs regardless of prediction accuracy, and proposed over-sparsity workarounds may reduce efficiency or expressiveness; their performance impact remains insufficiently studied.

Abstract

from arXiv · show

Large language models (LLMs) face low hardware efficiency during decoding, especially for long-context reasoning tasks. This paper introduces Step-3, a 321B-parameter VLM with hardware-aware model-system co-design optimized for minimizing decoding costs. Step-3 innovates in two key dimensions: (1) A novel Multi-Matrix Factorization Attention (MFA) mechanism that significantly reduces both KV cache size and computation while maintaining high attention expressiveness, and (2) Attention-FFN Disaggregation (AFD), a distributed inference system that decouples attention and Feed-Forward Network (FFN) layers into specialized subsystems. This co-design achieves unprecedented cost efficiency: Step-3 significantly reduces theoretical decoding costs compared with models like DeepSeek-V3 and Qwen3 MoE 235B, with the gains widening at longer context. Step-3 achieves low cost while activating 38B parameters per token (more than DeepSeek-V3 and Qwen3 MoE 235B), demonstrating that hardware-aligned attention arithmetic intensity, MoE sparsity, and AFD are critical to cost-effectiveness. We perform a head-to-head comparison with DeepSeek-V3 in its favorable scenarios. Our implementation on Hopper GPUs achieves a decoding throughput of up to 4,039 tokens per second per GPU under 50ms TPOT SLA (4K context, FP8, no MTP). It is higher than DeepSeek-V3's 2,324 in the same setup and sets a new Pareto frontier for LLM decoding.

1 Introduction

Step-3 targets the high cost and low hardware efficiency of LLM decoding through model-system co-design. Its reported design improves the decoding-cost frontier by aligning attention, MoE, and deployment choices with hardware constraints.

  • Contribution: Step-3 contains 321 billion total parameters and activates 38B parameters per text token while targeting lower decoding costs.The authors emphasize that its activated count is slightly larger than representative open-weight models such as DeepSeek V3.
  • Motivation: Decoding is costly because its low MFU makes each token more expensive than training and prefill, while longer reasoning increases the value of cheaper decoding.The paper also links faster, cheaper decoding to faster reinforcement-learning training.
  • Contribution: Step-3 largely improves the Pareto frontier of activated parameters and theoretical decoding costs, with its advantage widening at longer context.The comparison searches deployment strategies using AFD and combinations of H800, H20, A800, and Ascend 910B hardware.
  • Results: ∼40% decoding cost reduction versus comparable models is reported despite Step-3 having the highest activation parameters among the compared models.The supplied result passage also reports lower theoretical cost than DeepSeek V3 despite hardware and parameter-count differences.
  • Design rationale: Attention design can matter more than parameter count because KV-cache size alone does not determine cost and excessive arithmetic intensity can burden weaker hardware.The paper identifies hardware-aware MoE sparsity as another requirement, since overly sparse models may run inefficiently despite fewer activated parameters.
  • System design: AFD separates attention and FFN deployment, enabling independent hardware and parallelism choices, dynamic attention scaling, high-MFU FFN batching, and overlapped communication.The listed system advantages also include balanced pipelining, reduced scale requirements versus DeepEP, and heterogeneous hardware support.

2 Step-3 Model Card

Step-3 is a Transformer-based VLM whose model design targets efficient decoding through MFA attention and a sparse FFN architecture.

  • Step-3 uses Transformer blocks containing attention and FFN modules.
  • MFA applies low-rank factorization in the QK circuit to scale attention heads while reducing KV cache overhead.
  • 316 billion parameters are present in the model, with 38 billion activated per token.
  • Further model-side details are planned for future release.

3 Attention-FFN Disaggregation

AFD separates attention and FFN execution across specialized GPU subsystems, enabling independent hardware and parallelism choices while requiring tightly coordinated communication and pipeline scheduling.

  • Rationale: Disaggregation addresses differing attention and FFN computational and memory-access patterns that can otherwise cause suboptimal GPU utilization.
  • Design goals: AFD enables independent parallelism strategies, hardware selection, and architectural optimization for attention and FFN subsystems.
  • AFD architecture: AFD deploys attention and FFN layers on separate GPU sets and transmits hidden states between them during decoding.
  • Pipeline design: Communication latency can be comparable to computation latency, so A/F processing and network communication must be jointly orchestrated to avoid stalls.
  • Pipeline design: The target is 50ms time per output token through a 3-stage A/F/communication pipeline with 16.6ms allocated to each stage.
  • Related work: Step-3’s AFD system is positioned as model-system co-design, unlike Megascale-Infer’s primarily system-level optimization.

4 Cost Analysis for LLM Decoding

The analysis evaluates decoding cost from computation, KV memory access, accelerator characteristics, and deployment choices. Step-3 has the lowest reported theoretical costs, with its advantage increasing at longer contexts.

  • Deployment caveat: Over-sparse models such as DSv3, Kimi K2, and Llama 4 Maverick may experience FFN costs doubling or tripling on H800 in real deployment.
  • Cost model: The theoretical attention cost is the larger of computation and KV-memory-access costs, plus linear projection computation.
  • Overall cost: 0.055 per 1M decoding tokens is Step-3’s cost at 8K context, versus 0.068 for DSv3 and 0.062 for Qwen3 MoE.
  • Parameter count: Total and activated parameter counts do not reliably indicate decoding cost: Qwen3 32B has fewer parameters than DSv3 and Step-3 but the highest cost in Figure 2.
  • Context scaling: Attention becomes significantly more expensive than FFN by 8K context, and the gap grows with longer context because FFN cost is context-independent.
  • Hardware effects: Step-3’s MFA is more hardware-friendly than DSv3’s MLA and Qwen3’s GQA, showing smaller cost differences across weaker hardware.
  • Design implications: Training and decoding costs differ: training tracks activated parameters more closely, whereas low decoding cost requires model-system co-design.

5 Model-System Co-design

Step-3 co-designs MFA and AFD around hardware-aware decoding costs. The design balances attention arithmetic intensity, KV access, MoE sparsity, and deployment strategies to reduce cost while preserving attention expressiveness.

  • Attention Design: MFA’s attention cost advantage comes from balancing arithmetic intensity with hardware rooflines, not merely minimizing KV cache size.Step-3’s MFA arithmetic intensity is 128 with 8-bit KV, versus 512 for DSv3’s MLA; matching hardware computation-to-bandwidth ratios improves efficiency.
  • Attention Design: One-fourth the computation of DSv3 and one-third the memory access of Qwen3 allow MFA to maintain low costs across hardware.These reductions hold as context length grows from 8K to 32K, while MFA retains effective rank 16,384, matching DSv3’s MLA and exceeding Qwen3’s 8,192.
  • Quantization and MTP: Further KV quantization and MTP may provide additional savings for MFA, but MTP can increase FFN costs regardless of prediction accuracy.Quantization and MTP raise arithmetic intensity; the paper therefore cautions that MTP requires careful enablement under AFD.
  • MoE Design: Hardware-aware MoE sparsity is necessary because optimal sparsity varies substantially across platforms and excessive sparsity can worsen network bottlenecks.H20 tolerates the sparsest configurations, whereas H800 is least friendly to very sparse MoE; Step-3 chooses sparsity around 0.08 including the shared expert.
  • System Co-design: AFD separates attention and FFN cost analysis, enabling each component to use a deployment and parallelism strategy suited to its hardware characteristics.FFN may use TP-only, EP-only, or hybrid TP+EP deployment under AFD.
  • MoE Design: Step-3 avoids over-sparsity through small TP, EP, or hybrid TP+EP deployments, reducing expert-imbalance impact without routing restrictions.By contrast, workarounds for over-sparsity can reduce efficiency through expert imbalance or adversely affect model expressiveness.

6 Non-Flagship Hardware Support

AFD enables attention and FFN to scale independently across hardware, making non-flagship accelerators viable when latency and pipeline constraints are satisfied. Step-3’s hardware support depends especially on total FFN parameters, bandwidth, and deployment scale.

  • AFD Architecture: AFD independently scales attention and FFN, allowing non-flagship hardware to be assigned to either subsystem or both.The architecture separates GPU sets and communicates hidden states between attention and FFN stages through a pipelined network.
  • Latency Constraints: A three-stage AFD pipeline requires attention and FFN computations to complete within approximately 272µs per layer to meet the SLA.Both attention and FFN servers must satisfy the latency budget; weaker hardware can therefore become unusable despite theoretical cost advantages.
  • Attention Support: Step-3’s MFA has the smallest KV volume and moderate arithmetic intensity, making it relatively hardware-friendly on weaker accelerators.The attention workload can remain memory-bandwidth bound while using alternative hardware configurations.
  • FFN Support: For Step-3’s roughly 300 GB of FFN weights, six L20 servers, or 48 cards, are estimated to meet performance requirements in EP.The estimate assumes each L20 uses a batch size that places FFN in the compute-bound region and accommodates 7.1 GB per layer aggregate.
  • Deployment Boundaries: Weaker hardware must still meet every AFD stage’s latency requirement, and excessive scale can introduce expert imbalance and stability concerns.A weaker L4 configuration would require 144 cards for the FFN latency target, while large deployments raise additional operational concerns.
  • Hardware Selection: Total FFN parameter count is the primary factor affecting weaker-hardware friendliness, and Step-3 is reported to balance this at the L20 level.Using at least L20-class cards is recommended for models with hundreds of billions of parameters because stronger cards reduce FFN server count and aid reliability and load balancing.

7 Implementation and Results

Step-3 implements Attention-FFN Disaggregation with staged communication and heterogeneous-accelerator support, then evaluates throughput, quantization, and attention designs. The system reaches higher decoding throughput than DeepSeek-V3 under the reported comparison settings, while MFA shows the lowest attention latency among tested designs.

  • AFD implementation: AFD separates attention instances, which manage KV caches and routers, from FFN instances, which perform pure MoE computation.This specialization enables independent handling of attention and FFN workloads.
  • Pipeline and communication: A multi-stage pipeline overlaps attention, network transfer, and FFN computation to hide communication overhead while maintaining throughput and latency.Samples are processed sequentially, transmitted to FFN instances, and orchestrated so stage times remain near-balanced.
  • Pipeline and communication: AFD must transmit FP8 tokens, scales, expert distribution, and BF16 activations between attention and FFN instances within 272 µs for a three-stage pipeline.Existing communication libraries can struggle with this requirement because communication consumes GPU SM resources.
  • End-to-end performance: 3,910 TGS long-term average and 4,039 TGS peak were achieved at 4,096 average context under a 20 tokens/s SLA, about 74% higher than DeepSeek-V3.The comparison uses FP8 GEMM and reports 4,039 tokens/GPU/s in a peak minute.
  • End-to-end performance: 18% lower throughput with 3A2F attention than FP8 attention still left Step-3 substantially ahead of DeepSeek-V3.The 3A2F result was 3,321 TGS, using a total batch size of 6048.
  • Attention ablation: MFA-Step3 has the lowest attention latency among MFA, MLA-DSv3, and GQA-Qwen3, with the performance gap widening on H20 and A800.The ablation treats each attention-layer result as the performance of an attention instance in AFD deployment.

8 Conclusion and Future Work

The paper concludes that Step-3’s model-system co-design achieves state-of-the-art decoding efficiency among similarly sized LLMs and identifies interconnect limits as a constraint on further MoE sparsity.

  • Conclusion and future work: Step-3 achieves state-of-the-art decoding efficiency among LLMs of similar sizes through model-system co-design.Future work includes enabling MTP, exploring new attention variants, and improving interconnects for greater FFN sparsity.

Core System Contributors

The listed core system contributors are Bin Wang and Bojun Wang.

  • Core system contributors: Core system contributors listed here are Bin Wang and Bojun Wang.

Core Model Architecture Contributors

The listed core model architecture contributors are Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, and Ka Man Lo.

  • Core model architecture contributors: Core model architecture contributors listed here are Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, and Ka Man Lo.

Contributors (Pretrain, Post-train, Multi-modal, System, Data)

This section lists contributors associated with the paper across the supplied contributor passages.

  • The contributor list includes Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, and Changxin Miao.
  • Additional contributors listed include Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, and Jialing Xie.
  • The list continues with Longlong Gu, Mei Chen, Mengqiang Ren, Ming Li, and Mingzhe Chen.
  • Further listed contributors include Wang You Wei, Ji Wen Sun, Wenjin Deng, Wenqing He, and Wenzhen Zheng.
  • The supplied passages also list Yuhang Deng, Yuhe Yin, Yujie Liu, Yukun Chen, and Yuling Zhao.
  • Additional names include Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, and Yuyang Chen.

Sponsors

This section lists the individuals named in the supplied sponsor passages.

  • The sponsor passages name Heung-Yeung Shum and Xiangyu Zhang.
  • They also list Zheng Ge, Zheng Gong, Zhenyi Lu, Zhewei Huang, and Zhichao Chang.
  • Additional named individuals are Zhiguo Huang, Zhirui Wang, Zidong Yang, Zili Wang, and Ziqi Wang.
  • The final supplied sponsor passage also names Zixin Zhang.
Loading 2507.19427v1…