PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

DeepSeek-AI, :, Anyi Xu, B. Li, Bangcai Lin, Bing Xue, BingCheng Xian, Bingzheng Xu, Bochao Wu, Bowei Zhang, Boyi Deng, C. C. Yu, Chao Jin, Chaofan Lin, Chen Dong, Chenbing Wang, Chenfan Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyuan Zhang, Chenhao Xu, Chenqi Zhao, Chenze Shao, Chuhao Wang, Chuqi Zhang, Damai Dai, Dejian Yang, Deli Chen, Di Huang, Di Wu, Donghao Li, Erhang Li, Eric Fu, F. Zhou, Fangwei Zhou, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanglin Li, Guanting Chen, Guoai Cao, Guofan Fan, Guolai Meng, Guowei Li, Haichuan Zhang, Haiyang Ma, Haiyang Shen, Han Li, Han Yu, Han Zhang, Hangyuan Deng, Hanwei Xu, Hanxiang Xu, Hanxun Zhong, Hao Guo, Hao Jiang, Hao Li, Hao Qin, Haodong Wen, Haofen Liang, Haofeng Huang, Haohua Liu, Haoling Zhang, Haoming Luo, Haoran Yang, Haotian Xu, Haotian Yuan, Haoting Huang, Haowen Luo, Haoyang Cai, Haoyu Chen, Haozhe Ji, Hengran Zhang, Hengrui Wang, Hengxu Wu, Honghui Ding, Hongxuan Tang, Huadong Wang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, J. Yang, J. H. Jin, J. H. Zhang, J. X. Zou, Jia Yu, Jiahui Zhou, Jiajun Chen, Jialiang Huang, Jialin Zhao, Jiamin Tang, Jian Zhou, Jianan Tong, Jianwen Li, Jiaqi Zhu, Jiarui Wang, Jiasheng Ye, Jiashi Li, Jiaxin Xu, Jiaying Ding, Jibai Lu, Jiewen Hu, Jin Yan, Jincheng Zhai, Jingchang Chen, Jingcheng Hu, Jingli Zhou, Jingsheng Xu, Jingting Xiang, Jingyan Yun, Jingyang Yuan, Jingyuan Cheng, Jinhua Zhu, Jinpeng Wang, Jinyi Chen, Jinyi Hu, Jiping Yu, Jueliang Guo, Junbo Pei, Junbo Sun, Junguang Jiang, Junjie Qiu, Junkang Zhou, Junqi Liu, Junren Li, Junxian Li, Junxiao Song, Junyi Guo, Kai Dong, Kaifeng Chen, Kaige Gao, Kang Guan, Kangdong Yuan, Ke Hong, Ke Xu, Kefan Zhao, Kexin Ji, Kexin Zhang, Kexing Zhou, Kuai Yu, Lan Zhang, Lean Wang, Lecong Zhang, Lei Wang, Letian Gao, Liang Zhao, Liansheng Xu, Lihua Guo, Lingxiao Luo, Lingyue Fu, Litao Deng, Litong Wang, Liyue Zhang, Longhao Chen, Lu Chen, Luotian Huang, Luyao Ma, Luyao Wang, M. S. Di, Max Mei, Menghao Ye, Miao Cui, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingjing Zhang, Mingqi Wei, Mingshu Chen, Mingxing Liu, Mingxu Zhou, Mingyu Xu, Mingyu Yang, Mingze Wang, Muyang Chen, Ni Shentu, Ning Wang, Niufang Ning, Panpan Huang, Peixin Cong, Peiyi Wang, Peiyuan Xin, Pengfei Ren, Pengfei Yan, Pengle Zhang, Qi Kang, Qi Tang, Qiancheng Wang, Qiang Li, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Qizhou Guo, Rongxian Xu, Rui Ding, Rui Hu, Rui Tian, Rui Yu, Ruidong Zhu, Ruifan Xu, Ruihan Yang, Ruihang Xia, Ruijie Lu, Ruilin Geng, Ruipeng Hong, Ruiqi Ge, Ruisong Zhang, Ruize Sun, Ruizhe Pan, Runji Wang, Runqian Chen, Runxin Xu, Ruohong Tian, Ruomeng Shen, Ruoyu Zhang, Ryan X., S. H. Liu, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoyuan Chen, Shengding Hu, Shengkai Lin, Shengwen Ran, Shengyu Liu, Shengyuan Jia, Shi Bai, Shi Feng, Shicheng Xu, Shichun Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shiyuan Feng, Shufan Gong, Shuhan Lin, Shuiping Yu, Shunfeng Zhou, Shuo Yang, Shuomeng Wang, Shuting Guo, Shuting Pan, Shuying Yu, Sinuo Cao, Siyi Lin, Sizhe Chen, Songyang Chen, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye, Tianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tongrui Xiong, Wangding Zeng, Wei Liu, Wei Zhang, Weibin Xu, Weihao Zeng, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun Gao, Wenkai Shao, Wenkai Yang, Wenli Zhang, Wenlu Wang, Wenlve Huang, Wenqian Yan, Wentao Zhang, Xi Gao, Xiang He, Xiang Li, Xiangli Li, Xiangwen Wang, Xiangying Zhang, Xiankui Wei, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojian Qu, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xiaoyao Zou, Xiaoyuan Li, Xicheng Guo, Xieting Chu, Xin Cheng, Xin Liu, Xin Xie, Xinbo Xu, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xintong Yao, Xinyang Chen, Xinyong Jiang, Xinyu Yang, Xinyu Yang, Xu Chen, Xuanyu Wang, Xubei Zhong, Xuecheng Su, Xuejie Liu, Xuheng Lin, Xujie Fan, Xuncheng Zhao, Xuwei Fu, Y. C. Yan, Y. H. Jiang, Y. T. Wu, Y. W. M., Y. Z. Wang, Yafei Gao, Yang Yang, Yang Zhang, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Meng, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yaoyang Ye, Yehang Yin, Yexinrui Wu, Yi Qian, Yi Tao, Yi Yu, Yichao Zhang, Yichen Jiang, Yicheng Wang, Yifan Ding, Yifan Shi, Yifeng Peng, Yifeng Zhai, Yijia Wu, Yiliang Xiong, Yilun Wang, Ying He, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yiping Wang, Yisong Wang, Yixiang Zhang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyao Yang, Yiyuan Liu, Yizai Cai, Yizhen Wei, Yizhi Wang, Yonglun Yang, Yongqi Zhuo, Yongqiang Guo, Yongtong Wu, Yu Wu, Yu Zhang, Yuan Bian, Yuan Cheng, Yuan Ou, Yuan Sun, Yuanfan Xu, Yuanhang Sun, Yuanhao Li, Yuchen Liu, Yuchen Yao, Yudong Han, Yuduan Wang, Yuhan Wu, Yuhao Meng, Yuheng Zou, YuKun Li, Yunchuan Wang, Yunfan Xiao, Yunfan Xiong, Yupeng Chen, Yuqian Cao, Yuqian Wang, Yuqing Chen, Yushun Zhang, Yutong Lin, Yuwei Xiao, Yuxian Gu, Yuxiang Chen, Yuxiang Huang, Yuxiang Luo, Yuxiang You, Yuxin Chen, Yuxin Xiang, Yuxuan Liu, Yuxuan Zhou, Yuyang Zhou, Yuzhe Guo, Yuzhen Huang, Yuzhuo Bai, Z. Y. Z., Zanlin Ni, Zehao Wang, Zehua Zhao, Zehui Ren, Zejun Zhao, Zhangli Sha, Zhanying Wang, Zhaochen Zhang, Zhaoshuai Du, Zhe Fu, Zhean Xu, Zhenda Xie, Zheng Liu, Zhengyan Zhang, Zhenhua Dong, Zhewen Hao, Zhibang Wang, Zhibin Gou, Zhicheng Ma, Zhihao Li, Zhihong Shao, Zhihuan Huang, Zhijie Li, Zhirui Lu, Zhixian Huang, Zhixuan Chen, Zhixuan Chen, Zhixuan Pan, Zhiyu Wu, Zhizhou Ren, Zhu He, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zili Zhang, Zilin Li, Zilong Hou, Zilong Lyu, Ziqiao Wang, Ziwei Xie, Ziya Zhang, Ziyi Gao, Zizheng Pan, Zonglin Li, Zongqing Yao, Zui Chen, Zuofan Wu, Chenchen Ling, Chengyu Hou, Chong Chen, D. Li, Di Qi, Dongjie Ji, Fang Wei, Fanyi Xia, Fei Xie, Feiyi Tan, Hailong Guo, Haiyan Zhai, Hui Zhou, Huihui Tan, Huijie Li, Jia Luo, Jia Song, Jialu Cai, Jian Liang, Jiangting Zhou, Jiaqi Gao, Jiayi Shao, Jie Chen, Jieyu Yang, Jin Chen, Jingde Zhang, Jingzi Zhou, Jinqian Wang, Jinyang Liu, JinZhao Sun, Junhua Ling, Junmin Zheng, Kaicheng Yang, Ke Xu, Le Su, Leyi Xia, Liangfeng Ding, Lin Zhuo, Linwang Ma, Linyan Zhu, Liyu Cai, Luqi Yao, M. K. Zhang, Meng Li, Miao Lin, Miaojun Wang, Min Zhang, Mingming Li, Mingming Wang, Mingze Yin, Minmin Han, Nan Cao, Ning Wang, Ningxin Ma, Panpan Wang, Peihan Lin, Peng Sun, Peng Zhang, Qian Ying, Qiang Xiang, Qiao Wang, Qingmiao Mao, Qiwei Jiang, Rongli Jin, Ruyi Chen, Sha Tao, Shangmian Sun, Shaoqing Wu, Shichao Zou, Si Lei, Tianyang Zhang, Tianyu Sun, Tingting Yin, W. L. Xiao, Wei An, Wei Li, Wei Wang, Weiwei Lin, Wenqing Hou, X. Lin, Xiangfei Meng, Xianzhu Huang, Xiao Peng, Xiaoqian Li, Xiaoting Zhang, Xiaowen Sun, Xiaoxiang Wang, Xiaoyu Ye, Xinrou Zhang, Xinyu Zhang, Xue Cao, Xueyin Chen, Yanan Zhou, Yanhong Xu, Yao Xia, Yao Xu, Yi Shao, Yihong Zhang, Yiling Ma, Ying Tang, Yining Lou, Yiru Chen, Yishi Piao, Yixuan Chen, Yong Xiong, Yuchen Xuan, Yuehan Yang, Yuer Xu, Yukun Zha, Yunxian Ma, Yuping Lin, Yuting Yan, Yutong Xie, Yuwen Sheng, Yuxuan Zhu, Zekai Zhang, Zhe Ju, Zhenzhen Lin, Zheren Gao, Zheyang Sun, Zhigang Yan, Zhongyu Wu, Zi Wang, Zihua Qu, Ziling Yan, Ziyi Wan

arXiv:2609.19969v1cs.CL

TL;DR

Long-context serving still faces computationally expensive prefill and KV caches that strain HBM, SSD capacity, and data-transfer bandwidth. The model jointly optimizes architecture, cache precision, and deployment through CED, CSA2 cross-layer KV reuse with FP4 caching, and SWA Bounded Replay. At equal sequence lengths, global KV cache is 890 bytes per token, roughly 1/4 of DeepSeek-V4-Flash’s; persistent KV cache footprint is roughly 1/8 of its.

  • Problem

    Long-context serving still faces computationally expensive prefill and KV caches that strain HBM, SSD capacity, and data-transfer bandwidth.

    Source: §1 · p. 4

  • Method

    The model jointly optimizes architecture, cache precision, and deployment through CED, CSA2 cross-layer KV reuse with FP4 caching, and SWA Bounded Replay.

    Source: §1 · pp. 4–5, §6 · p. 37

  • Results

    At equal sequence lengths, global KV cache is 890 bytes per token, roughly 1/4 of DeepSeek-V4-Flash’s; persistent KV cache footprint is roughly 1/8 of its.

    Source: §6 · p. 37

  • Takeaways & Limitations

    CED activates 8B parameters per token during prefill, compared with 16B during decode, improving cost efficiency for input-heavy agentic workloads.

    Source: §6 · p. 37

    Potential CSA2 selection errors and approximate SWA Bounded Replay reconstruction may degrade capabilities in untested boundary cases.

    Source: §6 · p. 37

Abstract

from arXiv · show

The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads. To push the limits of KV cache compression, DeepSeek-V4.1-Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching. These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline. In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions. We pretrain DeepSeek-V4.1-Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash.

1. Introduction

As long-horizon agents make ultra-long-context workloads more common, KV-cache storage and transfer join computation as deployment bottlenecks. DeepSeek-V4.1-Flash addresses them with a multimodal CED model and joint architectural, cache-precision, and deployment optimizations.

1. Introduction

  • The model has 552B backbone parameters, supports up to one million tokens, and activates 8B parameters per token during prefill and 16B during decode.Its Causal Encoder-Decoder architecture projects decoder global KV from the final encoder hidden states, targeting input-heavy agentic workloads.Source: §1 · p. 4
  • DeepSeek-V4.1-Flash delivers better overall performance than DeepSeek-V4-Flash, matches closed-source frontier models on most benchmarks, and completes over 95% of real-world tasks.It is on par with frontier models on standard agentic benchmarks, but gaps remain on science-oriented agentic tasks and against giant closed-source systems overall.Source: §1 · pp. 4, 6
  • CSA2 reuses global KV and indexer K across layers, while FP4 global KV caching brings runtime cache storage to approximately one-quarter of DeepSeek-V4-Flash.CSA2 uses Full, Reindex, and Reuse modes; every layer retains its own global Q and SWA KV. DeepSeek-V4.1-Flash uses pure CSA2 rather than DeepSeek-V4’s CSA–HCA hybrid.Source: §1 · pp. 4–5
  • SWA Bounded Replay reconstructs required SWA KV by replaying only the most recent n_win tokens, reducing persistent KV cache footprint to approximately one-eighth of DeepSeek-V4-Flash.It incurs negligible performance degradation and avoids persisting SWA KV to SSD at the cost of a small amount of prefill recomputation.Source: §1 · p. 5
  • Across a 256-fold context increase from 4K to 1M, DeepSeek-V4.1-Flash’s single-token Decode FLOPs increase by only 1/4.The architecture also adds Single-Pass mHC, whose Mega-mHC kernel halves activation-memory traffic versus the original four-kernel implementation, Engram conditional memory, and DSpark speculative decoding.Source: §1 · p. 5
  • DeepSeek-V4.1-Flash-Base improves held-out evaluations by 5%–10% over DeepSeek-V4-Pro-Base while using 1/3 the total parameters and 1/4 the activated parameters.It is pretrained on 45T multimodal tokens, with sparse attention trained from scratch at 64K sequence length and no dense-attention warmup.Source: §1 · p. 6
  • Post-training follows standard SFT, RL, and on-policy distillation without algorithmic changes, while scaling automated data synthesis, environment construction, tasks, and rollouts.The resulting evaluation profile includes strong reasoning and coding, visual reasoning that surpasses top-tier open-source competitors such as Kimi-K3, and practical visual-agent workflows.Source: §1 · p. 6

2. Architecture · 2.1. Overview · 2.2. Causal Encoder-Decoder (CED)

DeepSeek-V4.1-Flash is a multimodal MoE Transformer whose CED architecture reduces long-context prefill computation by sourcing decoder global KV from encoder outputs while retaining layer-wise SWA. Its vision and language pathways, modality-specific routing, and additional architectural components support multimodal processing and efficient inference.

2.1. Overview

  • The language backbone has 40 causal Transformer layers split into a 20-layer encoder and 20-layer decoder, with global and sliding-window attention except in the first two layers.The model has 552B backbone parameters and 196B Engram parameters, activating 8B per token during prefill and 16B during decode.Source: §2.1 · p. 7
  • CSA2 shares global KV across layers to reduce cache storage and reuses sparse selections to reduce indexing work, while a Hierarchical Sparse Indexer narrows later indexers’ candidate pools.The model combines global attention with layer-local sliding-window attention.Source: §2.1 · p. 8
  • The architecture retains shared and fine-grained routed experts, adds Single-Pass mHC and Engram, omits MTP during backbone pre-training, trains DSpark separately, and compresses the main KV cache to FP4.DSpark is used for speculative decoding; Single-Pass mHC revises residual-stream mixing for more efficient kernel fusion, and Engram adds sparsely accessed conditional memory.Source: §2.1 · p. 8

2.1.1. Multimodal Architecture

  • DeepSeek-ViT supports varying image resolutions using 2D-RoPE and reduces visual token count ninefold with 3 × 3 pixel-unshuffle before projecting features into the language backbone.The resulting visual embeddings are inserted at image-token positions and processed jointly with text embeddings.Source: §2.1.1 · p. 8
  • Separate expert-wise correction biases for image and text tokens balance expert utilization within each modality while preserving original routing scores to weight selected outputs.The two bias sets are updated independently according to their respective expert loads after each training step.Source: §2.1.1 · p. 8

2.2. Causal Encoder-Decoder (CED)

  • For global attention, decoder KV entries are projected from the L/2-th layer’s hidden state using layer-dependent weights, so prefill computes only the first half of the layers.This acquires the upper-layer global KV cache with minimal computational cost.Source: §2.2 · p. 9
  • For SWA, CED retains layer-wise KV computation but bounds decoder replay to the prompt’s last nwin tokens, reducing the overhead of replaying nwin × L/2 tokens.This approach is motivated by prior work showing SWA’s effective receptive field is much smaller than its theoretical window.Source: §2.2 · p. 9
  • For sequence lengths N≫nwin, CED reduces prefill complexity from O(NL) to O(NL/2 + nwin × L/2) ≈ O(NL/2), effectively halving overall computation.The paper also reports that CED maintains performance comparable to the baseline.Source: §2.2 · p. 9

2.3. Compressed Sparse Attention 2 (CSA2)

CSA2 jointly reuses main KV, indexer K, and Top-K indices across layers, combining cross-layer cache and index reuse with a simplified compressor and a Hierarchical Sparse Indexer. In the CED decoder, hierarchical candidate restriction reduces repeated scoring by later indexers while preserving a full-range pass in the first Full Mode layer.

2.3. Compressed Sparse Attention 2 (CSA2)

  • CSA2 jointly shares main KV and indexer K across layers and reuses Top-K indices, decoupling cache sharing from index reuse.It simplifies compression by removing overlap and absolute positional embeddings, and derives indexer K by projecting main KV entries rather than using a separate hidden-state compression path.Source: §2.3 · p. 10

2.3.1. Cross-Layer KV and Index Reuse

  • Full Mode computes main KV, indexer K, and fresh Top-K indices; Reindex Mode shares KV and K but selects indices anew, while Reuse Mode reuses the latest selection.All modes compute their own query and sliding-window KV; with CED, the decoder Full Mode layer obtains global KV from the final causal-encoder layer.Source: §2.3.1 · pp. 10–11

2.3.2. Hierarchical Sparse Indexer

  • For a fixed candidate-pool size, later decoder indexers score a number of positions bounded independently of context length, while the first Full Mode layer still scans the full causally visible range.The first Full Mode layer selects high-scoring blocks using each block’s maximum index score and gathers their positions into a shared candidate pool; Reindex layers select their own Top-K entries within it, while Reuse layers use the latest selection. The same candidate restriction is applied during training and inference.Source: §2.3.2 · pp. 11–12

2.4. Efficient Architectural Extensions

Section 2.4 introduces extensions to reduce activation traffic and KV-cache storage while adding conditional memory and speculative decoding. The designs combine Single-Pass mHC and Mega-mHC, Engram, DSpark, and FP4 main-KV caching.

2.4.1. Single-Pass mHC

  • Single-Pass mHC shifts input-mixing coefficients by one block, enabling Mega-mHC to fuse residual updates, input mixing, and coefficient prediction in one kernel with negligible performance degradation.Mega-mHC reaches the ideal (n+1)d reads and (n+1)d writes, halving activation memory traffic versus the original implementation.Source: §2.4.1 · pp. 12–13

2.4.2. Engram

  • Engram (Cheng et al., 2026c) adds conditional memory, retaining tokenizer compression, multi-head hashing, context-aware gating, and multi-branch integration while omitting short causal convolution and optimizing embeddings with momentum updates and Sinkhorn balancing.Its 196B parameters are split across two modules; deterministic addressing enables host-memory prefetching via background RDMA transfers.Source: §2.4.2 · p. 13

2.4.3. DSpark

  • DSpark (Cheng et al., 2026a) combines semi-autoregressive drafting with confidence-scheduled verification, dynamically selecting verification length from acceptance estimates and profiled throughput to target token throughput under system load.It is trained in a dedicated post-pretraining stage with the backbone frozen, then alongside the backbone without propagating DSpark-objective gradients into it.Source: §2.4.3 · pp. 13–14

2.4.4. FP4 Main KV Cache

  • FP4 main-KV caching with post-training QAT nearly halves storage versus DeepSeek-V4’s FP8 main-KV cache, both in HBM and when offloaded to SSD.The cache uses E2M1 with one E4M3 scale per 16 channels and omits the global scale; this causes no measurable accuracy decrease, while the SWA KV cache remains FP8.Source: §2.4.4 · p. 14

2.5. Optimization

The section adapts optimization to the architecture with head-wise Muon and a memory-saving Sinkhorn-balanced momentum update for large embedding-like matrices. It also specifies optimizer assignments and the vision-encoder training schedule.

2.5. Optimization

  • Head-wise Muon uses different preconditioners across attention heads and empirically outperforms vanilla Muon.The design is motivated by heterogeneity across heads.Source: §2.5 · pp. 14–15
  • For Engram embeddings, token embeddings, and the prediction head, a momentum update with Sinkhorn balancing reduces optimizer-state memory versus Adam and empirically outperforms it.Like Muon, the update requires only a momentum buffer.Source: §2.5 · p. 15
  • AdamW handles normalization-layer weights and other non-matrix parameters, while Muon handles backbone, Engram-projection, and vision-language-projector matrices; Query and Key weights use head-wise Muon.Muon uses decoupled weight decay and Nesterov momentum; the Sinkhorn-balanced update uses Nesterov momentum without weight decay. The vision encoder is frozen until learning-rate decay, then unfrozen with a smaller learning rate, while its final normalization layer and projector remain trainable.Source: §2.5 · p. 15
  • Sinkhorn balancing replaces Newton–Schulz orthogonalization in the Muon-like workflow, approximately equalizes row- and column-wise RMS, masks near-zero rows, and uses a learning-rate correction factor of 0.18.The rows correspond to token indices or n-gram identities, and columns to hidden features.Source: §2.5 · p. 15

3. General Infrastructures

Section 3 presents distributed training and inference infrastructure for multimodal workloads and CSA2, including persistent-cache management and bounded replay. The designs coordinate communication and computation while reducing cache storage and inference work.

3.1.1. Multimodal Training Infrastructure

  • Multimodal training overlaps contrastive feature all-gathers with text computation, separates encoder and LLM phases, and load-balances image loading across context-parallel ranks.Images are loaded once, while rollout image transfers are incremental and decoding/preprocessing outputs are cached on a distributed file system for reuse.Source: §3.1.1 · pp. 16–17

3.1.2. Attention Sharing Training for CSA2

  • CSA2 training uses shadow indexers, pipeline payload extensions, and micro-batch shared-state management to support cross-stage attention sharing without special pipeline scheduling.Synchronization and gradient aggregation keep shadow replicas consistent; shared states are retained through their final consumer, and workflow adaptations integrate CSA2 with existing distributed training.Source: §3.1.2 · pp. 17–18

3.1.3. Engram

  • Engram shards embedding tables by row, overlaps prefetch and gradient transfers with vision computation, and stores and fetches embeddings in FP8.Sinkhorn scaling avoids repeated writes of the full normalized matrix; keeping tables resident in GPU memory during RL rollouts reduces host-memory pressure and fragmentation-related out-of-memory failures.Source: §3.1.3 · p. 18

3.2. Inference System

  • In CSA2 Reuse Mode, transformer layers use 15 kernels during prefill and 11 during decode; EPD disaggregation lets vision encoding, prefill, and decoding scale independently and overlap.Kernel fusion keeps hardware resources pipelined inside a small number of fused kernels.Source: §3.2 · pp. 18–19

3.2.1. Persistent KV Cache Management

  • 1/8: Under identical workloads, V4.1 reduces persistent KV cache footprint to 1/8 of V4’s by removing SWA KV from persistent storage and compressing global KV to 1/4 of V4’s footprint.SWA KV instead resides in a distributed memory pool with a minutes-long TTL; bounded replay recovers it on misses by recomputing only n_win tokens rather than a full L × n_win-token pass.Source: §3.2.1 · p. 19

3.2.2. SWA Bounded Replay

  • Bounded replay regenerates encoder SWA KV from the last n_win cached-prefix tokens and decoder SWA KV from the last n_win prompt tokens, with negligible response-quality impact.Encoder replay reuses cached global KV; decoder replay uses its reconstructed SWA KV only for decoding, and the same replay is simulated during post-training for train-aware adaptation.Source: §3.2.2 · p. 20

4. Pre-Training

The section describes text and multimodal data curation, model and training configurations, and evaluations of DeepSeek-V4.1-Flash-Base against predecessor models. It reports broad benchmark performance and lower BPB on held-out internal tasks.

4.1. Data Construction

  • Text curation uses a scaling ladder, filters low-information model-generated and machine-translated content, and adds expert quality dimensions and recent code sources.The authors also explore model-in-the-loop data iteration as a foundation for future large-scale synthetic data.Source: §4.1 · pp. 20–21
  • Multimodal curation prioritizes cleaned native web data, staged image-aware filtering, domain-specific visual and OCR datasets, and image-code pairs and computer-use trajectories.The dataset includes image-text pairs, interleaved image-text data, and domain-specific data; web crawling was re-bootstrapped from Common Crawl to improve multimodal coverage.Source: §4.1 · p. 21
  • The final corpus merges text-only and multimodal sources at a 7:1 token ratio, pre-splits ultra-long documents, minimizes overlap, and limits packing padding to 10^-4.For overlapping samples, multimodal versions replace text-only versions and use the larger epoch count.Source: §4.1 · p. 21

4.2.1. Model Setups

  • The 40-layer, 5120-hidden-dimension CED model uses CSA2 across encoder and decoder layers, MoE blocks, and a 32-layer vision encoder; it has 552B backbone parameters.It activates 8B parameters per token during prefill and 16B during decode; encoder and decoder each have 20 layers.Source: §4.2.1 · pp. 21–22

4.2.2. Training Setups

  • Training spans 45T multimodal tokens, begins with 64K sequences, and extends to 1M tokens at 34T; the batch size stays at 100.6 million tokens.The learning rate warms up for 2,000 steps, remains at 2.6 × 10^-4 until 28T tokens, then decays to 2.6 × 10^-5 by 40T and stays there through 45T.Source: §4.2.2 · p. 22
  • The vision encoder is contrastively pretrained on about 47B image-text pairs at 224 × 224 resolution, then fine-tuned autoregressively on 236B tokens at 544 × 544 to 1344 × 1344.The autoregressive stage connects the encoder to a 4B MoE LLM; that LLM is discarded afterward, retaining the optimized vision encoder.Source: §4.2.2 · pp. 22–23

4.3.2. Evaluation Results

  • DeepSeek-V4.1-Flash-Base uses fewer parameters and a heavily reduced KV cache than DeepSeek-V4-Pro-Base while performing on par overall, with comparable knowledge and comprehension and close-to-or-better reasoning and coding.The paper attributes improvements in part to data curation and native multimodal training.Source: §4.3.2 · pp. 23–24, Table 1 · p. 24
  • DeepSeek-V4.1-Flash-Base achieves the lowest BPB on all held-out internal evaluation tasks.The evaluation covers internal documentation, proprietary code, and academic materials; lower BPB indicates better performance.Source: §4.3.2 · p. 24, Figure 6 · p. 25

5. Post-Training

This section describes the post-training approach, including standard SFT, RL, and OPD, data and environment pipelines, scalable agent training, controllable reasoning effort, and asynchronous infrastructure. It reports reasoning and agentic evaluations covering benchmark performance, reasoning-effort tradeoffs, robustness across agent scaffolds, and preliminary multi-agent benefits.

5.1. Post-Training Pipeline

  • The recipe retains standard SFT followed by RL and OPD, while concentrating effort on automated task and environment synthesis, filtering, deduplication, and difficulty calibration.The authors report that, under a fixed optimization procedure, improvements in synthesized data and environments account for essentially all observed gains.Source: §5.1 · p. 25

5.1.1. Large-Scale Agent Task Synthesis

  • Tasks are formalized as problem–environment–verification triplets and iteratively trained for difficulty and correctness, with real-workflow and coding pipelines producing monitored RL data.General-agent environments replay real-workflow failures; coding environments use filtered sessions or public repositories and undergo construction, testing, and independent inspection.Source: §5.1.1 · pp. 25–26

5.1.2. RL in Synthesized Tasks

  • Performance improves with cumulative RL steps across single, related, and heterogeneous scaffolds; checkpoint merging further improves task performance and token efficiency.Disconnected curve segments in Figures 7 and 8 reflect successive RL runs after model reinitialization.Source: §5.1.2 · pp. 26–27, Figure 7 · p. 27, Figure 8 · p. 28

5.1.3. Running Agents at Massive Scale: DSec

  • More than 2,500 concurrent live containers per physical node were supported before measurable degradation, up from roughly 1,000 under comparable workloads.DSec combines NUMA-local worker VMs with memory overcommitment to increase sandbox density.Source: §5.1.3 · p. 28

5.1.4. Controllable Reasoning Effort in RL

  • A scalar effort level b, ranging from 1 to 100, lets one checkpoint vary reasoning strength at deployment, including interpolated behaviors between trained effort levels.Changing b moves the model along a learned cost–quality frontier to accommodate latency, token-budget, and solution-quality requirements.Source: §5.1.4 · pp. 29–30

5.2. Asynchronous Post-training Infrastructure

  • Asynchronous post-training mitigates rollout long tails, while sample-level dispatch sustains concurrency and dataset limits, short-sample discarding, off-policy bounds, and stale-token masking address training-quality risks.Fast interruption and resumable rollouts also let jobs respond to cluster preemption without losing progress.Source: §5.2 · p. 30, §5.2.1 · p. 30, §5.2.2 · p. 31

5.2.4. Large-Scale On-Policy Distillation

  • The final full-vocabulary OPD stage uses over 40 heterogeneous teacher models and supports negligible-cost switching among them and consistent training-configuration changes.Teachers can differ in architecture and may come from different stages of model development.Source: §5.2.4 · pp. 31–32

5.3.1. Evaluation Setup

  • Evaluation covers reasoning and agentic benchmarks, but exploit-seeking persists despite safeguards, motivating stronger benchmark defenses; Table 3 compares Flash with open- and closed-source models.The evaluation setup restricts internet access and removes Git histories and transient caches, yet reports exploit-seeking behavior during testing.Source: §5.3.1 · pp. 32–33, Table 3 · p. 33

5.3.2. Evaluation Results

  • In core reasoning tasks, DeepSeek-V4.1-Flash surpasses DeepSeek-V4-Flash and DeepSeek-V4-Pro on Codeforces and matches leading baselines on MathArena Apex.Its Codeforces rating is 3471 versus 3289 for DeepSeek-V4-Flash and 3348 for DeepSeek-V4-Pro; its MathArena Apex Pass@1 is 65.6%, matching Kimi-K3 (65.6%) and exceeding DeepSeek-V4-Pro (65.3%), while its GPQA Diamond score is 90.9% versus 89.9% for DeepSeek-V4-Flash.Source: §5.3.2 · p. 33
  • DeepSeek-V4.1-Flash improves over DeepSeek-V4-Flash across reasoning and agentic benchmarks, matching or outperforming top-tier open-source and proprietary models.On DeepSWE v1.1 it scores 74.2% versus DeepSeek-V4-Flash at 54.4%, Opus-5 at 74.0%, and GPT-5.6 Sol at 73.0%; on Terminal-Bench 2.1 it scores 90.6% versus Opus-5 at 89.1% and GLM-5.3 at 88.2%.Source: §5.3.2 · p. 33
  • The model establishes a new open-source state of the art on cybersecurity tasks and outperforms Kimi-K3 on visual-agent tasks, while retaining a gap to leading closed-source alternatives.The paper encourages responsible use of dual-use cybersecurity capabilities, including defensive research and vulnerability remediation; visual-agent strengths include visual reasoning and complex professional-chart analysis.Source: §5.3.2 · pp. 33–34

5.3.3. Performance across reasoning efforts

  • Reasoning effort provides a controllable accuracy–cost tradeoff, with accuracy and output length increasing as effort rises across reasoning and agentic benchmarks.Raising effort from 25 to 100 increases average Pass@1 from 67.1% to 76.3% on eight reasoning-intensive benchmarks, DeepSWE v1.1 from 66.0% to 74.2%, and Terminal-Bench 2.1 from 82.4% to 90.6%, at roughly 2.5× more output tokens. Effort 60–80 recovers most maximum-setting accuracy at less than half its token budget; API presets low, high, and max map to effort values 50, 75, and 100.Source: §5.3.2 · p. 34, §5.3.3 · p. 34, Figure 9 · p. 35

5.3.4. Performance across agent scaffolds

  • The model’s agentic capabilities transfer across scaffold families with different prompts and tool interfaces and are not tightly coupled to one scaffold design.The comparison covers eight configurations from six scaffold families, holding the checkpoint, decoding configuration, and task set identical while changing the surrounding harness; Table 4 reports results at max reasoning effort.Source: §5.3.4 · pp. 34–35, Table 4 · p. 35

5.3.5. Multi-Agent

  • In preliminary tests, multi-agent configurations outperform single-agent counterparts at every tested deadline on ProgramBench and FrontierSWE v2.On ProgramBench, Almost@1 rises from 13.59% at 1 hour to a peak of 30.04% at 8 hours for multi-agent, versus 12.79% and 20.39% for single-agent; on FrontierSWE v2, Mean@5 rises from 13.50% at 1 hour to 32.90% at 20 hours, versus 10.50% to 28.20%.Source: Figure 10 · p. 36, §5.3.5 · p. 36

6. Conclusion, Limitations, and Future Directions

DeepSeek-V4.1-Flash combines architectural and deployment changes to reduce compute and KV cache demands while delivering strong benchmark performance. The conclusion also identifies untested robustness boundaries and a remaining gap on the hardest tasks, and outlines future evaluation, scaling, and model–harness co-design.

Conclusion

  • 890 bytes per token is DeepSeek-V4.1-Flash’s global KV footprint, roughly 1/4 of DeepSeek-V4-Flash’s, while SWA Bounded Replay reduces persistent footprint to roughly 1/8.CSA2 cross-layer KV reuse and FP4 caching reduce the global footprint, which is always in HBM; the persistent cache is on SSD or in host memory. CED activates 8B parameters per token during prefill versus 16B during decode.Source: §6 · p. 37
  • Despite a significantly smaller parameter footprint than GLM-5.3 and Kimi-K3, DeepSeek-V4.1 achieves comparable—and on several tasks, superior—performance across key benchmarks.The authors also report substantially better overall performance than DeepSeek-V4-Flash despite DeepSeek-V4.1-Flash’s much smaller KV cache footprint.Source: §6 · p. 37

Limitations and Future Directions

  • Internal evaluations found no systematic capability degradation, but untested extremes may expose failures from CSA2 selection errors or SWA Bounded Replay state reconstruction.The authors plan to focus stress testing on sparse retrieval over long contexts and SWA state reconstruction at cache-resumption boundaries, while monitoring real-world workloads.Source: §6 · p. 37

Limitations

  • Although benchmark scores are narrowly separated from Fable-5 and GPT-6 Astra, DeepSeek-V4.1-Flash still trails frontier systems on difficult reasoning and edge cases.The passage says standard benchmarks increasingly reach saturation and describes the model as offering a highly comparable user experience in daily applications.Source: §6 · p. 37

Future Directions

  • Future work will update evaluation protocols, scale data, model capacity, and RL, and co-design models with harnesses while pursuing cost reduction and broader access.The authors identify large-scale data synthesis and RL scaling as challenges to address.Source: §6 · p. 37

Appendix · B. Evaluation Details · B.1. Scaffold Configurations

The appendix standardizes scaffold evaluation in Linux task containers without adding an experimental system prompt, then documents each scaffold’s version and tool setup. It also describes a Claude Code comparison across four versions, with averages computed from unrounded Pass@1 scores.

B.1. Scaffold Configurations

  • All scaffolds run in Linux task containers with shared evaluation settings, using each runtime’s prompts, task templates, and tool definitions without an added experimental system prompt.Runs start from the benchmark task description.Source: §B.1 · p. 47
  • Claude Code uses the Agent SDK and each version’s native tool interface; Table 5 compares four versions, and its average uses unrounded Pass@1 scores.The versions are v2.1.105, v2.1.238, v2.1.251, and v2.1.259; Table 4 reports v2.1.251.Source: §B.1 · p. 48, Table 5 · p. 48
  • Codex v0.147.0 uses standard app-server mode with adapted tool schemas.Source: §B.1 · p. 48
  • OpenCode v1.18.15 uses the build agent with shell and file tools, plus native task delegation.Source: §B.1 · p. 48
  • Pi v0.84.2 runs in RPC mode with file and shell tools and a search extension for search, open_page, and find_in_page.Source: §B.1 · p. 48
  • mini-SWE uses the mini_swe_v2 port1 with a single bash tool, requiring a tool call each turn and a submission marker to finish.Source: §B.1 · p. 48
  • DeepSeek Harness provides Minimal with one bash tool, Standard with 26 initial function tools, and PTC with run_code for TypeScript programs using 24 underlying tools.Standard and PTC use v0.1.1+ custom; Standard includes web search and fetch.Source: §B.1 · p. 48

B.2. Reasoning Efforts across Scaffolds

Across six scaffold–benchmark panels, higher reasoning effort consistently lengthens trajectories, while Pass@1 improves only loosely and non-monotonically. Scaffold calibration and performance ordering vary by benchmark.

B.2. Reasoning Efforts across Scaffolds

  • Mean output tokens per trajectory grow monotonically with reasoning effort in all six scaffold–benchmark pairs, but Pass@1 has intermediate plateaus and dips.Pass@1 improves overall, but its response to increased effort is not monotonic in most panels.Source: §B.2 · p. 48
  • On DeepSWE v1.1, Claude Code adds comparatively few tokens as effort rises, while DeepSeek Harness (Minimal) starts lowest and gains most at the largest token cost.mini-SWE falls between the other two scaffolds on this benchmark.Source: §B.2 · p. 48
  • On Terminal-Bench v2.1, scaffold curves occupy a narrow band, and maximum-effort ordering favors DeepSeek Harness (Minimal), then mini-SWE, then Claude Code.Once tasks are nearly saturated, scaffold choice matters at least as much as effort tier.Source: §B.2 · p. 48

B.3. Detailed Results of Reasoning Benchmarks across Reasoning Efforts

Across eight reasoning benchmarks, increasing reasoning effort smoothly increases response length while maintaining or improving accuracy. The effort setting provides a predictable cost–accuracy control for deployment.

B.3. Detailed Results of Reasoning Benchmarks across Reasoning Efforts

  • MathArena Apex 2025 gains +40.3 points (25.3%→65.6%), Apex 2025 Shortlist gains +11.5, and AIME 2026 reaches 100% as effort increases.No benchmark degrades with increased effort; GPQA Diamond remains stable, while LiveCodeBench gains +2.6.Source: §B.3 · pp. 48–49
  • Raising effort from 25 to 100 increases average response length by 2.0–3.1× across benchmarks, from 4.6k to 11.4k tokens on AIME 2026 and 29.1k to 86.1k on MathArena Apex 2025.The passage reports no runaway growth or anomalies.Source: §B.3 · p. 48
  • The effort setting acts as a predictable knob for selecting a cost–accuracy operating point to match deployment latency and compute budgets.Length scales smoothly, and accuracy responds in the right direction across the benchmarks.Source: §B.3 · pp. 48–49

C. Exponential Token Penalty in Reasoning Effort Control

The section motivates an exponential token-penalty schedule as a deployment-time control over the cost–quality trade-off without a hard token budget. Under a simplified local model, it predicts an affine relationship between requested effort and preferred reasoning length, with important limits on that prediction.

C. Exponential Token Penalty in Reasoning Effort Control

  • Lower effort applies a stronger token penalty, while higher effort permits more computation without imposing a hard token budget.The effort variable controls the cost–quality trade-off during deployment.Source: C. Exponential Token Penalty in… · p. 49
  • The exponential schedule’s penalty coefficient is anchored by k0 at the lowest effort and decays at a rate controlled by τ.The length deduction is measured against a reference length and capped by Cmax.Source: C. Exponential Token Penalty in… · p. 49
  • Assuming marginal solve-probability gains decay exponentially with reasoning length, the local model predicts an affine trend between requested effort and preferred reasoning length.The marginal benefit is instance-dependent, and the derivation assumes an interior optimum with the penalty cap inactive.Source: C. Exponential Token Penalty in… · pp. 49–50
  • The affine trend is a local reward-level approximation, not a guarantee that measured average reasoning lengths are linear or pointwise monotonic.Realized behavior can vary with strategy changes, stochastic generation, differing trajectory turn counts, and subgroup reward normalization; once the cap is reached, the marginal token penalty is zero.Source: C. Exponential Token Penalty in… · pp. 50–51
Loading 2609.19969v1…