PaperlayerView exact revision on arXiv

Source-linked AI summary

SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

Dongfang Li, Xiaodong Luo, Ruoyu Sun, Xuhui Chen, Linyuan Qiu, Jian Meng, Zhengxuan Lu, Yiting Wang, Yucheng Xie, Tao Guo, Tianxiang Fang, Jing Li, Sihang Chen, Shihao Hong, Chang Liu, Weihua Dai, Zirong Zeng, Ziwei Zhu, Zhuohan Wang, Zhengjun Yue, Igor Vasilyev, Min Liu, Weijian Sun, Xin Chen, Yingmeng Gao, Jinhua Zhou, Taolue Chen, Chenwei Wu, Dong Zhang, Wenlong Jin, Jinmin Xiang, Barkova Maria, Ushakov Anton, Xianfei Jin, Tian Ding, Zhihang Lin, Qian Chen, Linxin Yang, Mingzhe Yang, Bingwei Zhang, Hongzhang Yang, Fangxue Zhang, Shijun Qin, Jie Yu, Cuihua Hu, Tolstykh Vasiliy, Nosov Ivan, Abdullin Amir, Zhicheng Zhou, Xin Zhang, Zhixiong Ning, Xutong Zhao, Junjie Huang, Jiajun Liu, Weiyan Kong, Zheng Zhang, Wenhan Luo, Lin Hu, Yangbo Guo, Li Zeng, Shihao Zhang, Baotian Hu, Min Zhang, Haizhou Li, Zhiquan Luo

arXiv:2607.20145v2cs.CLcs.AI

TL;DR

Trillion-parameter LLM post-training on SIMD hardware remains underexplored, while OR integration into LLM training and evaluation remains at an early stage. SLAI T-Rex couples full-stack Ascend SuperPOD training optimization with solver-grounded OR CPT–SFT, reporting 34.22% MFU and a 2.93× improvement over the default baseline.

  • Problem

    Trillion-parameter LLM post-training on SIMD hardware remains underexplored, while OR integration into LLM training and evaluation remains at an early stage.

    Source: §1 · p. 4

  • Method

    SLAI T-Rex couples full-stack Ascend SuperPOD training optimization with solver-grounded CPT–SFT specialization for OR.

    Source: §5 · p. 47

  • Results

    34.22% MFU, with a 2.93× improvement over the default baseline, is reported for Ascend training; the DeepSeek-V4-Pro workflow improves average OR score by 7.17 percentage points.

    Source: §1 · p. 5, §5 · p. 47

  • Takeaways & Limitations

    The results validate the training efficiency and practical feasibility of large-scale post-training on Ascend NPU SuperPOD.

    Source: §5 · p. 47

    Evaluation remains to be extended to more diverse OR problem families and general-capability benchmarks.

    Source: §5 · p. 48

Abstract

from arXiv · show

Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficient kernel execution. While most large-scale LLM training systems are built around GPU-based clusters, this report presents an end-to-end optimization practice on the Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, we develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution. The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe while maintaining training stability. Building on this optimized infrastructure, we further establish a CPT and SFT workflow for complex Operations Research (OR) tasks. We refer to the integrated framework as SLAI T-Rex. Using DeepSeek-V4-Flash, we develop OR-oriented CPT and SFT data pipelines that combine collected domain resources with solver-verified synthetic optimization documents. The resulting dataset contains 10K high-quality SFT samples spanning four task categories and three problem representations. The specialized model achieves the highest average zero-shot Pass@1 score among the evaluated models, reaching 71.81% and outperforming GPT-5.4-Mini and the base DeepSeek-V4-Flash model by 3.98 and 11.27 percentage points, respectively. Overall, this work demonstrates a full-stack pathway from efficient trillion-parameter model post-training on Ascend infra to domain-specialized Flash models for solver-grounded mathematical modeling, advancing frontier-model systems for complex reasoning.

1. Introduction

SLAI T-Rex combines Ascend-native system optimization for DeepSeek-V4-Pro with solver-grounded OR post-training developed on DeepSeek-V4-Flash and applied to Pro. The introduction reports gains in training efficiency and OR benchmark performance.

1 Introduction

  • Prior LLM-based OR work is constrained by small-scale benchmarks, brittle semantic-to-structure mapping, and insufficient correctness guarantees in complex industrial scenarios.The introduction identifies domain-specific OR training, dedicated data preparation, and verifiable optimization-environment feedback as under-explored needs, especially for trillion-parameter models.Source: §1 · p. 4

1. Introduction

  • 34.22% MFU for DeepSeek-V4-Pro, up from 11.67% and a 2.93× improvement over the open-source baseline recipe.The system jointly optimizes multi-dimensional parallelism, communication orchestration, memory management, and kernel execution on Ascend SuperPOD.Source: §1 · p. 5
  • 71.81% average score across four OR benchmarks, exceeding GPT-5.4-Mini by 3.98 percentage points and original DeepSeek-V4-Flash by 11.27 percentage points.SLAI T-Rex-Flash combines task-specific SFT supervision with OR-domain modeling knowledge from CPT, improving executable optimization-program generation and formulation correctness.Source: §1 · p. 5
  • Direct SFT raises DeepSeek-V4-Flash B4O-Feasible from 60.47% to 65.93% and B4O-ORGEval from 34.26% to 48.73%; selected-CPT initialization further raises them to 71.22% and 59.39%, respectively.The compared SFT runs use the same data and optimization settings.Source: §1 · p. 5
  • 77.33% average OR score for DeepSeek-V4-Pro after applying the workflow, compared with 70.16% before.The report presents the Pro result alongside system optimization as evidence that SLAI T-Rex supports both trillion-parameter post-training and solver-grounded OR specialization on Ascend SuperPOD.Source: §1 · p. 5
  • AuraKernel introduces OR-guided tiling optimization into AscendC kernel optimization, using an OR solver and dedicated harness for accurate tiling selection and broader search.The work frames this as an early application of Operations Research to Ascend kernel optimization, beyond manual and heuristic approaches.Source: §1 · p. 5
  • The OR-oriented CPT–SFT workflow covers solver-grounded CPT data construction, self-distilled SFT generation, contract-aware cleaning, and benchmark evaluation.The report also profiles how DeepSeek-V4 architecture interacts with Ascend kernels, communication, memory hierarchy, and distributed orchestration, and proposes SuperPOD-aware orchestration for complex MoE communication and parallelism.Source: §1 · p. 5

2. Training Infrastructure on Ascend CloudMatrix384 SuperPOD

This section describes optimization of full-parameter training for DeepSeek-V4-Pro on Ascend CloudMatrix384 SuperPOD, spanning distributed parallelism and scheduling, memory management, and Ascend-specific kernel and dataflow changes. It covers AuraKernel’s solver-guided, hardware-verified kernel search, operator-chain fusion or rewriting, and Ascend-aware changes to mHC, limited-SwiGLU, and rotary-embedding paths.

2 Training Infrastructure on Ascend CloudMatrix384 SuperPOD

  • The 1.6-trillion-parameter DeepSeek-V4-Pro workload stresses parameter and optimizer-state capacity, expert routing with intensive all-to-all communication, and irregular sparse-attention computation.These demands expose memory, expert-parallel scalability, token-dispatch, communication-overlap, and kernel-performance bottlenecks.Source: §2 · p. 6

2.1 Bottlenecks of Training Trillion-parameter LLM on Ascend SuperPOD

  • Communication accounts for 52.2% (40.76 s) of device kernel time; TP/DP collection and pipeline transfers dominate, while MoE communication is comparatively minimal.Pipeline bubbles also arise from imbalanced computation across stages; VPP’s fine-grained interleaving is used to mitigate them while staying within memory limits.Source: Table 1 · p. 7, §2.1 · p. 7, §2.2 · p. 10

2.3.1 Bottleneck Kernel Analysis

  • Profiling identifies sparse-attention backward as a major kernel bottleneck, alongside a fragmented, memory-bound tail that motivates kernel-internal tuning and operator fusion or rewriting.Sparse attention contributes roughly a quarter of non-communication kernel time, while eager non-matmul operators account for 91.3% of launches but 66.7% of Task Duration; Cast alone contributes 6.02% of Task Duration. Vector-core kernels occupy 36.5% of Task Duration at about 15% vector-compute utilization, versus approximately 68% MTE load and 40% store ratios.Source: §2.1 · p. 7, §2.3.1 · p. 12, Figure 6 · p. 13

2.2 Optimizing Trillion-parameter LLM Training System on Ascend SuperPOD

  • Attention and MoE use decoupled parallelism: attention TP=2 and MoE ETP=1; VPP mitigates pipeline bubbles, with peak reserved memory reaching 51.7–54.6 GB per NPU.All-to-all is 14.5% of communication time. Double-buffered swapping overlaps state transfers with optimizer computation, substantially reducing HBM consumption with limited runtime overhead.Source: §2.2 · pp. 9–11

2.3.2 AuraKernel: Performance Optimization Agent for Complex Ascend Kernels

  • AuraKernel targets complex AscendC kernel optimization, where agent-based tuning is challenging because Ascend hardware and programming differ from CUDA and operators combine host scheduling with device kernels.The framework uses an agent-driven path for top-ranked bottleneck kernels, complementing fusion and re-expression of fragmented operator chains.Source: §2.1 · p. 8, §2.2 · p. 9, §2.3.2 · p. 13
  • AuraKernel uses a hardware-aware parameter model and profiling-driven loop, while K-Search explores a branchable, rollbackable tree and verified gains become reusable skills.Its persistent memory retains optimization decisions, profiling observations, and correctness constraints across long optimization trajectories.Source: §2.3.2 · pp. 14, 17, 18, Figure 7 · p. 14
  • 6.55% execution time reduction on frequent LLM MatMul shapes was achieved by optimizing tiling alone, giving iterative kernel optimization a stronger starting configuration.The solver models workload partitioning, memory-level tile staging, transfer-compute overlap, and writeback; its cycle-level cost model targets bottlenecks under a steady-state pipelined execution assumption.Source: §2.3.2 · p. 15
  • Up to 1.24×, AuraKernel accelerates the shared-key-value sparse-attention backward step in DeepSeek-V4-Pro training; it also optimizes 14 reduction-heavy operators to 2.06× average and 11.90× best speedup.The sparse-attention step is the largest device-time contributor; the operator gains come from reducing redundant global-memory traffic rather than raising raw compute utilization.Source: §2.3.2 · p. 18

2.3.3 Ascend NPU-aware Kernel Fusion and Rewrite

  • The fusion-and-rewrite path preserves operator arithmetic while keeping intermediates in UB, returning whole rows or stage outputs, and following stage-specific dtype transitions.It targets mHC compression, limited SwiGLU backward, and unified rotary embedding; the mathematical, frontend-artifact, and hardware execution contracts are used to audit the rewrites.Source: §2.3.3 · pp. 18–19, Figure 8 · p. 19
  • The mHC suite combines weighted expansion, two-dimensional tiling, and mixed-precision dataflow, using fused AscendC stages to reduce framework-level kernel boundaries and manage memory movement.The post-projection kernel tiles rows across cores and the hidden dimension into 512 B-aligned blocks, scheduling MTE2/VEC/MTE3 movement within the fused operator.Source: §2.3.3 · pp. 19–20, Figure 9 · p. 20
  • A limited-SwiGLU backward chain of roughly 401 μs is replaced by one fused kernel that keeps intermediates on chip and writes gradient halves directly to their outputs.BF16/FP16 inputs are promoted once for local derivative computation; unsupported dtypes and non-positive limits fall back to the original path. The formula is checked against a PyTorch autograd reference, including clamp boundaries.Source: §2.3.3 · p. 21, Figure 10 · p. 22
  • About 82 ms of eager MoE routing-gradient work disappears when the fused kernel reconstructs activations and reduces routing gradients within the same pass.The reduction uses unweighted incoming gradients, keeps reconstructed activations in FP32, and combines partial sums with global-memory atomics when hidden rows are split across tiles.Source: §2.3.3 · p. 21, Figure 10 · p. 22
  • One unified rotary operator serves q, kv, and o paths, avoids autograd’s slice-assignment scatter chain, and caches position-dependent frequency conversions after the first step.Real arithmetic implements rotation through cosine multiplication, adjacent-pair swapping, and sign-flipped sine; validation across all three sites reports relative error within accepted training tolerance.Source: §2.3.3 · pp. 21–22
  • The mHC pre-projection’s early FP32 promotion doubled transfer volume in a memory-bound stage, motivating a BF16-aware, UB-resident contraction and pipelined tile execution.The stage runs 896 forward and 448 backward calls per step; its cited AIV vector-compute ratio is ≈0.06 versus an MTE2 load ratio of ≈0.87. The optimized pre-BMM moves the path from movement-bound to compute-bound.Source: §2.3.3 · pp. 20–22

3. OR-Oriented Post-Training Workflow

This section presents an Ascend CPT-to-SFT workflow for adapting DeepSeek-V4-Flash to solver-oriented OR modeling, motivated by an analysis of its existing capabilities and failure modes. It covers OR knowledge building through a domain-dominant CPT corpus and solver-verified synthetic data, followed by self-distilled, cleaned, and progressively trained SFT data and matched checkpoint transfer evaluation.

3.1.1 Benchmark-Level Signal

  • Under Pass@1, B4O-Feasible reaches 60.47% versus 34.26% for B4O ORGEval; despite a 79.19% ORGEval code/build pass rate, executability and structural equivalence are only weakly aligned.Source: §3.1.1 · p. 24

3.1.2 Error Spectrum Across Settings

  • The error spectrum spans structural mismatches, protocol and schema failures, missing discrete-variable semantics, complex constraints, ratio modeling, and solver-incompatible nonlinear formulations.Structural equivalence errors dominate overall, while Pass@1 most visibly fails on protocol, API, and schema compliance.Source: §3.1.2 · p. 25, §3.1.3 · pp. 26–27

3.1.3 Inference-Setting Shift

  • Pass@16 reduced failures from 425 under 5-shot to 370 but left 127 structural ORGEval errors (34.3%), versus 108 (25.4%) under 5-shot.Additional sampling improves value-based benchmarks but does not reliably select canonical variable–constraint graphs.Source: §3.1.3 · pp. 25, 27, Figure 12 · p. 26

3.1.4 Two-Stage Training Adaptation: Why Both CPT and SFT

  • CPT targets persistent OR knowledge and structural-modeling gaps, while SFT stabilizes solver-facing protocols and formulations that prompting or sampling can activate.The workflow uses CPT to strengthen OR priors before SFT aligns generation with verified Gurobi code, output contracts, and canonical formulations.Source: §3.1.3 · p. 27, §3.1.4 · p. 28

3.2 OR-Oriented Continued Pre-Training

  • CPT uses a domain-dominant mixture centered on solver-verified OR documents to address gaps in variable semantics, nonlinear and ratio formulations, and canonical model structure.Mathematical, code, and general-English data support symbolic reasoning, solver implementation, and broad language capability.Source: §3.2 · p. 28, §3.2.1 · p. 30

3.2.1 OR-CPT Data Construction and Domain-Dominant Mixture

  • The OR-CPT Data Engine synthesizes documents bidirectionally from parameterized optimization instances, checking reconstructed formulations against independent Gurobi solver references.It applies task-family contracts and accepts reconstructed programs only when they reach valid solver status and match the reference objective within tolerance.Source: §3.2.1 · pp. 28–29
  • The engine’s accepted documents are scalable, solver-verifiable, and structurally informative, linking natural-language requirements to variable families, constraints, code, and objective values.The CPT corpus combines these synthetic documents with collected OR resources in a pool of approximately 100K samples, alongside supporting math, code, and general-English corpora.Source: §3.2.1 · p. 30

3.2.2 Training Execution in the Ascend Ecosystem

  • Ascend CPT uses MindSpeed and TorchTitan-NPU with a shared objective, data interface, monitoring, and checkpoint workflow, supporting long-context and sparse-MoE execution.The setup includes context and expert parallelism, token dispatch, grouped expert computation, model-specific operators, MTP, recomputation, and memory-aware optimizer execution.Source: §3.2.2 · p. 30

3.2.3 Checkpoint Selection and CPT-to-SFT Transfer Validation

  • CPT checkpoints are assessed for OR likelihood, general-capability retention, and downstream transfer, with matched SFT-only and CPT-plus-SFT routes isolating CPT’s contribution.Selection favors reduced OR validation loss, stable optimization, preserved general capability, and positive transfer across benchmarks including NL4OPT, OptiBench, B4O-Feasible, and B4O-ORGEval.Source: §3.2.3 · p. 31

3.3 Data-Efficient Supervised Fine-Tuning for Operations Research

  • The authors compare 3K, 10K, and 50K synthetic datasets and select 10K for cleaning because scaling to 50K did not recover NL4OPT and OptiBench performance.The 10K dataset improved Bench4Opt protocol-oriented tasks but remained below baseline on NL4OPT and OptiBench.Source: §3.3 · p. 31, §3.3.3 · p. 33

3.3.1 Solver-Oriented SFT Objective

  • Solver-oriented SFT teaches structured modeling trajectories, and masks prompt, system, and formatting tokens so gradients update only target-response tokens.Targets cover sets, parameters, variables, objectives, constraints, structured data, Gurobi-style APIs, and LP writing when required.Source: §3.3.1 · p. 32

3.3.2 Self-Distilled OR Modeling Data Flywheel

  • The self-distillation flywheel transforms seed problems through canonical, semantic, and synthetic IRs, then renders and validates varied tasks while filtering textual and structural near-duplicates.Accepted synthetic IRs re-enter the generation pool, reducing dependence on original seeds while using them as anchors against distributional drift.Source: §3.3.2 · pp. 32–33

3.3.3 Contract-Aware Cleaning and CoT Enhancement

  • Clean-CoT exported 8,881 of 10,000 samples after reviewer filtering and validation, compared with 8,731 for clean-only.For tagged Markdown targets, short structured <think> blocks reinforce modeling checks, while code-only and LP-equivalence tasks retain strict output contracts.Source: §3.3.3 · p. 34

3.3.4 Progressive SFT with General Capability Anchoring

  • Progressive SFT first aligns solver APIs on code-only OR targets at 5 × 10−7, then trains on Clean-CoT 10K at 1 × 10−6 with 10%–15% general-purpose instruction data.The stages separate solver syntax from full modeling adaptation, while general-purpose data anchors language capability and helps preserve NL4OPT performance.Source: §3.3.4 · p. 34

3.4 AI-Assisted Data Construction and Refinement

  • An independent Reviewer detected a hidden cost mismatch; across 10,000 Clean-CoT samples, reviewers intercepted 826 Cleaner-accepted samples and local validation caught 293 more.The Cleaner had changed a warehouse-to-store cost in the model description while the Python code retained the original value.Source: §3.4 · p. 35

4. Experimental Results

The experiments evaluate AuraKernel and operator fusion on profiled workloads, alongside Ascend SFT stability and OR-specific data scaling, cleaning, and CPT initialization. They examine how CPT and SFT contribute to OR post-training and how the complete workflow affects DeepSeek-V4-Pro, broad capability, and OR performance.

4.1 Efficient Kernels via AuraKernel

  • AuraKernel brings 14 Triton-Ascend operators to 100% completion at a 2.06× average speedup and an 11.90× best speedup.For sparse attention, forward speedups are 1.23× and 1.09×, and backward speedups are 1.24× and 1.21× at compression ratios 128 and 4. Batching queries reduces repeated memory traffic; RMS-normalization tiling and FP32 accumulation shift work from scalar to vector execution.Source: Table 7 · p. 36, §4.1 · pp. 37–38, Figure 16 · p. 37

4.2 Performance Improvements from Kernel Fusion

  • mHC pre-projection, backward, and post-projection fusion reach module-level speedups of 3.03×, 1.88×, and 2.74×, respectively.Limited SwiGLU fusion reaches 8.86× forward and 3.27× backward, while unified RoPE reaches 2.33× forward and 3.54× backward. Gains come from removing eager-chain launches, intermediate tensors, casts, and writebacks; post-projection backward remains unfused in this profile.Source: §4.2 · pp. 38–40, Table 8 · p. 39, Figure 19 · p. 39

4.3 Long-term Training Stability with Ascend SuperPOD

  • The 50K self-distilled SFT run reaches 800/800 steps on 8 nodes with 128 Ascend 910C NPUs, without skipped or NaN iterations.With GBS=128, MBS=1, and SEQ_LEN=8192, final lm loss is approximately 0.079 and mtp_1 loss approximately 0.093, supporting the stability of the full-parameter SFT workflow.Source: Figure 20 · p. 41, §4.3 · p. 41

4.4 SFT Scaling and Cleaning Analysis

  • SFT-10K raises B4O-Feasible from 60.47 to 65.07 and B4O-ORGEval from 34.26 to 47.21, but lowers NL4OPT from 84.08 to 81.66 and OptiBench from 63.33 to 62.67.Increasing the uncleaned data to 50K further lowers OptiBench to 58.68; the results indicate that scaling this self-distillation pipeline without stronger verification can amplify weak or biased samples.Source: §4.4 · pp. 41–42, Table 10 · p. 42
  • Clean-CoT improves NL4OPT by 5.27 percentage points, OptiBench by 1.50 percentage points, B4O-Feasible by 0.86 percentage points, and B4O-ORGEval by 1.52 percentage points over uncleaned SFT-10K.Its gains are strongest on natural-language tasks where structured reasoning can appear in the output; cleaning and CoT reduce wrong-objective/model errors, but do not reduce OptiBench nonlinear/Gurobi-form errors. The final SFT samples retain a rule-driven, auditable history through reviewer approval and validator execution.Source: §4 · p. 36, §4.4 · p. 42, Table 11 · p. 42

4.5 Effect of CPT Initialization under Matched SFT Conditions

  • Under matched SFT conditions, CPT initialization adds +2.59 pp on NL4OPT, +2.95 pp on OptiBench, +5.29 pp on B4O-Feasible, and +10.66 pp on B4O-ORGEval.Both routes use the same 10K chain-of-thought OR modeling samples, 220 SFT iterations, peak learning rate 5 × 10−6 with cosine decay, global batch size 128, and sequence length 8192; therefore, performance differences can be attributed to CPT initialization rather than different SFT data or optimization settings.Source: §4.5 · p. 43

4.6 Performance of the Proposed Post-Training Pipeline

  • Evaluation covers four OR benchmarks and six general benchmarks, using shared prompts and decoding budgets for comparison.B4O-Feasible requires runtime execution and contract adherence, while B4O-ORGEval measures semantic structural match; future work is proposed on broader CPT task coverage, solver-verified documents, and retention-aware data mixtures.Source: §4.6 · p. 44, Table 13 · p. 44
  • CPT+SFT consistently beats Base and SFT on 0-shot Pass@1 across OR benchmarks, with its largest stated gains on B4O-ORGEval and B4O-Feasible.Over SFT, gains are +10.66 pp on B4O-ORGEval 0-shot Pass@1 and +5.29 pp on B4O-Feasible 0-shot Pass@1.Source: §4.6 · p. 44

4.7 Scale-Up Validation on DeepSeek-V4-Pro

  • 10–15-point drops on general benchmarks under Pure OR, with no OR Overall improvement over Balanced, indicate that exclusive OR pretraining overfits.Balanced mixes OR and general data, supporting OR gains while retaining broad reasoning capability.Source: Figure 21 · p. 45, §4.7 · p. 46

4.6 Performance of the Proposed Post-Training Pipeline

  • 76.14 is the CPT+SFT three-setting mean on B4O-Feasible, versus 69.39 for SFT; SFT remains below Base in 5-shot.The results indicate that SFT supports format and protocol alignment, while CPT contributes domain knowledge important for executable code and structural reasoning.Source: §4.6 · pp. 44–45
  • CPT+SFT attains the highest overall score across four OR benchmarks, while general-benchmark results remain comparable to Base without signs of catastrophic forgetting.The reported general-benchmark changes include +0.9 on MMLU and -1.7 on MATH.Source: §4.6 · p. 45, Table 15 · p. 45, Table 14 · p. 45

4.7 Scale-Up Validation on DeepSeek-V4-Pro

  • 77.33% average OR score under zero-shot Pass@4 gives SLAI T-Rex-Pro a 7.17-point gain over original DeepSeek-V4-Pro.It also exceeds the SFT-only average of 76.74%.Source: Table 16 · p. 46, §4.7 · p. 46
  • The complete SLAI T-Rex-Pro configuration improves original-model NL4OPT from 86.51% to 92.04% and B4O-ORGEval from 43.91% to 61.93%, with the largest improvement on B4O-ORGEval.Compared with SFT alone, it scores 85.17% versus 86.04% on B4O-Feasible. Pro CPT was not separately ablated.Source: §4.7 · p. 46

5. Conclusion, Limitations, and Future Directions

SLAI T-Rex combines Ascend SuperPOD training optimization with solver-grounded CPT–SFT specialization for Operations Research. The report demonstrates stable trillion-parameter post-training and OR-modeling gains on DeepSeek-V4-Flash and DeepSeek-V4-Pro, while identifying future work in training efficiency, data, and evaluation.

5. Conclusion, Limitations, and Future Directions

  • 90.00% AIME 2024 and 86.67% AIME 2025 are SLAI T-Rex-Pro’s best results, while HumanEval is unchanged and CMMLU falls slightly to 93.12%.LiveCodeBench improves from 55.00% to 67.00%, though SFT-only has the highest LiveCodeBench result; the evaluated general benchmarks remain broadly stable.Source: §5 · p. 47, Table 17 · p. 47
  • 34.22% MFU is achieved through end-to-end training orchestration and bottleneck-level kernel optimization for trillion-parameter MoE training on Ascend NPU SuperPOD.The report presents this as a full-stack optimization effort on a non-GPU platform and validates the practical feasibility of large-scale post-training there.Source: §5 · p. 47
  • With CPT initialization, DeepSeek-V4-Flash reaches 71.22% B4O-Feasible and 59.39% B4O-ORGEval, above direct SFT’s 65.93% and 48.73%.Direct SFT improves these metrics from 60.47% and 34.26%, respectively; CPT adds OR-domain modeling knowledge, with the largest additional improvement on B4O-ORGEval. The same overall trend appears on NL4OPT and OptiBench.Source: §5 · p. 47
  • 77.33% average OR score for SLAI T-Rex-Pro exceeds the 70.16% baseline; SFT-only reaches 76.74%, and the complete workflow gains 7.17 percentage points.The largest gain is on B4O-ORGEval; the workflow remains effective on the optimized Ascend SuperPOD stack.Source: §5 · p. 47
  • Targeted contract-aware cleaning and concise CoT-oriented enhancement improve 10K SFT data more reliably than scaling self-distilled data to 50K.The improvements are especially noted on NL4OPT, OptiBench, and ORGEval; the report highlights modeling-error repair, output-contract enforcement, compact reasoning checklists, and solver-grounded verification.Source: §5 · p. 47
  • Released workflow artifacts include prompt templates, format contracts, curriculum configurations, monitoring callbacks, and configurable data and evaluation tools; optimized AscendC kernels are planned for separate release.The released configurations support inspection, reproduction, adaptation, and extension of the introduced procedures; kernel release is intended to enable reproduction of operator-level improvements and adaptation to related Ascend workloads.Source: §5 · p. 48
  • Future work will extend post-training toward AgenticRL, automate parallelism exploration under system constraints, scale solver-verified OR corpora, and broaden evaluations and structural verification.Planned coverage includes nonlinear expressions, ratio constraints, quadratic terms, and Gurobi-specific modeling patterns.Source: §5 · p. 48

Appendix · B Post-training Recipe Details

The appendix outlines recipe-level settings for optimization, parallel execution, memory use, and reproducibility. It distinguishes CPT and SFT training recipes from the inference-only agentic rollout configuration.

B Post-training Recipe Details

  • The appendix focuses on recipe-level settings that affect optimization, parallel execution, memory usage, and reproducibility.Source: B Post-training Recipe Details · p. 61
  • CPT uses a recipe covering long-context setup, optimization, parallelism, and memory-saving mechanisms, as summarized in Table 18.The passage identifies the table’s scope but provides no individual settings or values.Source: B Post-training Recipe Details · p. 61
  • SFT trains on instruction-response trajectories, masking prompt-token loss so gradients apply only to target responses.Its recipe-level configuration is reported in Table 19.Source: B Post-training Recipe Details · p. 61
  • The reported SFT configuration is selected to match instruction-following alignment, solver-oriented modeling, and pipeline-optimized distributed execution on Ascend NPUs.Source: B Post-training Recipe Details · p. 61
  • Agentic rollout is evaluated in inference mode without updating model parameters, so its configuration is kept separate from CPT and SFT training recipes.Source: B Post-training Recipe Details · p. 61
  • Rollout settings, if reported, should cover inference behavior and future reinforcement-learning reproducibility, including candidate counts, review and repair rounds, solver timeout, and success and tolerance criteria.Source: B Post-training Recipe Details · p. 61

C Solver-Verified OR-CPT Data Synthesis: Engine Design and Illustrative Cases

The appendix describes a solver-verified pipeline that turns parameterized optimization instances into self-contained CPT documents. Language-model fluency is not treated as evidence of mathematical correctness; deterministic checks govern sample admissibility.

C Solver-Verified OR-CPT Data Synthesis: Engine Design and Illustrative Cases

  • The pipeline converts parameterized optimization generators into self-contained CPT documents while preserving mathematical invariants across language-model transformations.The appendix focuses on the method for constructing the synthesized OR corpus component.Source: C Solver-Verified OR-CPT Data… · p. 61
  • Language models transform verified instances into business problems and back into mathematical models and executable programs, while generator contracts, static checks, and independent solver execution determine admissibility.Language-model fluency is explicitly not used as a proxy for mathematical correctness.Source: C Solver-Verified OR-CPT Data… · p. 61

C.1 Role in the post-training workflow

The OR-CPT Engine constructs solver-grounded, self-contained CPT documents upstream of and separately from the Ascend 910C training stack. CPT builds OR knowledge, while SFT teaches the model to use it under task instructions and output contracts.

C.1 Role in the post-training workflow

  • The OR-CPT Engine constructs documents for autoregressive continued pre-training, while the Ascend 910C stack tokenizes, packs, samples, and optimizes the resulting corpus.The engine establishes mathematical fidelity, semantic coverage, and provenance; it exports plain, self-contained documents rather than instruction–response conversations.Source: §C.1 · p. 62
  • CPT exposes the model to OR terminology, formulation patterns, solver APIs, and verification habits.Source: §C.1 · p. 62
  • SFT teaches the model to elicit CPT knowledge under task instructions and output contracts, while the stages retain distinct data representations and optimization objectives.Both stages can use the same OR taxonomy.Source: §C.1 · p. 62

C.2 End-to-end engine architecture

The engine creates CPT documents through bidirectional verification, linking independently solved optimization seeds to natural-language problems and executable reconstructions. Static and contract-based checks remain necessary because matching objectives does not prove symbolic equivalence.

C.2 End-to-end engine architecture

  • Matching reference and reconstructed objectives is a strong executable consistency signal, but not proof of symbolic equivalence, so static and contract-based checks remain necessary.Different formulations can share the same optimum on a particular numerical instance.Source: §C.2 · p. 63
  • The engine admits CPT documents through bidirectional verification: it filters independently solved seeds, backtranslates them, reconstructs executable models, and checks status and objective before rendering.A natural-language filter checks numeric coverage, unit consistency, business grounding, and solution leakage.Source: §C.2 · p. 62, Figure 22 · p. 63

C.3 Mathematical seeds and domain contracts · C.4 Prompt-controlled bidirectional synthesis · C.5 Controlled scenarios and CPT rendering

The workflow turns structured optimization seeds into solver-grounded CPT documents through domain contracts, fidelity-controlled bidirectional synthesis, and scenario variation that preserves mathematical structure. Independent validation and executable rendering ensure generated explanations, formulations, and code remain tied to the verified instance.

C.3 Mathematical seeds and domain contracts

  • Seeds encode declared sets, coefficients, objectives, domains, bounds, constraints, artifacts, semantic metadata, and deterministic seeds for regeneration.Semantic metadata includes task family, modeling concepts, difficulty axes, and formulation variant.Source: §C.3 · pp. 63–64
  • Generator profiles specify canonical mathematical signatures, while family contracts preserve shared invariants grounded in seed facts and lineage metadata.Lineage records generator, scenario configuration, prompt revision, validation status, and rendering status outside the training text.Source: §C.3 · p. 64
  • Before language generation, seeds are filtered for solution and constraint activity, objective recomputation, numerical precision, and duplicate signatures.Feasibility alone is insufficient when decisions are all zero, fixed at bounds, or lack genuine trade-offs.Source: §C.3 · p. 64

C.4 Prompt-controlled bidirectional synthesis

  • Backtranslation preserves verified facts and structure while hiding answers and artifacts; forward modeling reconstructs the formulation and executable code using only declared data.Forward modeling must not hard-code the reference answer or infer missing coefficients from the optimum.Source: §C.4 · p. 64
  • Generated self-checks are claims, not validators; static inspection and independent execution remain authoritative for detecting omitted or incorrectly indexed code constraints.The self-check accompanies the reconstructed explanation, formulation, and code, but cannot replace independent validation.Source: §C.4 · p. 64

C.5 Controlled scenarios and CPT rendering

  • Scenario controls vary industry, trigger, organization, planning frame, naming, and units while keeping coefficients, variables, objectives, constraints, and periods invariant.Negative guidance prevents changing model families, such as turning transportation into vehicle routing without vehicles and sequencing.Source: §C.5 · pp. 64–65
  • After executable verification, rendering produces a self-contained CPT document with the problem, model explanation, formulation, solver code, interpreted solution, and validation checks.Style and section order may vary, but mandatory mathematical and validation content remains fixed.Source: §C.5 · p. 65

C.6 Illustrative case I: capacitated facility location

The case illustrates capacitated facility location as a joint site-opening and service-allocation problem, with a feasible optimal plan that balances fixed costs, service costs, demand, and capacity.

C.6 Illustrative case I: capacitated facility location

  • The optimal plan opens B and C, allocates all zone demand, fills B to its 65-unit capacity, and yields an objective of 590.B serves 30 units of Z1 and all 35 of Z2; C serves the remaining 10 units of Z1 and all 25 of Z3. The objective is 95 + 80 + 6(30) + 8(10) + 3(35) + 2(25) = 590.Source: §C.6 · p. 65
  • The example requires jointly choosing opened sites and service allocations: neither B nor C can meet total demand alone, despite C’s low fixed cost and Z3 service cost and B’s low Z2 service cost.This is a fixed-charge trade-off, not a matter of selecting the cheapest coefficient in each demand column.Source: §C.6 · p. 65
  • Applications such as clinic siting, repair depots, and computing hubs preserve three invariants: fixed activation, demand satisfaction, and capacity linking.Forward reconstruction is accepted only if it recovers those invariants and reproduces the reference objective within tolerance.Source: §C.6 · p. 65

C.7 Illustrative case II: executable but structurally wrong models

The case shows that syntactic validity, execution, and an OPTIMAL solver status do not establish that a reconstructed model faithfully represents the intended business problem. It motivates structural and objective validation, with missing data handled without inventing values.

C.7 Illustrative case II: executable but structurally wrong models

  • Table 21 contrasts the faithful formulation with two executable but structurally incorrect reconstructions, showing that execution success alone is insufficient.All three programs can be syntactically valid, executable, and solved to OPTIMAL; that status certifies only the optimum of the constructed program, not its faithfulness to the intended business problem.Source: §C.7 · p. 66, Table 21 · p. 66
  • The example motivates three distinct validation checks: static presence of required terms, family-level structural contracts, and independent objective comparison.When a problem statement lacks a coefficient or bound, regenerate or reject the model, or recover the information from an authoritative source table; do not invent a value to make the model bounded or approach the reference objective.Source: §C.7 · p. 66

D CPT–SFT–Deployment–Evaluation Provenance … D.4 Confidence levels

SLAI T-Rex introduces lightweight provenance across CPT, SFT, weight conversion, deployment, and evaluation, linking scores to their training data, checkpoints, and deployed weights. Stage manifests and a local file-based registry support attribution, reproducibility, and auditability without database dependencies or workflow disruption.

D CPT–SFT–Deployment–Evaluation Provenance

  • The provenance system links training artifacts and evaluation scores to exact checkpoints, supporting improvement attribution, regression analysis, automated tracking, and auditability.It uses per-stage manifests and a local file-based registry without database dependencies or disruption to existing workflows.Source: D CPT–SFT–Deployment–Evaluation… · p. 66

D.1 Design principles

  • Model identity resides in the weight-directory artifact manifest and training lineage, not the service endpoint, allowing one endpoint name to serve multiple unambiguously identified versions.The served model name, IP address, and port identify the inference entry point rather than the model version.Source: §D.1 · p. 66
  • Each stage records actual paths, environment variables, and upstream relationships in a manifest upon completion, preserving execution-time ground truth instead of fragile post-hoc documentation.Captured environment details include CANN version, parallelism configuration, and tokenizer path.Source: §D.1 · p. 66
  • Registration tooling can retroactively link legacy configuration snapshots, checkpoint directories, and converted artifacts into the provenance graph for cross-run comparison and trend analysis.Source: §D.1 · p. 67

D.2 Provenance chain and registry layout

  • The five-stage chain stores manifests at each output and registers copies in a stage-organized JSON registry, while co-located copies preserve discoverable lineage if the registry is unavailable.The stages are CPT, SFT, HF artifact conversion, deployment, and benchmark evaluation.Source: §D.2 · p. 67

D.3 Stage-level linkage mechanism

  • CPT–SFT linkage matches the SFT checkpoint load path to the CPT checkpoint save path, recording linked status with path_match confidence or missing status.This link enables transfer gains to be traced to the specific CPT initialization checkpoint.Source: §D.3 · p. 67
  • Conversion, deployment, and evaluation manifests connect HuggingFace weights and served endpoints to their training lineage, while evaluation can extend benchmark summaries with provenance when given a deployment manifest.Without the deployment manifest reference, evaluation proceeds normally but scores cannot be automatically traced to specific weights or the corresponding dashboard experiment.Source: §D.3 · p. 67

D.4 Confidence levels

  • Each provenance link carries a confidence annotation distinguishing exact manifest-based linkage from path-based or heuristic matching.The confidence levels are summarized in Table 22.Source: §D.4 · p. 67

D.5 Dashboard monitoring and retroactive registration

The provenance registry powers a dashboard that displays experiment runs, metrics, and provenance status. A build script refreshes the unified view and supports retroactive registration of legacy runs.

D.5 Dashboard monitoring and retroactive registration

  • The dashboard maps each training run to its job type, iterations, loss and benchmark scores, plus provenance status and link-confidence annotations.Complete manifest coverage is labeled exact_manifest or path_match; legacy runs are marked missing but remain functional in the dashboard.Source: §D.5 · p. 67
  • A build script aggregates archive and registry manifests into frontend JSON, while registration tooling can backfill legacy runs from configurations, checkpoints, or artifacts.Newly completed runs appear after the next rebuild, enabling active and historical experiments to be queried and audited in one view.Source: §D.5 · p. 68

E Training Schedule and Monitoring Design

The training design balances OR adaptation with broader capabilities through a predefined data curriculum and a monitoring dashboard. The curriculum’s stages and transitions are set before training, while the dashboard supports manual inspection of stability and capability retention.

E Training Schedule and Monitoring Design

  • Overly concentrated OR adaptation can weaken general capabilities, motivating a schedule that balances OR, mathematics, code, and general-domain data.The curriculum, stage boundaries, and transition rules are specified before training.Source: E Training Schedule and Monitoring… · p. 68
  • A monitoring dashboard supports manual inspection of training stability and capability retention.It serves as an observability tool during training.Source: E Training Schedule and Monitoring… · p. 68

E.1 Static Curriculum Learning Design · E.2 Monitoring Dashboard

The sections describe a manually fixed, three-stage data-mixture curriculum and a human-facing dashboard for monitoring training health and retained capabilities.

E.1 Static Curriculum Learning Design

  • A fixed three-stage mixture schedule moves from math-and-code foundations through an OR ramp-up to code and LP-writing contract hardening.The first 30% uses α_math = 0.35, α_code = 0.35, α_gen = 0.20, and α_OR = 0.10; from 30–70%, α_OR rises to 0.40 as α_code falls to 0.15. In the final 70–100%, α_code rises to 0.25, using validated samples.Source: §E.1 · p. 68
  • OR formats are introduced from DP to DT to DPS, with each transition gated by more than 85% execution success on the current format.The threshold is a hard rule, not an adaptive algorithm.Source: §E.1 · p. 68
  • The curriculum is a manually designed default trajectory whose proportions, thresholds, and stage boundaries remain fixed during training.Parameters were set from preliminary runs and the data-mixture ablation in Section 4.4 (Figure 21).Source: §E.1 · p. 68

E.2 Monitoring Dashboard

  • The dashboard logs metrics every 500 training steps as an observability layer for human developers, not a closed-loop controller.Source: §E.2 · p. 69
  • Training-health monitoring tracks loss, gradient norm, NaN counts, and MFU, with gradient spikes or NaN loss prompting manual investigation.Source: §E.2 · p. 69
  • Capability-retention metrics cover OR performance, general code quality, and general-language fluency to monitor domain and general capabilities.OR measures include format-partitioned validation perplexity and code_reward and wl_reward on proxy subsets of NL4OPT and OptiBench; code syntax health checks 200 held-out Python problems, while language perplexity uses a fixed Wikipedia-and-books corpus.Source: §E.2 · p. 69
  • WandB dashboards support manual inspection, with engineers choosing thresholds and actions rather than relying on an automated state machine.A persistent syntax-health decline could prompt pausing OR-centric training, inspecting recent data, or adjusting the mixture; the dashboard design is inspired by Qi et al. (2026), and its instrumentation code is released.Source: §E.2 · p. 69

F Representative Original-Checkpoint Error Cases

The appendix traces original DeepSeek-V4-Flash failures to concrete evaluation examples spanning structural, protocol, semantic, and solver-compatibility errors. The cases show that executable optimization code can still violate the reference formulation or encode the wrong problem.

Case Study: Canonical LP-Graph Mismatch under ORGEval

  • ORGEval failures persist even when LP code executes: BENCH4OPT_1 fails variable-count matching, while BENCH4OPT_12 has matching counts but an incorrect WL graph.The pharmaceutical-blending example shows that ORGEval checks variable and constraint families and their graph structure, not merely whether the generated LP can be written.Source: F Representative Original-Checkpoint… · p. 70

Case Study: Output-Contract and Schema Failures

  • Markdown fences cause a syntax failure on BENCH4OPT_4, while string shift labels used as list indices cause a TypeError on BENCH4OPT_176.These code-only protocol and schema failures motivate SFT on response contracts and schema-safe data-access templates.Source: F Representative Original-Checkpoint… · p. 71

Case Study: Discrete Semantics Error in NL4OPT

  • NL4OPT_15 predicts 28125 against a ground truth of 28250 because continuous workforce variables relax the required integer decision.The problem asks for worker counts, but the model uses continuous variables and therefore yields a fractional optimum.Source: F Representative Original-Checkpoint… · p. 72

Case Study: Loss/Yield-Driven Conservation Error

  • BENCH4OPT_2 Pass@1 predicts 265.5768398591736 versus ground truth 531.1536797183472 by mixing evaporation into both cost and flow conservation.The error treats evaporation as an additive unit cost and also deducts it from conservation; 5-shot and Pass@16 recover the reference value.Source: F Representative Original-Checkpoint… · p. 73

Case Study: Ratio, Unit, and Total-Quantity Coupling Error

  • OPTIBENCH_6 Pass@1 predicts 10000.0 versus ground truth 18000.0, reflecting incorrect coupling of investment-return units and total savings.The case exposes difficulty tracking ratios, units, and total quantities; in 5-shot, the error changes shape and the model can drop the scale variable.Source: F Representative Original-Checkpoint… · p. 74

Case Study: Solver-Compatible Nonlinear Reformulation Failure

  • OPTIBENCH_13 fails because its fourth-order objective is passed directly to Gurobi, which requires a linear or quadratic objective.Later inference settings avoid the build error but still return the wrong distance, indicating a need for solver-aware reformulation or analytic simplification.Source: F Representative Original-Checkpoint… · p. 75

Related papers

Loading 2607.20145v2…