Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs

1.1 Background and Motivation

• MoE models have shown strong performance, but training typically depends on costly high-performance accelerators, motivating Ling-Lite and Ling-Plus for resource-constrained development.Ling-Lite has 16.8B total and 2.75B activated parameters; Ling-Plus has 290B total and 28.8B activated parameters.Source: §1.1 · p. 1

1.2 Computing Environment for Model Training

• Hardware scarcity motivates heterogeneous training, but cross-platform operator and framework compatibility, petabyte-scale resource synchronization, and cost-performance trade-offs remain challenges.The report identifies portability, reliable data and checkpoint movement, and balancing compute efficiency with precision and architecture design as needs.Source: §1.2 · pp. 1, 3

1.3 Optimization for Model Training

• Architecture scaling, unified training frameworks, XPUTimer, EDiT, and storage optimizations target resource efficiency; reported gains include up to 66.1% less training time and 50% lower I/O time overhead.XPUTimer is reported to reduce memory usage by 90%.Source: §1.3 · p. 3
• The proposed anomaly-handling strategy combines real-time, multi-level detection with automated checkpoint recovery, while evaluation uses domain-specific datasets, Flood, and automated analysis.Flood evaluations across clusters have an average deviation below 0.5%.Source: §1.3 · p. 4
• Knowledge graphs and generalized calling instructions generate diverse tool-use data, complemented by rejection sampling, error correction, and self-reflective multi-agent dialogues.Source: §1.3 · p. 4
• Nearly 20%: training one trillion tokens on lower-specification hardware costs around 5.08 million RMB versus 6.35 million RMB on device D, while Ling models perform comparably to similar-size open-source models.Ling-Plus pre-training covered 9 trillion tokens across five hardware configurations.Source: §1.3 · p. 4

1.4 Challenges and Lessons Learned

• Cross-accelerator migration remains challenging: hardware factors and small network changes can affect stability and convergence, while accumulated precision errors can produce outcome differences across platforms.Source: §1.4 · p. 4

2 Infrastructure, Scaling, and Efficiency

• The infrastructure approach responds to growing demand for high-performance accelerators by expanding computational capacity through additional hardware.Source: §2 · p. 4
• DLRover supports workload optimization and migration across platforms, while XPUTimer monitors performance, EDiT provides asynchronous training, PCache and synchronization address I/O, and Flood supports data synthesis and evaluation.Source: §2 · p. 5

2.1 Lightweight Profiler

• XPUTimer selectively monitors critical training code and uses a tracking daemon’s real-time data to diagnose anomalies across the training stack.Its two components are lightweight selective tracing and a diagnostic engine.Source: §2.1 · p. 5, §2.1.1 · p. 5

2.1.1 Lightweight Selective Tracing

• As Figure 4 compares memory use across methods, XPUTimer’s optimized tracing averages approximately 1.5 MB per accelerator per step, about 90% less memory.The tracing design uses asynchronous event management, event reuse, background processing, and compressed logs.Source: §2.1.1 · p. 6, Figure 4 · p. 6

2.1.2 Diagnostic Engine

• O(1) time complexity replaces O(logN) for error localization through call-stack analysis and in-kernel tracing.The diagnostic engine also combines throughput and kernel-launch-latency metrics to detect anomalies across computation, communication, and data loading.Source: §2.1.2 · p. 6

2.2 High-Performance Training Strategy

• EDiT synchronizes parameters layer by layer during forward propagation and overlaps communication with computation through prefetching.Figure 5 compares EDiT with traditional distributed training; Figure 6 illustrates the EDiT pipeline.Source: §2.2 · p. 7
• Pseudo-gradient penalties combine anomaly elimination, norm-weighted averaging, and gradient clipping to suppress loss spikes and limit noisy or oversized updates.Worker pseudo-gradients are tracked with exponential moving averages to identify anomalous workers for exclusion from synchronization.Source: §2.2 · p. 7
• Time-based synchronization lets faster nodes perform more local updates before syncing, addressing fixed-straggler issues in heterogeneous environments.Unlike synchronization after a fixed iteration count, this mechanism triggers synchronization using a time threshold.Source: §2.2 · p. 8
• 66.1% speed-up is reached when the ideal-environment baseline approaches 5.49e−2 step/s as accelerator count increases.In practice, slow-step duration tends to grow with accelerator count or model size, making EDiT’s acceleration more pronounced; Figure 8 compares speeds.Source: §2.2 · p. 8, Figure 8 · p. 8

2.3.1 Distributed Storage System

• PCache is an all-flash distributed file caching system designed for large-scale model training and its performance and scalability requirements.It uses a mixed deployment on compute nodes, with localized processing intended to reduce deployment costs and network latency as cluster capacity expands.Source: §2.3.1 · pp. 8–9
• The cross-cluster storage service is designed for broad hardware compatibility and a cost-performance balance across heterogeneous clusters.The section identifies inconsistent provider storage, hardware compatibility, operational costs, and differing network capabilities as challenges.Source: §2.3.1 · pp. 8–9
• 8 TB/s aggregate throughput is reported for 10,000 accelerators, while a 1,000-accelerator cluster achieves 1 TB/s.For single-client large-file writes, throughput is 3–4 GB/s in single-threaded scenarios and 20–30 GB/s in multi-threaded scenarios.Source: §2.3.1 · p. 9
• 50% lower checkpoint-writing latency and 60% lower peak memory were reported for a 5,000-accelerator MoE training task.The optimization distributes data-parallel group checkpoint writing across physical nodes rather than concentrating rank_0 writers.Source: §2.3.1 · pp. 9–10, Table 2 · p. 10

2.3.2 Cross-Cluster Synchronization Mechanism.

• Babel synchronizes large unstructured datasets and high-frequency checkpoints across clusters and regions, using adaptive sharding, metadata prefetching, and multidimensional verification.Its stated scope includes petabyte-scale datasets, billions of files, and frequently updated training-state files.Source: §2.3.2 · p. 10
• 190 million files were listed in approximately ten minutes with parallel metadata prefetching, versus over six hours serially, a roughly 36-fold performance increase.The mechanism uses concurrent OSS List operations and scheduling algorithms; Figure 10 depicts the mechanism.Source: §2.3.2 · p. 10, Figure 10 · p. 10
• Content-sampling CRC verification reduces verification time for a 100GB file to approximately three seconds while reducing CPU consumption without compromising transmission accuracy.Babel provides runtime and post-transfer verification modes.Source: §2.3.2 · pp. 10–11

2.4 High-Efficiency Offline Inference Framework

• Flood adopts fully pipeline-parallel architecture for offline inference to reduce communication overhead, which can exceed half of execution time for tensor parallelism without high-speed interconnects.The paper contrasts this with mainstream frameworks designed for online inference under latency constraints.Source: §2.4 · p. 11
• A many-to-one process-to-accelerator mapping and separate streams are used to reduce inter-process communication and keep accelerator resources utilized.For a single node with eight accelerators, the example uses nine processes, with one waiting for the first pipeline-stage accelerator.Source: §2.4 · p. 11
• The segment cache allocates KV-cache in contiguous memory, reserving space per request for prompt and output to enable larger block sizes.Figure 11 illustrates the segment cache; conservative initial allocation can be extended, appended, or made to wait when output exceeds the allocated segment.Source: §2.4 · pp. 11–12, Figure 11 · p. 11
• The segment cache supports prefix caching by storing a shared batch-request prefix in one or more segments.The stated benchmark compares Flood with vLLM on shareGPT AI using generated tokens per second; Table 3 is identified as the comparison.Source: §2.4 · p. 12, Table 3 · p. 12

3.1 Pre-Training Data

• The approximately 9-trillion-token corpus combines Chinese, English, code, and other data, refined through tailored cleaning, quality assessment, labeled selection, and data-mixture experiments.The corpus comprises 1 trillion Chinese tokens, 5.5 trillion English tokens, and 2.5 trillion code tokens.Source: §3.1 · pp. 12–13

3.2.1 Fine-Grained Experts

• Ling replaces dense FFNs with routed experts, increases expert count while reducing each expert’s intermediate size at equivalent total capacity, and adds an all-token shared expert for general ability.The fine-grained design is intended to increase expert specialization and represent a wider range of knowledge.Source: §3.2 · p. 13, §3.2.1 · p. 13

3.2.2 Expert Routing

• Training uses dropless routing, balance and z-losses, and stochastic warmup that blends learned and random logits to balance initial expert activation and reduce overload, collapse, and out-of-memory risks.During warmup, the routing distribution gradually shifts toward learned logits.Source: §3.2.2 · pp. 13–14

3.2.3 NormHead

• L2-normalizing LM-head weights addresses output-norm instability during loss spikes, and the experiments report significantly improved training stability.The report links output-norm instability to convergence and overall model performance.Source: §3.2.3 · p. 14

3.3.1 Scaling Laws for Hyper-parameters

• For a given compute budget, neither MoE architecture nor auxiliary losses significantly changed optimal batch size or learning rate, which were primarily determined by compute budget.The fitted batch-size and learning-rate scaling behaviors were consistent with those observed in dense models.Source: §3.3.1 · pp. 14–15

3.3.2 Scaling Laws for Model Performance

• MoE consistently achieves lower training loss than dense models at equal compute, with an average efficiency lever of approximately 3x that grows beyond 3.5 at 1e24 FLOPs.The reported lever is approximately 3 at 1e21 FLOPs.Source: Figure 13 · p. 15, §3.3.2 · pp. 15–16

3.4.1 Initial Pre-Training

• The first 9T-token stage used learning-rate and batch-size warmup; later stages extended context to 16K over 150B tokens, then annealed on curated high-quality data.During initial pre-training, the team monitored loss, gradients, router distributions, and benchmarks while adjusting the data mix and deduplicating samples online.Source: §3.4.1 · p. 16, §3.4.2 · p. 16, §3.4.3 · p. 16

3.4.4 Skip loss spikes and Sample retry mechanism

• Loss-spike handling skips affected updates, retries their data in later batches, and lowers the learning rate if spikes persist; the authors report improved benchmark metrics.The comparison of train-loss curves before and after applying the strategy is shown in Figure 14.Source: §3.4.4 · p. 16, Figure 14 · p. 17

4.1.1 Quality Assurance

• Rule-based checks and an LLM judge filter reasoning and non-reasoning synthetic responses, and the authors report eliminating many incorrect responses while minimizing rejection of correct ones.Code solutions undergo execution and synthetic-test validation; math answers and intermediate reasoning are checked by executing generated Python; logical solutions use majority voting.Source: §4.1.1 · p. 18

4.1.2 Data Redundancy

• Semantic deduplication removes highly similar instruction data, and removing approximately 10% to 20% of the most similar examples had no adverse effect on core capabilities.A text-embedding model maps prompts and responses into a vector space for cosine-similarity filtering.Source: §4.1.2 · p. 18

4.2 Direct Preference Optimization

• DPO proceeds from preference alignment for authenticity, relevance, harmlessness, and instruction following to robustness optimization for reasoning across math and coding tasks.Robustness optimization uses Best-of-N/Worst-of-N response selection and majority voting; it adds NLL regularization with weight 0.05.Source: §4.2 · p. 18
• A 3.7-fold speed increase comes from DPO data packing, while format-focused DPO reduces formatting-error penalties.The format phase masks non-format content in the DPO loss so valid reasoning in rejected responses is not penalized.Source: §4.2 · p. 19
• Length-regularized DPO shortened responses but did not improve overall performance and underperformed vanilla DPO particularly on math and coding tasks.The authors hypothesize that penalizing length can suppress the detailed responses complex problems require; repeated prompts with varied responses also slightly declined versus the original dataset.Source: §4.2 · p. 19

4.3 Long Context

• Ling-Plus achieves nearly perfect Needle in A Haystack performance across context lengths up to 64K.The model is trained for up to 16K-token contexts; progressive fine-tuning also benefits scaling from 16K to 64K.Source: §4.3.1 · p. 20, §4.3.2 · p. 20

4.3.1 Synthesis of Long-Context Instruction Data

• Long-context instruction data uses extended Chinese and English documents to synthesize retrieval, summarization, QA, and reasoning tasks, including single-needle, multi-needle, and multi-hop retrieval.Critical information is inserted into document middles to address the “lost in the middle” phenomenon Liu et al. [2024].Source: §4.3.1 · pp. 19–20

4.3.2 Progressive Fine-tuning Strategy

• Progressive fine-tuning first uses contexts of at most 4K tokens, then introduces 4K–16K contexts at a 95:5 short-to-long sample ratio; after long-context adaptation, short-context RL data improves preference alignment without degrading long-context performance.The progressive approach also yields greater benefits when scaling to longer contexts, such as 16K→64K tokens.Source: §4.3.2 · p. 20, §4.3.3 · p. 20

5.1.1 Evaluation Benchmarks

• The evaluation suite covers English, Chinese, math, and code, using perplexity-based metrics for language benchmarks and generation-based evaluation for math, code, and BBH.Ling-Lite-Base and Ling-Plus-Base are compared with open-source baselines of similar parameter scales.Source: §5.1.1 · p. 21, §5.1.3 · p. 25

5.1.2 Benchmarks Optimization

• Perplexity-based evaluation predicts option content rather than option labels, while generation-based evaluation uses guided prompts and stopping criteria to better reflect base-model ability.For math and code, prompt changes include refined few-shot examples, lightweight templates, and early stopping; code prompts specify task requirements and provide appropriate prefixes.Source: §5.1.2 · pp. 22–23
• Improved benchmarks reduce fluctuations in MMLU-Pro and GSM8K evaluation metrics as small models train, reflecting steadier capability changes.The evaluation changes target early-stage instability and weak instruction following in base models; Figure 18 compares improved and original benchmarks.Source: §5.1.2 · pp. 22–23, Figure 18 · p. 22
• Evaluation results are mapped to training-corpus ability dimensions to help identify problematic data segments when abnormal results appear after particular token ranges.Figure 19 illustrates using evaluation to guide the training process.Source: §5.1.2 · p. 23, Figure 19 · p. 23

5.1.4 Result Analysis

• Ling-Lite is close to Qwen2.5-7B overall and exceeds Llama3.1-8B and Mistral-7B v0.3 on code and math; Ling-Plus is comparable to Qwen2.5-72B in code, math, and Chinese.Ling-Plus is slightly below Qwen2.5-72B on English benchmarks but exceeds DeepSeek-V2-Base and Llama3.1-70B-Base; the comparisons are shown in Tables 5 and 6.Source: Table 5 · p. 24, Table 6 · p. 24, §5.1.4 · p. 25

5.2.1 Evaluation Benchmarks

• The instructed-model evaluation adds English and Chinese understanding, code, math, tool-use, and open-ended-generation benchmarks, comparing each Ling model with instructed baselines of corresponding scale.The benchmark set includes IFEval, GPQA-Diamond, SimpleQA, C-SimpleQA, MultiPL-E, LiveCodeBench, AIME, BFCL, Nexus, T-eval, and Arena-Hard.Source: §5.2.1 · p. 25, §5.2.2 · p. 25

5.2.3 Result Analysis

• Ling-Lite and Ling-Plus achieve the best results in most tool-use benchmark cases, especially BFCL-v2 and T-eval, while Nexus performance is comparable to baselines except for a four-point gap to Llama3.1-8B.The results are presented across five task groups, including English and Chinese language understanding, code, math, tool use, and open-ended generation.Source: §5.2.3 · p. 26, Table 8 · p. 27
• Ling-Lite is comparable to Qwen2.5-7B-Instruct on MMLU and Ling-Plus is comparable to DeepSeek-V2.5-Chat and Qwen2.5-72B-Instruct; both are comparable to baselines on ARC-Challenge.On Chinese C-Eval and CMMLU, both Ling models perform slightly better than DeepSeek and comparably to Qwen; DeepSeek performs better on C-SimpleQA.Source: §5.2.3 · p. 26
• Ling-Lite is comparable to Qwen2.5-7B on math and code, while Ling-Plus performs better than DeepSeek-V2.5 and closely approximates Qwen2.5-72B.On Arena-Hard, Ling-Lite outperforms Llama3.1-8B and Mistral-7B-v0.3, while Ling-Plus is comparable to DeepSeek-V2.5.Source: §5.2.3 · pp. 26–27
• On MATH, zero-shot prompting performs significantly better than few-shot prompting for the instructed models.The authors advise caution when using few-shot settings for math tasks.Source: §5.2.3 · p. 27
• Ling-Plus achieves almost identical benchmark results on Device-A and Device-D AI accelerators.The comparison is reported in Table 8.Source: §5.2.3 · p. 27, Table 8 · p. 27

5.3.1 Evaluation Benchamarks

• Arena Safety uses 803 risky questions and Llama-Guard3 ratings, while Cvalues uses 1,711 multiple-choice questions on responsibility in a Chinese context.Arena Safety questions are sampled from lmsys-chat-1m and identified as risky by the OpenAI moderation API.Source: §5.3.1 · p. 28
• Xstest and Orbench-Hard-1k test over-refusal on 250 and 1,000 non-risky questions, respectively, with GPT-4o judging refusals.The section motivates these benchmarks by noting that harmlessness improvements can reduce helpfulness.Source: §5.3.1 · p. 28
• Safety measures the proportion of safe responses, while false rejection measures the proportion of non-refusal responses; higher is better for both.Source: §5.3.1 · p. 28

5.3.2 Results analysis

• Ling-Plus achieves the best average safety and false-refusal results, while Qwen2.5-72B-Instruct also stands out in safety.The DeepSeek series has the least false refusal but lower safety on risky lmsys-chat-1m questions.Source: §5.3.2 · p. 28

6 Bitter Lessons

• The report compiles frequent LLM-training problems and strategies to help researchers and practitioners address technical difficulties in a resource-intensive process.Source: §6 · p. 28

6.1 Training Stability

• Retrying or skipping wide loss spikes, reducing the learning rate when they persist, and using HeadNorm and router zloss mitigated instability; balance loss supported stable training of a model with hundreds of billions of parameters.The authors report no observed loss divergence, wide loss spikes, or disruptions in expert routing balance.Source: §6.1 · p. 28

6.2 Cross-Platform Alignment

• Before large-scale migration, the authors ran preparatory experiments to validate basic operations and communication algorithms across platforms, allowing for minor numerical precision errors.Source: §6.2 · p. 29
• Basic-operation checks alone did not prevent divergent loss convergence; alignment therefore extended to framework modules and both forward and backward computations.The work addressed tensor-parallelism, auxiliary-loss, communication, and router-gradient discrepancies; the authors say full-pass alignment is imperative for new platforms or frameworks.Source: §6.2 · p. 29

• A 300B MoE LLM trained on lower-performance devices achieved performance comparable to similarly scaled dense and MoE models, with approximately 20% lower pre-training compute costs.Source: §7 · p. 30
• Ling-Lite and Ling-Plus target lower training costs through architecture, framework, and storage optimizations, alongside improvements to anomaly handling and evaluation.Source: §7 · p. 30
• The report also describes advancements in tool-use ability and says a Ling-series coder model is planned for future release.Source: §7 · p. 30