Source-linked AI summary
TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI
Yuheng Zhang, Yizhao Wang, Da Zhu, Hua Zhou, Yue He, Jiahui Hu, Shaman Tang, Hanlin Chen, Yuhua Wei, Anhua Liu, Shuang Su, Rui Xin, MingYuan Wang, MingHao Li, HaoJie Yang, Siqi Liu, Jianlei Zheng, WeiChao Huang, Qiman Wu, Hang Zhang, HongGou Yang, Xianming Liu
TL;DR
Physical AI foundation models need broad capability and extended-context processing without incurring dense-model inference costs and latency. Turing-20B-A2B combines token-adaptive Quantile Routing, hybrid Lightning/ full attention, continued pretraining, and deployment-time capacity control. At the base-model stage, it exceeds Qwen3-8B and approaches Qwen3.5-9B while maintaining strong long-context performance and favorable prefill-latency scaling.
Problem
Physical AI requires broad reasoning and long-context modeling, while larger dense models increase per-token computation, cost, and latency.
Method
Turing-20B-A2B uses sparse token-adaptive Quantile Routing with hybrid attention, progressive context extension, and capacity-constrained prompt-prefill execution.
Results
At the base-model stage, Turing-20B-A2B exceeds Qwen3-8B and approaches Qwen3.5-9B while maintaining strong long-context performance and favorable prefill-latency scaling.
Takeaways & Limitations
The reported results demonstrate a balance among model capability, long-context scalability, and inference efficiency under a compact active-parameter budget.
Takeaways & Limitations
The model has so far been primarily validated as a base language model, with its post-training capabilities still under exploration.
Abstract
from arXiv · showhide
We present Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates approximately 2B parameters per token, designed for long-context and latency-sensitive physical AI applications. The model adopts Quantile Routing in a dynamic top-k configuration, enabling token-adaptive expert allocation while maintaining balanced expert utilization and a controlled average compute budget. During deployment, we further apply capacity-constrained routing to prompt prefill for more regular and efficient expert execution, while retaining dropless routing during pretraining. Turing-20B-A2B also employs a hybrid attention architecture that combines Lightning Attention with a small number of full-attention layers for efficient long-context modeling. The model is pretrained with a progressive three-stage curriculum and extended to a native context length of 128K through continued pretraining, with further inference-time extension to 512K using YaRN. Despite its compact active-parameter budget, Turing-20B-A2B achieves, at the base-model stage, overall general capability exceeding Qwen3-8B Base and approaching Qwen3.5-9B Base, while maintaining strong long-context performance and favorable prefill-latency scaling. These results demonstrate an effective balance among model capability, long-context scalability, and practical inference efficiency.
1 Introduction
Physical AI foundation models must combine broad reasoning with long-context processing while controlling inference cost and latency. Turing-20B-A2B addresses this tension with sparse, adaptive routing and hybrid attention, achieving strong capability and favorable long-context efficiency under a compact active-parameter budget.
- Motivation: Physical AI applications require broad world knowledge, strong reasoning, and extended histories of observations, states, and actions.These requirements support perception, reasoning, decision-making, and prediction of action-conditioned environment dynamics.
- Motivation: Conventional top-k MoE routing can create imbalanced expert loads and irregular execution, limiting inference speedups from computational sparsity.The resulting gap is especially relevant for latency-sensitive physical AI systems.
- Approach: Turing-20B-A2B combines dynamic top-k Quantile Routing, Lightning Attention-dominated hybrid attention, continued pretraining, and capacity-constrained prompt-prefill execution.Quantile Routing adapts the number of experts per token while targeting balanced utilization and a controlled average compute budget.
- Results: At the base-model stage, Turing-20B-A2B outperforms Qwen3-8B Base overall and approaches Qwen3.5-9B Base while activating approximately 2B parameters per token.It is particularly strong on knowledge and math & STEM benchmarks while remaining competitive on reasoning and long-context evaluations.
- Results: From 32K to 128K on NVIDIA H800 GPUs with FP16 precision, its prefill-latency advantage over representative Qwen3 and Qwen3.5 baselines becomes increasingly pronounced.The reported results support a balance among capability, long-context scalability, and inference efficiency.
- Approach: The decoder stack alternates groups of five Lightning Attention layers with one full-attention layer to support efficient long-context modeling with periodic global interactions.Figure 2 presents this architectural configuration and summarizes the main model settings.
2 Model Architecture
Turing-20B-A2B combines hybrid attention with sparse MoE feed-forward modules to balance long-context efficiency, model capacity, and compact per-token computation. Its Quantile Routing dynamically allocates experts while controlling average routing and balancing expert utilization.
- Sparse Mixture-of-Experts: 20B total parameters and approximately 2B activated parameters per token combine high capacity with a compact computation budget.The model uses 256 routed experts and one shared expert in its sparse MoE modules.
- Context Length: The model supports a native 128K-token context and inference-time extension to 512K using YaRN.These context lengths are enabled alongside the hybrid-attention design.
- Hybrid Attention: Five Lightning Attention layers followed by one full-attention layer reduce long-context computational growth while retaining periodic global token interactions.The 24-layer decoder contains 20 Lightning Attention layers and four full-attention layers.
- Quantile Routing: Expert-specific thresholds support balanced expert utilization and average-budget control through a unified quantile-tracking process.Thresholds are updated using expert-load statistics and router-score distributions.
- Quantile Routing: Dynamic top-k Quantile Routing assigns different numbers of routed experts to tokens while keeping the average routing budget near a target.The target average routing budget is approximately eight routed experts per token.
- Capacity-Constrained Execution: Capacity-constrained routing bounds each expert’s prompt-prefill workload by retaining only the highest-scoring assignments when capacity is exceeded.The shared expert remains available to every token as a fallback when routed assignments are removed.
3 Training Recipe
Turing-20B-A2B is trained with a three-stage curriculum that shifts from broad knowledge coverage toward capability-dense and quality-focused data. After 4K pretraining, progressive continued pretraining extends the native context to 128K, with inference-time extension to 512K using YaRN.
- Pretraining Data: Stage 1 uses broad natural corpora, while later stages increase filtered, capability-dense, and in-house mathematics, code, and knowledge data.The data curriculum transitions from broad natural coverage toward higher-quality and capability-dense training signals.
- Main Pretraining Curriculum: The three-stage curriculum progresses from knowledge foundation to capability enhancement and quality annealing while keeping the architecture and objective unchanged.Stage 1 establishes broad coverage, Stage 2 emphasizes capability-dense data, and Stage 3 focuses on quality-oriented annealing.
- Optimization and Schedule: 770K optimization steps comprise Stage 1, followed by 180K steps each in Stages 2 and 3, with approximately 6.5T tokens in Stage 1 and 1.5T tokens in each later stage.Training uses a 4,096-token sequence length and approximately 8.4M tokens per optimization step.
- Optimization and Schedule: Stage 1 holds a 4 × 10^-4 learning rate after warm-up, Stage 2 uses 2 × 10^-4, and Stage 3 applies cosine decay from 2 × 10^-4 to 10^-6.A document-level attention mask is introduced in Stage 2 and retained in Stage 3.
- Long-Context Training: Continued pretraining progressively extends the context window from 4K to 32K and then 128K rather than directly training at the target length.The 128K checkpoint is extended to 512K with YaRN without additional parameter updates.
4 Evaluation
Evaluation shows that Turing-20B-A2B is competitive with Qwen base models while using approximately 2B active parameters per token. Its long-context performance and prefill latency scale favorably as sequence length increases, supported by progressive context training and hybrid attention.
- General Capability: Turing-20B-A2B reaches performance comparable to Qwen3-8B while activating approximately 2B parameters per token.Its capability profile is balanced, with particular strengths in knowledge-intensive, reasoning, and mathematical tasks.
- General Capability: Turing-20B-A2B remains broadly competitive with Qwen3.5 base models, although Qwen3.5-9B achieves higher scores on several benchmarks.Qwen3.5 base models may activate explicit thinking during evaluation, producing substantially longer generations on reasoning-intensive tasks.
- Long-Context Ability: Turing-20B-A2B maintains a RULER score above 90 through 64K and retains strong performance at the trained 128K context limit.Its performance gap over Qwen3 base models becomes increasingly pronounced as sequence length grows.
- Long-Context Ability: Against Qwen3.5, Turing-20B-A2B surpasses Qwen3.5-4B from 64K onward and approaches Qwen3.5-9B at 256K.The model exhibits substantially slower performance degradation as context length increases.
- Model Efficiency: 1.2×, 1.6×, and 2.2× relative prefill speedups over Qwen3 baselines occur at 32K, 64K, and 128K tokens, respectively.Latency differences are modest at short contexts but become increasingly pronounced as sequence length grows.
5 Ablation Studies
The ablations show that Quantile Routing improves expert-load stability and early training performance under the same average routed-expert budget, while capacity-constrained prefill trades limited capability changes for faster, more regular execution.
- MoE Routing Strategy: The routing comparison holds architecture, data, optimization, and average routed-expert budget constant, isolating the routing mechanism.Both variants were trained from scratch for approximately 100B tokens under the same configuration.
- MoE Routing Strategy: Quantile Routing maintains lower MaxVio over nearly the entire training trajectory than Loss-Free Routing.It is especially more stable during early training, whereas Loss-Free Routing shows an early load-violation spike and gradual recovery.
- MoE Routing Strategy: Quantile Routing outperforms Loss-Free Routing on seven of eight early-training benchmarks.GSM8K is the sole benchmark favoring Loss-Free Routing, with particularly clear gains for ARC-C, MBPP, and HellaSwag.
- MoE Routing Strategy: Quantile Routing assigns different numbers of routed experts to tokens according to contextual representations while maintaining an average budget of eight.Content-bearing tokens generally receive more experts than punctuation and frequent function words, and identical surface tokens can receive different allocations.
- Expert Capacity for Efficient Prefill: Capacity-constrained prefill remains broadly competitive with dropless routing but introduces a measurable, limited capability trade-off.The constrained variant improves on BBH and GPQA, while dropless routing scores higher on MMLU, CMMLU, MMLU-Pro, and GSM8K.
- Expert Capacity for Efficient Prefill: 1.53× average speedup is achieved by Turing MoE over conventional dropless top-k MoE across evaluated context lengths.Latency reductions persist from 16K to 256K and become more pronounced at 128K and 256K.
6 Applications and Practical Impact
Turing-20B-A2B targets physical AI workloads by combining token-adaptive computation, regularized prefill execution, and hardware-oriented architectural choices for long-context, latency-sensitive systems.
- Practical Impact: Quantile Routing gives different tokens different amounts of expert computation while maintaining balanced utilization and a controlled average compute budget.Capacity control during prompt prefill bounds per-expert workloads and makes execution more regular and predictable.
- Hardware–Model Co-Design: The architecture favors simple, regular, widely supported computation patterns across edge and latency-sensitive devices.This motivates the Lightning-Attention-based hybrid backbone, Dynamic Tanh normalization, and fine-grained sparse experts.
- Applications: The model is positioned for autonomous driving, intelligent-cockpit, integrated driving-and-parking, and embodied platforms.These applications share the need to process extended histories of observations, states, instructions, and actions under strict latency and compute constraints.
7 Conclusion and Future Work
The report concludes that Turing-20B-A2B balances capability, long-context scalability, and inference efficiency for physical AI, while identifying post-training and broader downstream evaluation as future work.
- Conclusion: Turing-20B-A2B exceeds Qwen3-8B Base and approaches Qwen3.5-9B Base in base-model capability while activating approximately 2B parameters per token.It also supports 128K native context and effective inference up to 512K using YaRN.
- Conclusion: The report characterizes the model as an effective balance among capability, long-context scalability, and practical inference efficiency.This conclusion is scoped to latency-sensitive physical AI systems.
- Future Work: The model has primarily been validated as a base language model, with post-training capabilities still under exploration.Planned work includes supervised fine-tuning, reinforcement learning, and broader downstream evaluation.
A Base Model Evaluation Details
The evaluation uses standardized benchmark-specific datasets, prompts, demonstrations, generation settings, answer extraction, and scoring procedures for base-model comparison.
- Evaluation Protocol: All evaluated models use identical dataset versions, splits, prompts, demonstrations, generation configurations, extraction, and scoring procedures.This includes consistent task-specific answer extraction for HellaSwag, WinoGrande, GPQA, and DROP.
- Evaluation Protocol: The general capability suite covers knowledge, reasoning, mathematics, and STEM across thirteen pretrained-base-model benchmarks.The benchmark set includes MMLU, MMLU-Redux, CMMLU, C-Eval, MMLU-Pro, BBH, DROP, WinoGrande, HellaSwag, ARC-C, GPQA, GSM8K, and MATH.
- Knowledge Benchmarks: MMLU uses 14,042 test questions across 57 subjects with five fixed development examples per subject.Accuracy is computed after extracting the first valid A–D option.
- Knowledge Benchmarks: MMLU-Redux 2.0 covers 5,330 questions across 57 subjects using five-shot prompting and accuracy scoring.The prompting format matches the stated MMLU evaluation procedure.
- Knowledge Benchmarks: CMMLU and C-Eval use Chinese multiple-choice prompts with five fixed development examples per subject.Their evaluation sets contain 11,582 and 1,346 questions, respectively.
- Reasoning Benchmarks: MMLU-Pro evaluates 12,032 test questions across 14 categories with five chain-of-thought validation examples.The final option is extracted after step-by-step reasoning.
B Model Efficiency Evaluation Details
The model-efficiency evaluation measures representative decoder-layer forward latency under a controlled single-GPU protocol and aggregates layer types according to each model’s architecture.
- Measurement protocol: Prefill latency is measured on a single NVIDIA H800 GPU using FP16 precision, batch size 1, and optimized attention implementations.The benchmark environment includes CUDA 12.9, PyTorch, Triton, Flash Linear Attention, FlashAttention-2, and causal-conv1d.
- Measurement protocol: Compilation and shape-specialization overheads are excluded from reported latency measurements.Each layer is compiled before timing, followed by warm-up and untimed forwards at evaluated sequence lengths.
- Measurement protocol: The benchmark reports mean latency from timed forward passes using a 100 ms warm-up and a 5,000 ms measurement period.CUDA-event timing and synchronization are used, with cache clearing between repetitions and sequence lengths.
- Layer composition: Turing-20B-A2B has 24 decoder layers: 20 Lightning Attention layers and four full-attention layers.Its Lightning Attention uses a custom Triton implementation, while full-attention layers use FlashAttention-2.
- Layer composition: Model-level prefill latency is estimated by aggregating representative decoder-layer latencies according to each model’s layer multiplicities.Qwen baselines use multiplicity-based aggregation, including separate treatment of linear- and full-attention layers where applicable.
C MoE Prefill Latency Benchmark Details
The MoE prefill benchmark compares complete routing and expert-execution paths under matched single-GPU conditions, contrasting Turing’s capacity-constrained Quantile Routing with dropless top-k routing.
- Benchmark scope: The benchmark measures complete isolated MoE-module forward latency, including routing, expert computation, weighting, and output aggregation.Attention and normalization overhead are excluded, and the measured module is the first full MoE layer.
- Benchmark setup: Measurements run on a single NVIDIA H800 GPU with batch size 1 and FP16 precision for sequence lengths from 16K to 256K.Inputs have shape [1, L, 2048] and are generated directly on the GPU.
- Benchmark setup: Each MoE variant uses 256 routed experts, a nominal routing budget of 8 experts per token, and routed-expert intermediate dimension 512.Both variants also share a 2,048-dimensional hidden state and a shared expert with intermediate dimension 2,048.
- Routing and execution: Turing MoE applies Quantile Routing with sigmoid scores, expert-specific thresholds, and capacity factor γ = 1.25.For each expert, assignments are ranked and only the highest-scoring capacity-bounded assignments are retained before batched expert computation.
- Routing and execution: The baseline retains all k selected token–expert assignments, producing variable per-expert workloads under dropless top-k execution.Fused kernels group assignments by expert and support variable workloads without naive expert-by-expert loops.
- Controlled comparison: Both MoE variants use identical model dimensions, expert configurations, input shapes, precision, compilation settings, and timing protocols.The comparison differs only in routing and expert execution.