ERNIE 5.0 Technical Report

1 Introduction

• Existing systems often keep multimodal outputs text-centric, while late-fusion modality-specific generators decouple generation from understanding and can trade off multimodal integration against language performance (Xu et al., 2025a; Seedream et al., 2025).The introduction identifies a unified paradigm that natively supports both multimodal understanding and generation while preserving unimodal capabilities as an open challenge.Source: §1 · p. 3
• ERNIE 5.0 trains text, image, audio, and video from scratch under shared-token Next-Group-of-Tokens prediction, using ultra-sparse MoE with modality-agnostic routing.The shared expert pool supports differentiation and collaboration among modality-specialized behaviors.Source: §1 · p. 3, Figure 1 · p. 4
• Elastic training samples sub-models of varying depth, width, and routing sparsity in the same run, enabling smaller, faster variants suited to diverse hardware, memory, and latency constraints.The sampled sub-models and full model are optimized together in one backpropagation process under the same autoregressive objective.Source: §1 · p. 3
• Unified multimodal reinforcement learning uses an unbiased replay buffer, multi-granularity importance sampling, positive-sample masking, and adaptive hint-based RL to address sampling bias, sparse rewards, and entropy collapse.A unified verifier system and scalable techniques support stable and efficient post-training.Source: §1 · p. 3
• Training infrastructure combines hybrid parallelism and fine-grained memory control with separate tokenizer GPU nodes, FlashMask for heterogeneous vision masks, and disaggregated RL coordination.These components support trillion-parameter MoE training and high-throughput, numerically consistent post-training.Source: §1 · pp. 3–4
• ERNIE 5.0 matches or outperforms specialized baselines; 25% inference top-k yields >15% decoding speedup, and elastic models preserve near-full performance with 53.7% activated and 35.8% total parameters.Experts specialize primarily by task requirements rather than modality boundaries.Source: §1 · p. 4

2 Architecture

• ERNIE 5.0 integrates language, image, video, and audio in one ultra-sparse MoE autoregressive framework for multimodal understanding and generation.The framework is presented in Figure 1.Source: §2 · p. 4
• A shared backbone supports unified sequence modeling, while visual and audio tokenizers convert multimodal inputs into a unified token sequence.Source: §2 · p. 4
• All modalities use a shared Next-Group-of-Tokens Prediction objective, enabling deep cross-modal interactions with end-to-end optimization.Source: §2 · p. 4

2.1 Unified Autoregressive Backbone with Ultra-Sparse Mixture-of-Experts

• ERNIE 5.0 projects text, image, video, and audio into a shared token space and trains them from scratch as one sequence under Next-Group-of-Tokens Prediction.This unified objective addresses the optimization challenges of heterogeneous modalities and supports deep token-level multimodal interactions.Source: §2.1 · pp. 4–5
• Multi-Token Prediction (MTP; Gloeckle et al., 2024; Liu et al., 2024) augments generation, while vision and audio use group-of-tokens prediction aligned with the text autoregressive objective.Vision uses Next-Frame-and-Scale Prediction (Ji et al., 2026), and audio uses Next-Codec Prediction to capture temporal and spectral structure.Source: §2.1 · p. 5
• Modality-agnostic routing dispatches unified token representations to shared experts, and an ultra-sparse fine-grained MoE keeps activation below 3%.Compared with modality-isolated routing in previous models (ERNIE Team, 2025), shared routing is designed to promote cross-modal generalization and single-modality performance; auxiliary-loss-free load balancing stabilizes trillion-parameter-scale training.Source: §2.1 · p. 5
• The unified backbone is designed to combine semantic-level understanding with fine-grained perceptual detail, so one model can support perception, reasoning, and creative generation.Semantic signals guide global generative consistency, while generative training strengthens fine-grained perception and detail-sensitive reasoning.Source: §2.1 · p. 5

2.2.1 Vision Tokenization

• A causal 2D multi-scale image tokenizer is inflated into a causal 3D video tokenizer, unifying visual tokenization while preserving spatial capability and adding temporal perception.Adversarial and semantic-regularization losses support token quality and training stability; progressively switching from low-bit to higher-bit tokenizers aims to smooth optimization and improve visual generation quality.Source: §2.2.1 · pp. 5–6

2.2.2 Visual Understanding with Dual-Path Hybrid Representation

• Attention-based patch aggregation of CNN and ViT features outperforms CNN-only and ViT-only baselines across benchmarks, with pronounced gains in document, chart, and general visual understanding.It attends jointly over aligned local patches and semantic features, then pools them into a compact representation that also supports fine-grained visual generation. Naive MLP-based fusion is inadequate.Source: §2.2.2 · pp. 6–7

2.2.3 Visual Generation with Next-Frame-and-Scale Prediction

• NFSP predicts images across spatial scales and videos frame by frame, separating low-to-high-resolution intra-frame structure from inter-frame temporal dependencies.Uni-RoPE encodes frame and spatial positions; training corrupts historical tokens for self-correction, while video generation adds windowed attention and frame masking. A separately trained diffusion refiner enhances high-resolution details while preserving backbone semantics and structure.Source: Figure 2 · p. 6, §2.2.3 · p. 7

2.3 Audio Modeling

• Audio modeling uses hierarchical discrete codec tokens to represent semantics and fine-grained acoustic details within a unified autoregressive framework.Depth-wise prediction across codec dimensions avoids the sequence length caused by flattening multi-codebook tokens.Source: §2.3 · pp. 7–8, Figure 3 · p. 8

2.3.1 Audio Tokenization

• At 12.5 Hz, RVQ tokenization assigns the first token high-level semantics and residual tokens progressively finer acoustic details.The first token is aligned to pooled Whisper encoder outputs to capture linguistic and phonetic cues; residual tokens preserve characteristics such as timbre and prosody.Source: §2.3.1 · p. 8

2.3.2 Audio Understanding and Generation with Next-Codec Prediction

• For understanding, level-specific embeddings of the audio codes are summed and processed alongside text tokens by the autoregressive backbone.The additive representation combines complementary information from coarse to fine residual levels.Source: §2.3.2 · p. 8, Figure 3 · p. 8
• For generation, Next-Codec Prediction sequentially predicts semantic and residual codes, feeding each code embedding back to condition the next level.Training uses teacher forcing; after all codes are predicted, an audio decoder produces waveforms, and speech synthesis can use a speaker embedding to control voice timbre.Source: §2.3.2 · pp. 8–9, Figure 3 · p. 8

3.1 Pre-Training Data

• ERNIE 5.0 trains from the outset on text, images, videos, and audio, using paired and interleaved multimodal data to learn cross-modal semantic relationships.The corpus is filtered for quality and safety, deduplicated, and decontaminated, and comprises trillions of text tokens and multimodal instances.Source: §3.1 · p. 9

3.3 Once-For-All with Elastic Training

• The elastic super-network varies active layers, available experts, and experts activated per token to support different depth, capacity, and inference-compute budgets.The full-depth network is trained 75% of the time and reduced-depth subnetworks 25%; full expert routing and default top-k routing each occur 80% of the time, with reduced configurations sampled in the remaining 20%.Source: Figure 4 · p. 10, §3.3 · p. 11

3.2 Training Recipe

• Pre-training progressively extends context from 8K to 32K and 128K tokens, using a warmup-stable-decay schedule initially and cosine decay during mid-training.The initial stage warms the learning rate to 1 × 10−4 over 2,000 steps and raises global batch size from 14M to 56M tokens; mid-training anneals the learning rate to 1 × 10−5. A fixed RoPE base of 1,000,000 supports context extension without reparameterization or interpolation, while posterior-based modality-loss weighting improves training stability.Source: §3.2 · p. 10

3.3 Once-For-All with Elastic Training

• Elastic training jointly optimizes full-scale ERNIE 5.0 and sub-networks during pre-training, producing smaller variants on demand without separate compression runs.Sub-networks vary in depth, total expert number, and activated expert number; variants can be instantiated for latency and memory constraints and used as starting points for mid-training or fine-tuning.Source: §3.3 · p. 11

4 Post-Training

• The post-training pipeline applies SFT for instruction following and long-chain reasoning, then multi-stage UM-RL across tasks, using a unified verifier for multimodal rewards.UM-RL combines reasoning, agent, and instruction-following tasks.Source: §4 · p. 11

4.1 Enhancing Rollout Efficiency with Unbiased Replay Buffer

• U-RB orders rollout data by iteration while generating in parallel, reducing idle time from long responses without biasing the training data distribution.Its inference pool generates rollouts while its training pool collects completed trajectories for the current iteration.Source: §4.1 · p. 12

4.2 Stabilizing Training with Mitigated Entropy Collapse

• MISC adapts trust regions to modality sensitivity, while WPSM masks redundant positive signals and shifts learning toward hard samples, alleviating entropy collapse.These methods address numerical mismatch and early overfitting to easy queries, which can destabilize multimodal RL.Source: §4.2 · pp. 13–14

4.3 Boosting Sample Efficiency with Hint-based Learning

• AHRL injects partial thinking sketches into hard queries, gradually reducing revealed hints as training improves to promote full self-exploration and mitigate sparse rewards.The revealed fraction is controlled by an annealing schedule, scaffolding exploration when valid reasoning paths are rare.Source: §4.3 · p. 15

5.1 Hybrid Parallelism for Training at Scale

• Training uses hybrid tensor, pipeline, expert, ZeRO-1 data, and context parallelism, plus DeepEP, virtual stages, and no token dropping.The final configuration uses 4-way tensor parallelism (Shoeybi et al., 2019), 12-way pipeline parallelism (Huang et al., 2019), 64-way expert parallelism (Lepikhin et al., 2020), ZeRO-1 data parallelism (Rajbhandari et al., 2020), and context parallelism (Liu et al., 2023a). FP8 activations, OOM-triggered activation offloading, sub-batch computation, and automatic CUDA virtual-memory defragmentation address memory pressure and fragmentation, including OOMs from unbalanced expert routing.Source: §5.1 · p. 16

5.2 Disaggregation Architecture for Multimodal Training

• Tokenizer-backbone disaggregation deploys modality-specific tokenizers as independently scalable services on dedicated nodes, letting tokenizers and backbone use workload-suited parallelization.Remote calls retrieve encoded representations; this separation addresses differing sequence lengths and compute costs that can cause load imbalance on homogeneous hardware.Source: §5 · p. 15, §5.2 · p. 16

5.3 FlashMask for Flexible Multimodal Attention

• FlashMask achieves up to 200% operator-level speedup over FlexAttention, more than 20% end-to-end training acceleration, and 80% improvement over Megatron-LM with context parallelism.It supports heterogeneous attention masks across modalities and samples, including causal text attention and visual attention with local bidirectional interactions.Source: §5.3 · p. 16

5.4 Scalable and Disaggregated RL Infrastructure

• A disaggregated RL system coordinates asynchronous training, inference, environment interaction, and reward evaluation to support high-throughput, deterministic multimodal RL.A shared FP8 engine and Rollout Router Replay reduce training–rollout numerical mismatch; an order-preserving replay buffer mitigates sequence-length bias, while elastic CPU pooling supports parallel simulation and improves iteration time and hardware cost efficiency.Source: §5.4 · p. 17

Benchmarks

• Pre-trained ERNIE 5.0-Base is reported to outperform strong baselines on knowledge, general reasoning, and STEM benchmarks.The report highlights advantages over DeepSeek V3.2-Exp-Base and Kimi K2-Base on knowledge tasks, best results on several exam-style benchmarks, and gains on MATH (CoT) and GPQA-Diamond.Source: Table 1 · p. 18, Benchmarks · p. 18
• Pre-trained ERNIE 5.0-Base attains state-of-the-art results on LiveCodeBench v6 and CRUXEval and significantly surpasses baselines on MMMLU and INCLUDE.The report describes strong generalization across diverse text tasks from the unified architecture and elastic pre-training strategy.Source: Benchmarks · pp. 18–19
• Post-trained ERNIE 5.0 matches or leads strong models across language tasks, though Gemini 3-Pro leads on several of the hardest reasoning benchmarks.It consistently matches or outperforms DeepSeek-V3.2-Thinking, Gemini 2.5-Pro, and GPT-5 (High) on most general reasoning, math, and coding benchmarks.Source: Table 2 · p. 19, Benchmarks · p. 19
• Post-training improves factual recall and strengthens instruction following and agent performance, including best results on MultiChallenge and Multi-IF.ERNIE 5.0 also shows competitive or leading agent results, particularly on ACEBench and BrowseComp-zh.Source: Benchmarks · p. 19

Visual Understanding & Generation Benchmarks

• Vision evaluation assesses visual reasoning, document understanding, visual question answering, video understanding, and image and video generation.The listed benchmarks span image-centric and video-centric tasks, providing a broad assessment of perception, reasoning, and generation.Source: §6.2 · p. 20, Visual Understanding & Generation… · p. 20

ERNIE 5.0-Base

• On GenEval, ERNIE 5.0 performs on par with Nano-Banana Pro (DeepMind, 2025b) and Qwen-Image (Wu et al., 2025), and comparably to GPT-Image and Seedream 4.0 (Seedream et al., 2025).The report describes its images as combining high aesthetic quality, semantic alignment, and fine-grained visual detail.Source: Table 5 · p. 21, ERNIE 5.0-Base · p. 21
• On VBench, ERNIE 5.0 achieves the best VBench-Semantic performance, remains competitive on overall and quality metrics, and produces semantically accurate, visually coherent videos.The report also describes its performance as on par with leading open-source models on video generation.Source: Table 6 · p. 21, ERNIE 5.0-Base · pp. 21–22
• ERNIE 5.0-Base performs strongly across visual reasoning, document understanding, and general VQA without instruction tuning, while instruction tuning further improves reasoning, alignment, and robustness.The report attributes base-model capabilities to early fusion of visual and textual tokens and modality-agnostic representation learning; instruction tuning refines rather than compensates for perception.Source: ERNIE 5.0-Base · p. 21, Table 4 · p. 21

Audio Understanding & Generation Benchmarks

• ERNIE 5.0 shows stable Chinese and English ASR performance across benchmarks, competitive VoiceBench results on MMSU and OpenBookQA, and favorable or comparable results across audio-understanding tasks.On SEED-TTS, it achieves competitive content consistency on Chinese and English test sets, comparable to Qwen3-Omni, while specialist CosyVoice-3 performs stronger.Source: Audio Understanding & Generation… · pp. 22–23

6.4.1 Modality-Agnostic Expert Routing

• Under shared modality-agnostic routing, ERNIE 5.0 develops both cross-modal expert reuse and modality- and task-specific activation patterns, with image, video, and audio activations more concentrated than text.Text increasingly overlaps with audio in deeper layers; image/video understanding and image/video generation show within-task overlap, while understanding and generation overlap remains relatively low.Source: §6.4.1 · pp. 23–25, Figure 8 · p. 24, Figure 9 · p. 24
• Text routing maintains high normalized entropy across layers, whereas visual understanding is less balanced at the lowest and highest layers and generative and audio tasks show shifting balance across depth.These patterns suggest differing phases of expert specialization and re-integration for generative and audio-centric tasks.Source: §6.4.1 · p. 25, Figure 10 · p. 25

6.4.2 Elastic Training

• In small-scale MoE experiments, elastic depth slightly improves full-depth performance and yields predictable reduced-depth degradation; elastic width slightly degrades at full capacity, while reduced routing budgets retain stable inference with graceful degradation.Depth and width training retain full configurations for 80% of samples and reduce the remaining 20%; width uses all 64 experts versus 32, while routing defaults to k=8 with reduced-activation instances.Source: §6.4.2 · p. 26, Table 11 · p. 26
• 75.17 vs. 75.55 average score: ERNIE 5.0-Exp-EA35.8% remains competitive with ERNIE 5.0-Exp despite using 53.7% of activated parameters and 35.8% of total parameters.The ERNIE 5.0-Exp-ES25.0% variant retains comparable benchmark performance with more than 15% faster decoding; the fully elastic variant also shows robustness on ZebraLogic and VisualPuzzle.Source: §6.4.2 · pp. 26–27, Table 12 · p. 27

• ERNIE 5.0 demonstrates competitive and balanced performance across modalities.Source: §7 · p. 27
• ERNIE 5.0 unifies text, image, video, and audio understanding and generation under a shared next-group-of-tokens objective.The report describes it as the first realization, to the authors’ knowledge, of multimodal understanding and generation within a unified, trillion-level autoregressive framework.Source: §7 · p. 27
• An ultra-sparse MoE architecture with modality-agnostic expert routing enables scalable cross-modal modeling without modality-specific designs.Source: §7 · p. 27
• Elastic training supports flexible deployment configurations within one pre-trained model while maintaining training efficiency and performance.The report also addresses reinforcement-learning challenges for large-scale multimodal models to ensure stable and efficient post-training.Source: §7 · p. 27