Source-linked AI summary
Aria: An Open Multimodal Native Mixture-of-Experts Model
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yanpeng Li, Chongyan Zhu, Xiaoyi Ren, Chao Li, Yifan Ye, Peng Liu, Lihuan Zhang, Hanshu Yan, Guoyin Wang, Bei Chen, Junnan Li
TL;DR
Open multimodal-native models are limited, while proprietary models’ training recipes are largely undisclosed. ARIA is an open MoE model trained from scratch through a four-stage pipeline; it outperforms leading open comparators across multimodal, language, and coding tasks and is competitive with proprietary models on various multimodal tasks.
Problem
Proprietary multimodal-native models’ training recipes are largely undisclosed, while most open models are modal-specialized or show subpar performance across modalities.
Source: §1 · p. 1Method
ARIA is an open MoE model pretrained from scratch through a four-stage pipeline that progressively develops language, multimodal, long-context, and instruction-following capabilities.
Source: §1 · p. 1Results
ARIA surpasses Pixtral-12B and Llama3.2-11B [Dubey et al., 2024] across multimodal, language, and coding tasks, and is on par with GPT-4o and Gemini-1.5 on various multimodal tasks.
Source: §1 · p. 2Takeaways & Limitations
ARIA’s weights and fine-tuning framework are released under Apache 2.0 for academic and commercial use, with fine-tuning possible using as few as one GPU.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.
1 Introduction
ARIA addresses the scarcity of open multimodal-native models by introducing training recipes for building one from scratch. It combines a fine-grained MoE decoder, curated language and multimodal data, and a four-stage training pipeline; reported comparisons span open and proprietary models, and the weights and training framework are released for adoption.
1 Introduction
- The fine-grained MoE decoder uses expert specialization for more efficient parameter utilization and activates 3.5B parameters per text token.Source: §1 · p. 1
- ARIA is pretrained on 6.4T language tokens and 400B multimodal tokens curated from diverse sources.The multimodal data includes interleaved image-text sequences, synthetic captions, document transcriptions and QA pairs, and synthetic video captions and QA pairs.Source: §1 · p. 2
- The four-stage pipeline progresses through language, multimodal, long-context multimodal, and multimodal post-training to build capabilities while retaining earlier ones.Source: §1 · p. 2
- ARIA surpasses Pixtral-12B and Llama3.2-11B [Dubey et al., 2024] across multimodal, language, and coding tasks, and is on par with GPT-4o and Gemini-1.5 on various multimodal tasks.The authors also report lower inference cost than the named open models because ARIA activates fewer parameters.Source: §1 · p. 2, Table 1 · p. 2
- The authors release ARIA under Apache 2.0 for academic and commercial use, with a training framework for fine-tuning on varied data sources and formats using as few as one GPU.Source: §1 · p. 2
2 Model
ARIA uses a fine-grained MoE decoder with modality-generic experts, a visual encoder that converts images into decoder-compatible tokens, and an infrastructure setup designed to reduce training communication overhead.
2.1 Fine-Grained Mixture-of-Experts
- Each MoE layer has 66 experts, including two shared experts, and the router activates six experts for each token.The design uses many fine-grained experts with smaller FFN hidden dimensions.Source: §2.1 · p. 3
- ARIA is pretrained from scratch with modality-generic experts, unlike prior multimodal MoEs that use modality-specific architectures or upcycle dense models.The authors report that expert specialization arises naturally after pretraining.Source: §2.1 · p. 3
2.2 Visual Encoder
- The visual encoder converts images or video frames into continuous tokens with the word-embedding feature dimension for integration with the MoE.Its ViT accepts images at native aspect ratios as variable-length patch sequences; the projection module uses cross-attention and an FFN.Source: §2.2 · p. 3
2.3 Infrastructure
- Training combines expert parallelism with ZeRO-1 data parallelism and avoids tensor parallelism, reducing communication overhead.The authors attribute the ability to train without tensor parallelism to the parallelism method and small model size.Source: §2.3 · p. 3
- To prevent routing collapse and stabilize training, ARIA balances load across groups of eight fine-grained experts and uses z-loss.Source: §2.3 · pp. 3–4
3 Training
ARIA’s four-stage training pipeline aims to add capabilities while retaining those learned earlier, with evaluation during training to check progress efficiently. The long-context stage extends the window to 64K tokens, after which the model reportedly solves needle-in-a-haystack tasks at that length and improves on long-video and document understanding.
3 Training
- The four-stage pipeline seeks to add capabilities while preserving earlier ones, with evaluation at each stage to support data- and compute-efficient training.Source: §3 · p. 4
3.1 Language Pre-training
- Language pretraining uses 6.4T curated tokens, minimum-spanning-tree data clustering, and an 8K-token context window.The authors report that the clustering approach produced a noticeable performance gain.Source: §3.1 · p. 4
3.2 Multimodal Pre-training
- Multimodal pretraining combines the decoder and visual encoder on language data and 400B multimodal tokens spanning web image-text, synthetic captions, documents, and video.The data sources include document transcription and QA, plus video descriptions, QA pairs, and summaries.Source: §3.2 · p. 4
3.3 Multimodal Long-Context Pre-training
- Long-context pretraining uses long videos, documents, and synthetic sequences, consuming 12B language tokens and 21B multimodal tokens to extend the context window to 64K.The RoPE base frequency is increased from 100K to 5M.Source: §3.3 · p. 5
- After long-context pretraining, ARIA perfectly solves needle-in-a-haystack tasks up to a 64K context window and improves substantially on long-video and long-document understanding.Source: §3.3 · p. 5
3.4 Multimodal Post-training
- Post-training anneals the learning rate and uses 20B tokens from open-source and human-annotated datasets to improve question answering and instruction following.The datasets cover multimodal, code, math, and reasoning domains.Source: §3.4 · p. 5
4 Evaluation and Analysis
The section evaluates Aria against open and proprietary systems and analyzes expert specialization alongside qualitative multimodal and long-context behavior. Examples show Aria tracking a video’s scene sequence and explaining code displayed in a frame.
4.1 Benchmark Results
- ARIA is the best-in-class open multimodal native model, outperforming Pixtral-12B and Llama3.2-11B across multimodal, language, and coding tasks, and competing with proprietary models on several multimodal tasks.The proprietary-model comparisons include document understanding, chart reading, scene text recognition, and video understanding.Source: §4.1 · p. 5
- ARIA significantly outperforms Qwen2-VL-7B [Bai et al., 2023] and LLaVA-OneVision-72B [Li et al., 2024] in long-context understanding, beats GPT-4o mini [Wu et al., 2024] on long video and Gemini-1.5-Flash on long documents, and outperforms open models on instruction-following benchmarks.The paper describes ARIA as a compute- and time-efficient choice for processing long multimodal data.Source: §4.1 · p. 6
4.2 Expert Modality Specialization
- Activation-rate ratios identify visual-specialized experts in most layers; seven layers (4, 5, 14, 15, 16, 17, 20) contain one expert specialized across all three visual domains.The analysis compares visual-token activation rate Rv with text-token activation rate Rt across natural-image, video, and PDF-image data.Source: §4.2 · p. 6, Figure 1 · p. 7
4.3.1 Multimodal Native Reasoning with Vision, Language, Coding Capabilities
- ARIA's report analysis found profit margin rose from approximately 20.91% in 2020 to approximately 25.88% in 2021, primarily due to increased net income and effective cost management relative to total net sales growth.Source: §4.3.1 · p. 11
4.3.2 Long Multimodal Context Window
- Long-context demonstrations include a LONGVIDEOBENCH summary describing up to hour-long videos, 3,763 videos, 6,678 questions across 17 categories, and timestamped scene descriptions for a video.Source: §4.3.2 · pp. 15, 17, 18, 19
- Aria describes scenes spanning Florence’s political history, Michelangelo’s David, the Cathedral of Florence, and the video’s conclusion.Source: §4.3.2 · pp. 19–20
- The HiMom class flattens input tensors, passes them through a sequential stack of fully connected layers and ReLU activations, and returns output logits with 10 features.The layers map 28*28 inputs to 512 features, then 512 to 512, and finally 512 to 10 output features.Source: §4.3.2 · p. 22