Source-linked AI summary
Arcee Trinity Large Technical Report
Varun Singh, Lucas Krauss, Sami Jaghouar, Matej Sirovatka, Charles Goddard, Fares Obied, Jack Min Ong, Jannik Straube, Fern, Aria Harley, Conner Stewart, Colin Kealty, Maziyar Panahi, Simon Kirsten, Anushka Deshpande, Anneketh Vij, Arthur Bresnu, Pranav Veldurthi, Raghav Ravishankar, Hardik Bishnoi, DatologyAI Team, Arcee AI Team, Prime Intellect Team, Mark McQuade, Johannes Hagemann, Lucas Atkins
TL;DR
Trinity addresses the need for efficient, open-weight MoE language models that can support controlled deployment and demanding long-context workflows. It develops a family combining sparse experts, specialized attention, Muon training, and a new load-balancing strategy. Training stability improved after the reported fixes, while the Preview release remains limited by light post-training.
Problem
Enterprise and software-system deployments require efficient, auditable, adaptable language models, while large MoE training can suffer from unstable routing and uneven expert utilization.
Method
The report develops Trinity Nano, Mini, and Large using sparse MoE layers, interleaved local/global and gated attention, Muon optimization, and SMEBU expert-bias updates.
Results
Expert utilization remained balanced and loss smoothly converged to a lower value after stabilization fixes, although the changes were not individually ablated.
Takeaways & Limitations
Trinity provides an open-weight family culminating in a 400B-parameter model with 13B active parameters per token, alongside smaller scaling-ladder models.
Takeaways & Limitations
Trinity-Large-Preview received relatively light post-training and is presented as a preliminary rather than fully post-trained release.
Abstract
from arXiv · showhide
We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models' modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.
1 INTRODUCTION
Trinity introduces an open-weight MoE model family designed for efficient, adaptable deployment, culminating in Trinity Large. The family combines sparse architecture, specialized attention, Muon training, and large-scale pretraining across multiple model sizes.
- Motivation: The models target deployment needs including tool use, code capabilities, long-context management, and inference-time efficiency.These requirements arise as LLMs increasingly serve as general-purpose components in software systems and long-running workflows.
- Motivation: Open-weight foundations support organizational control over auditing, hosting, adaptation, data provenance, licensing, and jurisdiction.This avoids reliance on opaque third-party checkpoints in enterprise deployments.
- Contributions: Trinity Large is the largest open-weight member of a family that also includes Nano and Mini as usable smaller models and scaling-ladder validation points.The smaller models validate the data pipeline, architecture, training recipe, and infrastructure for Trinity Large.
- Design: The architecture uses extremely sparse MoE layers, interleaved local and global attention, gated attention, and Muon optimization for efficient training and inference.Muon is described as enabling a larger critical batch size and higher sample efficiency than AdamW.
- Scale: 400B total parameters and 13B activated per token define Trinity Large, which is trained on curated web-scale and synthetic data.The report evaluates Trinity Large Base and Trinity Large Preview across standard benchmarks.
2 ARCHITECTURE
The Trinity architecture combines sparse expert modeling with a tokenizer designed for multilingual and domain coverage. Its 200,000-token BPE vocabulary improves compression in English and French, while CJK performance remains constrained by training-data timing.
- Architecture: The Trinity models are decoder-only sparse MoE transformers with no biases on linear layers.The architecture discussion includes the expert-balancing scheme.
- Tokenizer: The tokenizer uses a multi-stage pipeline with digit chunking, script-aware isolation, word and punctuation splitting, and byte-level fallback.The design supports place-aligned numerical tokens, independent script merges, and coverage without unknown tokens.
- Tokenizer: 200,000 vocabulary tokens were selected after empirical comparisons, with the largest gains appearing in CJK languages and French relative to smaller vocabularies.Smaller variants can be derived by truncating merge rank because BPE merges are deterministic and greedy.
- Tokenizer: SuperBPE reduced token counts by approximately 29% on English text and 27% on reasoning traces but did not improve downstream performance at the tested scale.The report therefore retained standard BPE at 200,000 tokens.
- Tokenizer efficiency: 4.84 B/T on C4-en and 3.98 B/T on French are the tokenizer’s strongest compression results among standard non-SuperBPE tokenizers.CJK compression is competitive but trails DeepSeek V3 and Qwen 3.
2.2 ATTENTION
Trinity attention combines grouped-query, normalized, gated, and interleaved local/global attention mechanisms. The design uses local sliding-window layers with RoPE and global layers without positional embeddings to support efficiency, stability, and long-context performance.
- Attention design: The attention stack combines GQA, QK-normalization, gated attention, and a 3:1 local/global layer pattern.Local layers use RoPE and sliding-window attention, while global layers omit positional embeddings.
- QK-normalization: QK-normalization applies RMSNorm to queries and keys before scaled dot-product attention, primarily to improve training stability with Muon.Prior work is cited regarding maximum attention-logit growth under Muon.
- Local/global pattern: Three local sliding-window attention layers followed by one global layer repeat throughout the model depth.This arrangement is chosen for long-context performance and efficiency gains at longer sequence lengths.
- Grouped-query attention: GQA maps multiple query heads to shared key/value heads, roughly matching MHA performance while reducing KV-cache size.Each query head attends over keys and values from its assigned shared KV head.
- Gated attention: Gated attention applies elementwise gating to attention outputs before the output projection.The report associates gating with reduced attention sinks, smaller excessive activations, improved evaluations, long-sequence generalization, and fewer loss spikes.
2.3 MIXTURE-OF-EXPERTS
The Trinity models use sparse MoE layers with sigmoid routing, shared and routed experts, and auxiliary-loss-free load balancing. Trinity Large introduces SMEBU, replacing discrete bias updates with magnitude-aware, soft-clamped momentum updates to address router instability.
- MoE formulation and routing: The MoE layers use fine-grained routed experts, an always-active shared expert, SwiGLU activations, and sigmoid routing with normalized scores.Expert gating scores are also applied to routed-expert outputs, while Trinity Large uses coarser-grained experts for throughput and replaces early MoE layers with dense layers.
- MoE formulation and routing: Routing selects Top-K experts using routing scores plus expert bias, while output gating uses routing scores without the bias.The decoupling reflects that expert bias is updated separately from the routing scores used to gate expert outputs.
- Load balancing: Trinity Mini and Nano use auxiliary-loss-free balancing with a decoupled expert-bias vector and re-centered bias updates.The expert bias is updated using per-expert token loads relative to the mean load, with γ controlling update speed.
- Load balancing: SMEBU replaces sign-based expert-bias updates with normalized, tanh-soft-clamped, magnitude-aware updates centered around zero and smoothed by momentum.The update uses a load-balance learning rate λ, momentum factor β, and maintained momentum buffer m.
- Load balancing: Router instability arose because sign-based updates remain fixed at ±λ, causing oscillation near local minima and preventing expert biases from settling precisely.The authors connect this issue to increasing per-layer bias-step norms as the number of experts grows.
- Load balancing: SMEBU uses bounded tanh updates that can converge to zero and momentum to damp noise; Trinity Large uses λ = 5 × 10^-4, β = 0.5, and κ = 2.A linear unclamped update reduced MaxVio early but later caused instability, motivating bounded updates.
2.4 NORMALIZATION
Trinity uses a depth-scaled sandwich normalization scheme that normalizes both inputs and outputs around each sublayer and applies additional RMSNorm before the language-modeling head.
- Sandwich normalization: Both the input and output of each attention, FFN, or MoE sublayer are normalized.The layer input is xℓ, the output is yℓ, and Mℓ(·) denotes the sublayer module.
- Depth scaling: The RMSNorm gain parameters for the second normalization in each layer are depth-scaled.The scaling is defined relative to the total number of layers L and the RMSNorm gain γ(·).
- Output normalization: RMSNorm is also applied before the language-modeling head.
2.5 INITIALIZATION
The initialization section specifies truncated-normal parameter initialization with width-scaled variance and discusses related embedding scaling choices and precedents.
- Parameter initialization: All trainable parameters are initialized from a zero-mean truncated normal distribution with a width-scaled standard deviation.The width parameter is the model dimension d.
- Parameter initialization: The initialization scale is described as broadly aligned with findings from Takase et al. and with DeepSeek-V3.
- Embedding scaling: Embedding activations are scaled during the forward pass by an embedding multiplier.The excerpt also notes related embedding_multiplier_scale settings in Grok checkpoints and analogous normalizer terminology in early Gemma models.
3 PRE-TRAINING
Trinity’s pre-training combines phased, increasingly specialized data curation with Random Sequential Document Buffer packing to reduce minibatch imbalance and training instability. Internal experiments and Trinity Large training results report lower heterogeneity and loss variance, while long-context evaluation reached strong scores beyond the training length.
- Data curation: Over 8 trillion synthetic tokens supplemented curated web-scale data, including approximately 6.5T synthetic web tokens and 800B synthetic code tokens.The curation stack used scalable infrastructure to generate web, code, and STEM data.
- Data curation: Three training phases progressively shifted the mix toward higher-quality and more domain-specific math, code, and science data.Trinity Nano and Mini used 10T-token mixes, while Trinity Large used 17T tokens from a distinct 20T-token mix.
- Data preparation: Random Sequential Document Buffer packing dynamically selects document content to reduce fragmentation and preserve random sampling across documents.The method was designed to address domain imbalance caused by sequential packing of long documents.
- Data preparation: 46% lower BatchHet and gradient-norm kurtosis of 14.6 versus 187 were observed with RSDB in small-scale internal experiments.Matching baseline loss variance required roughly 2× larger batches, while matching BatchHet required 7× larger batches; improvements occurred without dropping tokens.
- Data preparation: 4.23x lower BatchHet and 2.4x lower step-to-step variance were observed after RSDB was integrated into Trinity Large phase 3.Direct comparison was difficult because phase 2 and phase 3 used different data distributions.
- Context extension: Trinity Large achieved MK-NIAH scores of 0.994 at 256K, 0.976 at 512K, and 0.42 at 1M contexts after training at 256K.The report found no improvement from progressively increasing context length instead of training directly at the longest length.
4 POST-TRAINING
Trinity Large’s post-training was constrained by limited cluster allocation, so the reported Trinity-Large-Preview model is a preliminary release. The pipeline combines filtered instruction data, agentic coding supervision, supervised fine-tuning, and a short reinforcement-learning stage.
- Post-training scope: Limited cluster time constrained Trinity Large to a relatively light post-training phase, making Trinity-Large-Preview a preliminary rather than fully post-trained release.The authors plan more extensive post-training in a subsequent iteration.
- Dataset construction: The instruction-tuning mix combines public and custom data, including human-written prompts, teacher-generated instructions, quality filtering, and agentic coding trajectories.OpenCode is identified as one harness used to collect coding supervision.
- Supervised fine-tuning: Supervised fine-tuning uses offline pre-tokenization, greedy packing, modified TorchTitan, and Cut Cross-Entropy to improve training efficiency and reduce memory requirements.The setup parallels the context-extension phase without requiring context parallelism.
- Reinforcement learning: After supervised fine-tuning, a short asynchronous reinforcement-learning stage uses vLLM-backed rollout workers and a distributed FSDP2 trainer.The environments follow the verifiers API and favor objectively checkable rewards when available.
5 TRINITY LARGE EVALUATION RESULTS
Trinity Large Base is evaluated across coding, math, commonsense, knowledge, and reasoning benchmarks, while Trinity Large Preview is evaluated on instruction-tuned benchmarks. The report also compares Trinity Large Base with open-weight models and measures inference performance under FP8 quantization.
- Base-model evaluation: The Trinity Large Base suite spans coding/math, commonsense, knowledge, and reasoning tasks, with results reported in Table 3.The listed benchmarks include MBPP+, Minerva MATH500, HellaSwag, WinoGrande, MMLU, MMLU-Pro, TriviaQA, ARC Challenge, BBH, and GPQA Diamond.
- Base-model evaluation: Trinity Large Base achieves competitive scores with GLM 4.5 Base despite 4× higher sparsity and roughly 2.5× lower active parameter count.Comparisons to other open-weight base models are shown in Figure 3.
- Instruct-model evaluation: Trinity Large Preview is evaluated on MMLU, MMLU-Pro, GPQA Diamond, SimpleQA, and AIME25, with results reported in Table 4.These benchmarks target knowledge, reasoning, and instruction-oriented capabilities.
- Inference evaluation: Inference measurements use vLLM with all models quantized to FP8 on an 8xH200 node.The report attributes strong inference performance to Trinity Large’s extreme sparsity and interleaved local/global attention.
6 DISCUSSION
Trinity Large initially suffered router instability and collapsed experts, but a combined set of stabilization changes restored balanced utilization and smooth loss convergence. The report also evaluates throughput under a fixed FP8/vLLM/8xH200 setup.
- Training stability: Initial Trinity Large runs developed uneven expert loads, collapsed experts, loss plateaus, and stalled evaluation improvements.MaxVio remained stable before suddenly climbing as experts collapsed.
- Stabilization changes: SMEBU was adopted as one of six simultaneous changes targeting training stability.The change had previously been tested only at very small scale.
- Stabilization changes: The package also included BF16 fallbacks for selected MXFP8 operations, z-loss, and a sequence-wise auxiliary loss.The intended z-loss weight was 1 × 10^-4, but the effective mid-training weight was reduced to 1 × 10^-6 after larger values destabilized the network.
- Inference efficiency: Throughput measurements compare FP8-quantized models using vLLM on an 8xH200 node.The report attributes Trinity Large's strong performance to extreme sparsity and interleaved local/global attention.
- Stabilization outcome: The stabilization package stopped MaxVio divergence, preserved expert balance, and yielded smoothly convergent loss, but individual effects were not isolated.Because the changes were introduced together, controlled ablations were unavailable.
7 CONCLUSION AND FUTURE WORK
The report presents Trinity as an open-weight model family culminating in Trinity Large, while using smaller variants as scaling-ladder validation points. It identifies greater sparsity and large-batch training as future directions for efficient scaling.
- Conclusion: Trinity Large has 400B total parameters and 13B active per token, with Nano and Mini serving as smaller scaling-ladder validation points.The report emphasizes training stability and expert-utilization balance at the largest scale.
- Future work: Greater sparsity is identified as a promising direction for efficient scaling, especially through improved MoE load balancing and routing.The report links this direction to maintaining stable training as sparsity increases.
- Future work: Large-batch training is identified as another key direction for efficient scaling through algorithmic improvements to critical batch size.
ADDITIONAL CONTRIBUTORS
The report acknowledges the additional contributors to the Trinity work. Their names are presented in three groups.
- Additional contributors: Additional contributors include Curt Larson, Scott Zembsch, Gabriel Santos, Ben Langer, Sam Fraser, Eric Lau, and Mariam Jabara.
- Additional contributors: Additional contributors include James Weir, Davis Stone, Dante Simon, Molly Niland, Zachary Kirkendall, Mohit Khullar, Amro Abbas, and Rishabh Adiga.
- Additional contributors: Additional contributors include Cody Blakeney, Paul Burstein, Aldo Carranza, Spandan Das, Alvin Deng, Vineeth Dorna, Parth Doshi, Alex Fang, Tony Jiang, Siddharth Joshi, Brett Larsen, Jason Lee, Pratyush Maini, Kaleigh Mentzer, Luke Merrick, Haakon Mongstad, Ricardo Monti, Fan Pan, David Schwab, Darren Teh, Jason Telanoff, Jack Urbanek, Zhengping Wang, Josh Wills, Haoli Yin, Bogdan Gaza, Matthew Leavitt, and Ari Morcos.