PaperlayerView exact revision on arXiv

Source-linked AI summary

Command A: An Enterprise-Ready Large Language Model

Team Cohere, :, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, Neeral Beladia, Walter Beller-Morales, Alexandre Bérard, Andrew Berneshawi, Anna Bialas, Phil Blunsom, Matt Bobkin, Adi Bongale, Sam Braun, Maxime Brunet, Samuel Cahyawijaya, David Cairuz, Jon Ander Campos, Cassie Cao, Kris Cao, Roman Castagné, Julián Cendrero, Leila Chan Currie, Yash Chandak, Diane Chang, Giannis Chatziveroglou, Hongyu Chen, Claire Cheng, Alexis Chevalier, Justin T. Chiu, Eugene Cho, Eugene Choi, Eujeong Choi, Tim Chung, Volkan Cirik, Ana Cismaru, Pierre Clavier, Henry Conklin, Lucas Crawhall-Stein, Devon Crouse, Andres Felipe Cruz-Salinas, Ben Cyrus, Daniel D'souza, Hugo Dalla-Torre, John Dang, William Darling, Omar Darwiche Domingues, Saurabh Dash, Antoine Debugne, Théo Dehaze, Shaan Desai, Joan Devassy, Rishit Dholakia, Kyle Duffy, Ali Edalati, Ace Eldeib, Abdullah Elkady, Sarah Elsharkawy, Irem Ergün, Beyza Ermis, Marzieh Fadaee, Boyu Fan, Lucas Fayoux, Yannis Flet-Berliac, Nick Frosst, Matthias Gallé, Wojciech Galuba, Utsav Garg, Matthieu Geist, Mohammad Gheshlaghi Azar, Ellen Gilsenan-McMahon, Seraphina Goldfarb-Tarrant, Tomas Goldsack, Aidan Gomez, Victor Machado Gonzaga, Nithya Govindarajan, Manoj Govindassamy, Nathan Grinsztajn, Nikolas Gritsch, Patrick Gu, Shangmin Guo, Kilian Haefeli, Rod Hajjar, Tim Hawes, Jingyi He, Sebastian Hofstätter, Sungjin Hong, Sara Hooker, Tom Hosking, Stephanie Howe, Eric Hu, Renjie Huang, Hemant Jain, Ritika Jain, Nick Jakobi, Madeline Jenkins, JJ Jordan, Dhruti Joshi, Jason Jung, Trushant Kalyanpur, Siddhartha Rao Kamalakara, Julia Kedrzycki, Gokce Keskin, Edward Kim, Joon Kim, Wei-Yin Ko, Tom Kocmi, Michael Kozakov, Wojciech Kryściński, Arnav Kumar Jain, Komal Kumar Teru, Sander Land, Michael Lasby, Olivia Lasche, Justin Lee, Patrick Lewis, Jeffrey Li, Jonathan Li, Hangyu Lin, Acyr Locatelli, Kevin Luong, Raymond Ma, Lukáš Mach, Marina Machado, Joanne Magbitang, Brenda Malacara Lopez, Aryan Mann, Kelly Marchisio, Olivia Markham, Alexandre Matton, Alex McKinney, Dominic McLoughlin, Jozef Mokry, Adrien Morisot, Autumn Moulder, Harry Moynehan, Maximilian Mozes, Vivek Muppalla, Lidiya Murakhovska, Hemangani Nagarajan, Alekhya Nandula, Hisham Nasir, Shauna Nehra, Josh Netto-Rosen, Daniel Ohashi, James Owers-Bardsley, Jason Ozuzu, Dennis Padilla, Gloria Park, Sam Passaglia, Jeremy Pekmez, Laura Penstone, Aleksandra Piktus, Case Ploeg, Andrew Poulton, Youran Qi, Shubha Raghvendra, Miguel Ramos, Ekagra Ranjan, Pierre Richemond, Cécile Robert-Michon, Aurélien Rodriguez, Sudip Roy, Sebastian Ruder, Laura Ruis, Louise Rust, Anubhav Sachan, Alejandro Salamanca, Kailash Karthik Saravanakumar, Isha Satyakam, Alice Schoenauer Sebag, Priyanka Sen, Sholeh Sepehri, Preethi Seshadri, Ye Shen, Tom Sherborne, Sylvie Shang Shi, Sanal Shivaprasad, Vladyslav Shmyhlo, Anirudh Shrinivason, Inna Shteinbuk, Amir Shukayev, Mathieu Simard, Ella Snyder, Ava Spataru, Victoria Spooner, Trisha Starostina, Florian Strub, Yixuan Su, Jimin Sun, Dwarak Talupuru, Eugene Tarassov, Elena Tommasone, Jennifer Tracey, Billy Trend, Evren Tumer, Ahmet Üstün, Bharat Venkitesh, David Venuto, Pat Verga, Maxime Voisin, Alex Wang, Donglu Wang, Shijian Wang, Edmond Wen, Naomi White, Jesse Willman, Marysia Winkels, Chen Xia, Jessica Xie, Minjie Xu, Bowen Yang, Tan Yi-Chern, Ivan Zhang, Zhenyu Zhao, Zhoujie Zhao

arXiv:2504.00698v2cs.CLcs.AIcs.LG

TL;DR

Enterprise language models need broad capabilities, multilingual coverage, and efficient deployment while supporting diverse tasks such as code, safety, and retrieval. This report develops Command A through decentralised expert training and merging, and evaluates it across enterprise and public benchmarks. The resulting model is reported to combine best-in-class performance with lower serving requirements and broad enterprise versatility, although evaluation compute remains a bottleneck and fairness measurements cover only generation.

  • Problem

    The paper addresses how one enterprise LLM can support diverse capabilities, including code, safety, retrieval, and multilingual understanding, despite differing data and training dynamics.

  • Method

    Command A alternates centralised fine-tuning with decentralised training of domain-specific experts, then merges expert parameters into a single model.

  • Results

    Command A achieves reported best-in-class performance across diverse enterprise tasks while balancing performance, efficiency, and versatility.

  • Takeaways & Limitations

    Command A can run on two A100 or H100 GPUs and deliver up to 156 tokens/sec, supporting on-premises and private-cloud deployment.

  • Takeaways & Limitations

    Model-merging evaluation is computationally expensive, creating a production bottleneck when many capabilities and metrics must be optimised.

Abstract

from arXiv · show

In this report we describe the development of Command A, a powerful large language model purpose-built to excel at real-world enterprise use cases. Command A is an agent-optimised and multilingual-capable model, with support for 23 languages of global business, and a novel hybrid architecture balancing efficiency with top of the range performance. It offers best-in-class Retrieval Augmented Generation (RAG) capabilities with grounding and tool use to automate sophisticated business processes. These abilities are achieved through a decentralised training approach, including self-refinement algorithms and model merging techniques. We also include results for Command R7B which shares capability and architectural similarities to Command A. Weights for both models have been released for research purposes. This technical report details our original training pipeline and presents an extensive evaluation of our models across a suite of enterprise-relevant tasks and public benchmarks, demonstrating excellent performance and efficiency.

1 Introduction

Command A and Command R7B are developed for enterprise use, combining multilingual and agentic capabilities with strong benchmark performance and efficient deployment. Command A supports 23 business languages, performs strongly across human and agentic evaluations, and can serve on two A100 or H100 GPUs.

  • Enterprise focus: Command A and Command R7B target real-world enterprise settings with data, architectural, self-refinement, and model-merging optimisations.Both models are described as best-in-class for their respective sizes across established benchmarks.
  • Multilingual capability: 23 languages are supported, spanning major global business languages including English, French, Spanish, Chinese, Arabic, Japanese, and Persian.The supported language list also includes German, Portuguese, Korean, Russian, Polish, Turkish, Vietnamese, Dutch, Czech, Indonesian, Ukrainian, Romanian, Greek, and Hebrew.
  • Evaluation: Command A outperforms across extensive human evaluations and achieves strong results on enterprise agentic benchmarks such as Taubench.The report also describes best-in-class results within its size and efficiency range on common benchmarks such as MATH.
  • Efficiency: 156 tokens/sec is Command A’s reported maximum serving rate, 1.75x higher than GPT-4o and 2.4x higher than DeepSeek V3.The model is presented as requiring only two A100 or H100 GPUs, reducing computational overhead for privacy-preserving and on-premises enterprise deployments.
  • Availability: Model weights are released to the research community under a CC-BY-NC license with an acceptable-use addendum.The checkpoints are available on the HuggingFace model hub.

2 Pre-training

The report describes the pre-training foundations, data sources, Transformer architecture, scaling strategy, and precision choices used for Command A. These choices combine broad multilingual and code data with long-context training and throughput-oriented distributed computation.

  • Pre-training foundations: Pre-training uses trillions of unlabelled tokens and self-supervised next-token prediction to learn general language patterns for downstream adaptation.The passage characterizes pre-training as computationally intensive but foundational for broad language-model capability.
  • Data: Training data combines multilingual web text and code, internally generated synthetic datasets, human-annotated instruction data, and specialised vendor data.The web corpus is refined through educational-data enrichment, quality filtering, de-duplication, and heuristic filtering.
  • Architecture: Command A uses a decoder-only Transformer with SwiGLU, interleaved sliding-window and full-attention layers, and grouped-query attention for serving throughput.Sliding-window and full-attention layers are interleaved in a 3:1 ratio, with RoPE for sliding-window layers and NoPE for full-attention layers.
  • Scaling strategy: Hyperparameters are tuned on smaller models using µP and µTransfer before zero-shot transfer to larger models.Sweeps are performed separately for each model size because they assume a fixed number of layers.
  • Distributed training: Distributed training combines data, fully sharded data, and sequence parallelism to reduce activation communication in high-throughput pre-training.The framework uses JAX GSPMD and overlaps next-layer weight communication with current-layer execution through forward-loop unrolling.
  • Precision and throughput: FP8 tensor cores improve throughput, while FP32 is retained for main weights and optimiser states and BF16 is used for attention computation.A preliminary BF16 phase mitigates the downstream-performance degradation observed when the entire run uses FP8.
  • Training schedule: The learning rate is annealed from 2.5 × 10^-4 to 1 × 10^-6 over 50,000 steps while context length expands from 8k to 256k tokens.The context expansion proceeds through 8k, 32k, 128k, and 256k-token stages with long-context data interleaving.

3 Post-training

Command A’s post-training combines decentralised expert development, self-refinement, and model merging to improve diverse enterprise capabilities within one model. The pipeline covers multilingual, coding, reasoning, safety, and capability-recovery objectives while addressing the cost of evaluating many possible merges.

  • Overview: Command A alternates centralised fine-tuning with separately trained expert tracks whose parameters are merged to maximise performance across domains and capabilities.This decentralised approach is presented as a large-scale extension of sequential post-training, with parameter merging central to the recipe.
  • Preference Training with Self-refinement: SRPO learns a self-improvement policy that iteratively revises generations from a primary policy while remaining close to a reference policy.The method is described as independent of the preference-dataset sampling distribution, supporting generalisation and stability across deployment scenarios.
  • Multilingual: Command A supports 23 global-business languages, while multilingual expert training uses supervised fine-tuning and preference tuning; merging random-seed models helps at SFT but not preference tuning.The report identifies multilingual capability as a core enterprise requirement and notes that uniform merging produced only a slight SFT-stage boost.
  • Code: The coding pipeline combines synthetic and human data, but novel synthesis methods yield negligible improvement for larger models and still require human annotation and synthetic augmentation.The code experts target programming tasks including legacy COBOL and SQL, while augmentation also covers explanations, formatting, precision, and completion structure.
  • Expert Merging: Expert merging combines models specialised for different capabilities, enabling asynchronous team optimisation and preserving close-to-expert performance across multiple domains.Linear weight averaging is selected through heuristic and brute-force searches; more complex merging methods added complexity without significant performance gains.
  • Capability Recovery: Merging the expert with the original base model can recover a significant proportion of forgotten capabilities, especially long-context ability, while retaining the new expert capability.The report also identifies evaluation as a production bottleneck because each merge requires substantial inference time and compute.

4 Results

Command A performs competitively or best-in-class across academic, multilingual, agentic, coding, SQL, reasoning, safety, fairness, and enterprise evaluations. Its agentic code-use setup further improves performance over direct generation, while human evaluations show strong preference and efficiency.

  • Standard benchmarks: Command A outperforms similarly sized models on InFoBench and is surpassed only by Llama 3.3 70B Instruct on IFEval.
  • Multilingual and agentic evaluation: Command A outperforms Llama 3.3 70B Instruct and Llama 3.1 405B Instruct across all evaluated multilingual human-evaluation languages and is entirely robust to gender perturbations.
  • Multilingual and agentic evaluation: Command A outperforms widely adopted agentic solutions such as Mistral Large 2 and Gemini 1.5 Pro on multilingual τ-bench while remaining competitive with GPT-4o.
  • Code and SQL: Command A provides state-of-the-art code understanding, including COBOL generation and translation, and strong performance across Python and other business-critical languages.
  • Code and SQL: +5.9% pass@1 on LiveCodeBench, +14.3% on BigCodeBench, and +12.3% on LBPP result when Command A uses execution-feedback tools instead of direct code generation.
  • Code and SQL: Command A leads on Spider Dev and Bird and has the strongest average performance among similarly sized models across five enterprise-critical SQL dialects.
  • Reasoning: Merging preserves reasoning performance within a few percentage points across most mathematical benchmarks, while reasoning-expert gains are additive in coding and structured data manipulation.
  • Safety and fairness: Command A lies on the safety Pareto frontier and shows robust demographic fairness, with complete gender robustness and only 1% race-perturbation failures.

5 Conclusion

The report presents Command A as an enterprise-focused model combining broad capability coverage with efficient deployment. It highlights performance across enterprise tasks and public benchmarks, alongside innovations in data, architecture, self-refinement, and model merging.

  • Command A achieves best-in-class performance across a wide range of enterprise-relevant tasks and public benchmarks.
  • The model combines agentic workflows, multilingual understanding and generation, and instruction-following capabilities for enterprise applications.
  • Command A requires fewer serving resources and can run on just two A100 or H100 GPUs in on-premises or private-cloud environments.
  • Model weights were released under a non-commercial license to support community-based exploration and research.

B.2.1 Expert Training Considerations

The expert-training procedure uses supervised fine-tuning followed by multilingual preference tuning. These stages use distinct data sources, optimisation settings, and model-merging choices.

  • Supervised Fine-Tuning uses Adam optimisation, cosine learning-rate decay, β1 = 0.9, β2 = 0.95, and weight decay of 0.1.
  • Several models with the same SFT configuration but different random seeds are merged during the supervised fine-tuning stage.
  • Preference tuning follows SFT and uses human-annotated and synthetically generated multilingual preference data with DPO.
  • Preference tuning applies SFT regularisation with the SFT data mixture and does not merge models across multiple seeds.

B.2.2 Results

The multilingual evaluation reports competitive results on mathematical reasoning and regional-context benchmarks. Additional tables provide MGSM and INCLUDE-44 results, with a noted evaluation-template caveat for French.

  • Command A demonstrates highly competitive multilingual capabilities on MGSM and INCLUDE-44.
  • MGSM scores are reported using simple-evals, with an internal French score of 90.0% when decimal commas are handled correctly.
  • INCLUDE-44 results are presented for individual languages.

B.3.1 Expert Training Considerations

The expert-training procedure has three stages: large-scale supervised learning, high-quality-data supervised learning, and reinforcement learning over scored or preference-pair data.

  • Stage 1 trains the expert with large-scale supervised learning using Adam optimisation, cosine decay, regularisation, and gradient clipping.
  • Stage 2 fine-tunes the merged model using only high-quality data and follows a schedule similar to Stage 1.
  • Stage 3 applies reinforcement learning over scored or preference-pair data with a constant learning rate and regularisation parameter β of 0.06.

B.3.2 Results

This section reports code-evaluation results, including HumanEval and LBPP breakdowns across multiple programming languages, alongside externally sourced comparison figures.

  • B.3.2 Results: HumanEval and LBPP results are broken down across Python, C++, Rust, Java, Javascript, Go, and COBOL.Table 26 reports full pass@1 results and sample-weighted averages across six languages.
  • B.3.2 Results: Table 26 combines internal reproductions with externally reported values where indicated.External values are followed by footnote citations and internal reproductions.
  • B.3.2 Results: Code-understanding and code-editing benchmark comparisons draw on externally reported model-card and benchmark sources.The listed sources include BigCode Bench, LiveCodeBench, DeepSeek, Meta, and Hui et al. (2024).

B.4 Reasoning

The reported training procedures use Adam-based optimization for supervised fine-tuning and preference tuning, with distinct learning-rate schedules.

  • B.4 Reasoning: Supervised fine-tuning uses Adam with cosine decay from 2.5×10−5 to 2.5 × 10−6.The configuration also uses β1 = 0.9, β2 = 0.95, weight decay of 0.01, and gradient norm clipping at 1.0.
  • B.4 Reasoning: Preference tuning uses CoPG with Adam at a learning rate of 2×10−6 and no decay.It uses β1 = 0.9, β2 = 0.95, and gradient norm clipping at 1.0.

B.5 Long Context

The training configuration described for this section uses Adam with cosine learning-rate decay and gradient norm clipping.

  • B.5 Long Context: Training uses Adam with a peak learning rate of 2.5 × 10−5 decayed to 2.5 × 10−6.The configuration includes β1 = 0.9, β2 = 0.95, weight decay of 0.01, and gradient norm clipping at 1.0.

B.6 Safety

This section compares safety-mode performance among similarly sized models and presents absolute safety performance for large and small models in the default setting.

  • B.6 Safety: Safety-mode comparisons evaluate accuracy and over-refusal rates across similarly sized models.Higher accuracy and lower over-refusal rates indicate better performance.
  • B.6 Safety: The over-refusal set was developed by red-teaming Command R+ Refresh and is specifically challenging for Command A models.The table compares safety-mode instructions supplied as system messages or first messages.
  • B.6 Safety: Figures 15 and 16 present absolute safety performance for large and small models, respectively, in the default setting.Figure 15 covers large models; Figure 16 covers small models.

B.6.1 Expert Training Considerations

The safety expert is trained with supervised fine-tuning and offline preference tuning, using distinct learning-rate schedules and optimization settings.

  • Supervised fine-tuning uses Adam with β1 = 0.9, β2 = 0.95, and a peak learning rate of 10−4.The learning rate decays to 10−5 with a cosine schedule; gradient norm is clipped to 1 and weight decay is 10−3.
  • Offline preference tuning uses the same hyper-parameters as supervised fine-tuning except for a lower peak learning rate of 10−6.The learning rate decays to 10−7, and IPO uses KL regularisation with β = 0.03.
  • IPO-based preference tuning applies KL regularisation with β = 0.03 while retaining the other supervised fine-tuning hyper-parameters.

B.6.2 Results

Additional tables and figures report Command A’s safety-mode performance and emphasize competitive safety performance across large and small models.

  • Table 27 provides additional safety-mode performance results for the Command A models.
  • Figures 15 and 16 show absolute safety performance for large and small models, respectively, in the default safety setting.
  • The additional results further highlight the models’ competitive safety performance.

B.7 Evaluation on Standard Benchmarks

The standard-benchmark evaluation details the tasks used to measure academic and professional knowledge, specialized STEM reasoning, and instruction following, along with their prompting settings.

  • Knowledge and reasoning benchmarks: MMLU measures university-level academic knowledge across diverse subjects using 5-shot Chain-of-Thought prompting.
  • Knowledge and reasoning benchmarks: MMLU-Pro evaluates knowledge across professional and academic domains, including law, medicine, and engineering, using 5-shot Chain-of-Thought prompting.
  • Knowledge and reasoning benchmarks: GPQA measures graduate-level academic knowledge in specialized STEM topics using 0-shot Chain-of-Thought prompting and reports only the diamond subset.
  • Instruction following: IFEval measures instruction-following ability across 25 types of verifiable instructions, including output length, keyword inclusion or exclusion, and formatting.
Loading 2504.00698v2…