Source-linked AI summary
Mercury: Ultra-Fast Language Models Based on Diffusion
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, Volodymyr Kuleshov
TL;DR
Coding applications need lower latency, motivating diffusion language models that generate multiple tokens in parallel. Mercury applies this approach with Transformer-based Mercury Coder Mini and Small models, achieving strong coding quality alongside substantially higher throughput, including up to 10x over speed-optimized frontier models.
Problem
High per-user latency in coding applications such as auto-completion, code editing, and agentic workloads limits wider adoption.
Method
Mercury uses Transformer-parameterized diffusion language models trained to predict and refine multiple tokens in parallel for coding applications.
Results
Mercury Coder achieves strong coding benchmark performance while delivering up to 10x the throughput of frontier speed-optimized models.
Takeaways & Limitations
Mercury Coder offers a speed-quality operating point for latency-sensitive coding workloads, including assistants and agentic applications.
Abstract
from arXiv · showhide
We present Mercury, a new generation of commercial-scale large language models (LLMs) based on diffusion. These models are parameterized via the Transformer architecture and trained to predict multiple tokens in parallel. In this report, we detail Mercury Coder, our first set of diffusion LLMs designed for coding applications. Currently, Mercury Coder comes in two sizes: Mini and Small. These models set a new state-of-the-art on the speed-quality frontier. Based on independent evaluations conducted by Artificial Analysis, Mercury Coder Mini and Mercury Coder Small achieve state-of-the-art throughputs of 1109 tokens/sec and 737 tokens/sec, respectively, on NVIDIA H100 GPUs and outperform speed-optimized frontier models by up to 10x on average while maintaining comparable quality. We discuss additional results on a variety of code benchmarks spanning multiple languages and use-cases as well as real-world validation by developers on Copilot Arena, where the model currently ranks second on quality and is the fastest model overall. We also release a public API at https://platform.inceptionlabs.ai/ and free playground at https://chat.inceptionlabs.ai
1 Introduction
Mercury introduces large-scale diffusion language models for coding, targeting the latency constraints of coding applications through parallel generation while preserving competitive quality. The Mercury Coder family is positioned on a speed-quality frontier, with strong benchmark performance and throughput advantages over speed-optimized alternatives.
- Motivation: Mercury applies diffusion language modeling to coding, where high latency limits adoption of auto-completion, code editing, and agentic workloads.The report identifies coding as a latency-sensitive generative-AI use case.
- Approach: Parallel token generation enables Mercury Coder to improve computational efficiency by increasing arithmetic intensity on modern GPUs.The models generate tokens in a coarse-to-fine manner rather than one token at a time.
- Coding performance: Mercury Coder models improve accuracy, correctness, and in-filling capabilities across commonly used programming languages.These capabilities are linked to coding assistants, agentic workloads, chain-of-thought reasoning, and edge computing.
- Speed-quality frontier: Up to 10x higher throughput than frontier speed-optimized LLMs is reported while maintaining comparable quality on coding benchmarks.The comparison comes from third-party Artificial Analysis evaluations.
- Speed-quality frontier: Over 1000 tokens/sec on NVIDIA H100s is reported for Mercury models, a speed previously possible only with custom chips.This result is presented as a contribution of the Mercury family.
2 Inception Mercury Model Family
The Mercury family combines Transformer architectures with diffusion training and iterative parallel refinement, offering two coding models designed for efficient deployment. Its design supports standard language-model adaptation and serving workflows while targeting substantially higher throughput.
- Model family: Mercury introduces two speed-optimized diffusion language models for coding: Mercury Coder Mini and Mercury Coder Small.The models are the first focus of the Mercury family described in the report.
- Mercury Coder Mini: Mercury Coder Mini reaches 1100+ tokens/second on H100 GPUs while maintaining quality comparable to popular speed-optimized open-weight models.The model is optimized for latency-sensitive regimes.
- Mercury Coder Small: Mercury Coder Small matches popular speed-optimized frontier models while achieving 3-10x better throughput and speeds above 700 tokens/second across coding workloads.These claims apply in latency-optimized regimes.
- Diffusion process: The diffusion generation process iteratively refines outputs in parallel from random noise toward samples from the data distribution.The methods extend prior work through modifications to data and computation and are trained on trillions of tokens.
- Training: Training uses a forward noising process and a learned reverse denoising process, with parameters optimized so the reverse model fits the forward process.The denoising objective weights noise levels with γ(t) and predicts clean data from noisy data.
- Architecture and adaptation: Mercury models retain Transformer architectures, and their diffusion loss can replace autoregressive loss during pre-training, fine-tuning, and alignment.The models support adaptation through methods such as RLHF and DPO.
- Deployment: Mercury supports up to 32,768-token context natively, up to 128k with extension approaches, and standard zero-shot, few-shot, and chain-of-thought prompting.The serving interface is OpenAI-compatible, enabling use as a drop-in replacement for autoregressive models.
3 Capabilities
Mercury Coder is evaluated across coding benchmarks, programming languages, fill-in-the-middle tasks, speed, and human preference. The models combine competitive coding quality with high inference speed, with performance improving for the larger model.
- Evaluation Benchmarks: The evaluation covers standard code-generation benchmarks, six programming languages, fill-in-the-middle tasks, API speed, and human preference.Benchmarks include HumanEval, MBPP, EvalPlus, MultiPL-E, LiveCodeBench, BigCodeBench, and FIM settings.
- Overall Coding Performance: Mercury Coder Mini outperforms all open-weight models while being more than 8× faster and reaching around 1,100 output tokens per second.The result is reported for overall coding performance.
- Performance Across Programming Languages: Mercury Coder Mini and Small outperform open-weight models and show competitive performance with speed-optimized models across six programming languages.The comparison is reported on MultiPL-E, which measures the percentage of correct solutions in C++, Java, JavaScript, PHP, Bash, and TypeScript.
- Fill-in-the-Middle: Mercury Coder Mini and Small achieve state-of-the-art performance on fill-in-the-middle tasks, surpassing all evaluated models, including Codestral 2501.The tasks include single-line and random-span-light settings.
- Human Evaluation on Copilot Arena: On Copilot Arena, Mercury Coder Mini ties for second place on quality and is the fastest model, with 25 ms average latency, about 4 times faster than GPT-4o Mini.It surpasses GPT-4o Mini, Gemini-1.5-Flash, and GPT-4o in the reported preference comparison.
- Scaling: Mercury Coder Small performs consistently better than Mini across all evaluated benchmarks, indicating improved performance with the larger model.The report presents this as evidence of scaling potential for diffusion language models.