Source-linked AI summary
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, Iacopo Poli
TL;DR
Older encoder-only models remain important for efficient retrieval and classification but often use limited context, inefficient designs, and narrow training data. ModernBERT modernizes the architecture and training recipe, using 2 trillion tokens and long-context design choices. It achieves broad state-of-the-art encoder performance while improving efficiency, including on long-context retrieval and NLU.
Problem
Older encoder pipelines often rely on BERT despite 512-token limits, inefficient designs, suboptimal vocabularies, and narrow training data.
Method
ModernBERT combines an improved transformer architecture with efficiency and GPU-aware modifications, training on 2 trillion primarily English tokens including code.
Results
ModernBERT delivers a Pareto improvement over BERT and RoBERTa across evaluation categories, outperforms existing encoders on short-context retrieval, and sets strong NLU and long-context retrieval results.
Takeaways & Limitations
ModernBERT provides an encoder-only backbone combining broad classification and retrieval performance with efficient long-context processing on common GPUs.
Takeaways & Limitations
The study focuses exclusively on English and is not directly applicable to other languages, especially potentially lower-resource languages.
Abstract
from arXiv · showhide
Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.
1 Introduction
Encoder-only models remain central to retrieval and discriminative NLP pipelines because they offer efficient processing and a strong quality-size tradeoff. ModernBERT addresses older encoders’ limitations with a modern architecture, larger and broader training data, long context, and improved efficiency.
- Encoder-only models remain widely used for retrieval and discriminative tasks because they efficiently process large document corpora and offer a strong quality-size tradeoff.
- Semantic-search encoders are a core component of retrieval-augmented generation pipelines, supplying language models with context relevant to user queries.
- Older encoder pipelines often retain BERT despite 512-token sequence limits, inefficient designs, suboptimal vocabularies, narrow training data, and limited code or recent-event coverage.
- Recent modernization efforts improved training efficiency or context length but generally lacked broad simultaneous optimization for efficiency, classification, retrieval, and programming-related performance.
- ModernBERT uses an improved architecture and 2 trillion training tokens including code, releases base and large models, and processes 8192-token sequences almost two times faster than previous models.
- FlexBERT is released as a modular framework with intermediate training checkpoints to support further encoder-only model research.
2 Methods
ModernBERT combines recent transformer, efficiency, and hardware-aware design choices with large-scale English pretraining. Its method targets efficient long-context inference while preserving model capacity and downstream performance.
- Architectural Improvements: ModernBERT extends the standard transformer with recent architectural advances, efficiency-oriented modifications, implementation improvements, and GPU-optimized design informed by ablations.
- Efficiency Improvements: Alternating attention uses global attention every third layer and 128-token local sliding-window attention in the remaining layers.
- Efficiency Improvements: Unpadding removes padding tokens before embedding, concatenates batch sequences, and uses variable-length Flash Attention and RoPE implementations for more efficient processing.
- Efficiency Improvements: Flash Attention supplies memory- and compute-efficient kernels, while torch.compile improves training throughput by 10 percent with negligible compilation overhead.
- Hardware-Aware Design: Hardware-aware ablations target common server and consumer GPUs, balancing Deep & Narrow architecture with inference speed.
- Hardware-Aware Design: ModernBERT base and large use 22 and 28 layers with 149 and 395 million parameters, respectively, and dimensions chosen for efficient tensor-core tiling.
- Pretraining: Both models train on 2 trillion primarily English tokens from web documents, code, and scientific literature, using a modern BPE tokenizer and over 99 percent sequence-packing efficiency.
3 Downstream Evaluation
ModernBERT is evaluated across natural-language understanding, code, short- and long-context retrieval, using comparisons with existing encoders of similar size. It shows strong overall downstream performance, with especially notable results on code and long-context multi-vector retrieval.
- ModernBERT is the strongest overall model at both BASE and LARGE sizes, representing a Pareto improvement over BERT and RoBERTa across evaluation categories.
- Short-Context Retrieval: Both ModernBERT variants outperform existing encoders in DPR and ColBERT short-context retrieval, with ModernBERT-large leading GTE-en-MLM-large despite having fewer parameters.ModernBERT-large has 395M parameters versus 435M for GTE-en-MLM-large.
- Long-Context Retrieval - Single Vector: In long-context DPR retrieval, ModernBERT performs well without long-context finetuning, but trails GTE-en-MLM out of domain while performing similarly in domain.The authors suggest that additional adapted tuning may be needed for ModernBERT in this setting.
- Long-Context Retrieval - Multi-Vector: In long-context ColBERT retrieval, all long-context models outperform short-context models by at least 40 NDCG@10 points, while ModernBERT leads other long-context models by at least 9 NDCG@10 points at both sizes.These gains occur without specific finetuning; the authors leave explanations involving long pretraining and local attention for future work.
- Natural Language Understanding: ModernBERT-base surpasses all existing base models on GLUE, while ModernBERT-large nearly matches DeBERTaV3-large with one-tenth fewer parameters and half the processing time.
- Code: ModernBERT outperforms all other models on both CodeSearchNet and StackQA programming tasks.The authors attribute this result to ModernBERT being the only evaluated encoder trained with programming data.
4 Efficiency
ModernBERT is evaluated for throughput and memory across fixed and variable sequence lengths on an NVIDIA RTX 4090. It is especially efficient at long contexts, while also supporting larger batch sizes than competing encoders.
- Throughput: 46,801 tokens per second is ModernBERT-large’s processing speed at length 8192, close to GTE-en-MLM base’s 47,507 tokens per second.
- Throughput: 14.5-30.9 percent more tokens per second at low context lengths and 98.8-118.8 percent more at longer lengths, ModernBERT outpaces GTE-en-MLM on variable-length inputs.The advantage is attributed to local attention.
- Memory efficiency: ModernBERT-base processes batch sizes twice as large as every other model on both input lengths.ModernBERT is reported as the overall most memory-efficient model at both model sizes.
- Memory efficiency: ModernBERT-large processes batches at least 60 percent bigger than every other large model, although it is slightly less memory efficient than BERT-large on short-context inputs.
5 Conclusion
ModernBERT combines modern architecture and large-scale pretraining to advance encoder performance across classification and retrieval, including long-context and code tasks. It also delivers substantially faster inference with strong memory efficiency.
- Overall results: ModernBERT sets a new state of the art over existing encoder models across a wide range of classification and retrieval tasks.
- Architecture: 8,192 tokens is ModernBERT’s native sequence length, supported by GeGLU layers, RoPE embeddings, and alternating local-global attention.
- Benchmark results: At least 6.85 and 9.1 percentage points higher than the closest model, ModernBERT scores in code and ColBERT-style long-context retrieval benchmarks, respectively.It remains state-of-the-art on short-context retrieval in both single- and multi-vector settings.
- Efficiency: Twice as fast as DeBERTaV3 on short-context inputs and two times faster than the next-fastest model on long-context inputs, ModernBERT also has best-in-class memory efficiency.
- Overall results: ModernBERT improves performance over BERT and RoBERTa on classification and retrieval while supporting long-context and programming applications.
6 Limitations
ModernBERT has limitations involving language coverage, web-data bias, harmful-content generation, the MLM-only objective, and unexplored parameter scaling.
- The study focuses exclusively on English and is not directly applicable to other languages, especially lower-resource languages.
- Training largely on web data means ModernBERT’s representations are subject to biases present in that data.
- The MLM objective gives ModernBERT some ability to generate text by predicting replacements for [MASK], creating potential harmful-content risk.The paper notes that ModernBERT is not primarily generative and cannot generate longer text sequences.
- The authors leave extension to a combined MLM and RTD training objective as a promising direction for improving classification results.This limitation is motivated by DeBERTaV3’s strong classification but weak retrieval results.
- Parameter scaling remains unexplored beyond the study’s architectural modifications and data scaling.
8 Contribution Statement
The project was jointly led by BW, AC, and BC, with contributions spanning model design, training, coordination, evaluation, ablations, and paper writing.
- BW, AC, and BC jointly led the project and contributed to all parts of it.BW led model design and training, while AC co-initiated and coordinated the project and co-led evaluations and ablations.
A Training Settings
ModernBERT’s training settings combine standard tooling with batch-size warmup, center initialization for the large model, decoupled weight decay, checkpoint averaging, and GPU-oriented design choices.
- ModernBERT uses Composer as its training framework and optim¯ı for optimizer implementations.
- Batch-size warmup gradually increases the batch size to improve training with medium to large batches.The schedule is longer than learning-rate warmup and changes the effective early optimization dynamics.
- ModernBERT-large is initialized from ModernBERT-base using center tiling and Gopher layer scaling.The procedure centers base weights and fills the larger matrices using wraparound.
- Weight decay excludes bias terms and normalization layers, while the remaining decay is fully decoupled rather than PyTorch-style decoupled.
- ModernBERT-base’s final checkpoint averages its three best annealing checkpoints with the final checkpoint.The authors report that exponential moving average during annealing did not produce stronger performance in their experiments.
- The model design targets Tensor Core divisibility, 128 × 256 tile quantization, and GPU-specific streaming-multiprocessor utilization.Because wave quantization is impractical across varied GPUs, the authors optimize a basket of target devices instead.
C Training Log
The training log documents a sampling-order failure, a restarted large-model run with reduced optimization settings, and continued improvement throughout base-model training.
- C.1 Sampling Issue: ModernBERT-base’s first pretraining run diverged after a slow seesaw pattern in the loss.
- C.1 Sampling Issue: The authors attributed the sampling problem to sequential bias in PyTorch’s random sampler between 500 million and 1 billion samples.They replaced it with NumPy’s PCG64DXSM random sampler, while noting that the threshold was not rigorously analyzed.
- C.1 Sampling Issue: ModernBERT-large training was rolled back and restarted at a learning rate of 5e-5 and weight decay of 1e-6 for the final 800 billion tokens.The restart followed a plateau in training loss, validation metrics, and live MNLI evaluations at the earlier 5e-4 learning rate.
- C.1 Sampling Issue: ModernBERT-base continued improving across training loss, validation metrics, and live MNLI evaluations during its 1.719 trillion-token training phase.The improvement diminished over time, highlighting risks of a constant learning rate.
D Architecture ablations
ModernBERT’s architecture was selected through targeted ablations balancing downstream performance, speed, and implementation efficiency. The final design retains choices that were robust at scale while avoiding parallel attention because it degraded downstream performance for the tested model sizes and sequence lengths.
- Ablations compared GeGLU and SwiGLU, finding nearly identical results and selecting GeGLU for ModernBERT.
- RoPE dimensions of 50%, 75%, and 100% produced minimal differences, so the final models retained 100% to avoid hindering fully trained capabilities.
- LayerNorm and RMSNorm performed similarly, but LayerNorm was chosen because native PyTorch RMSNorm was unavailable and could reduce out-of-the-box efficiency.
- Parallel attention offered minimal speedups at the target sizes and pre-training length while significantly degrading downstream performance, so ModernBERT excludes it.The authors note that larger encoders or longer sequences may have a different trade-off.
- Alternating global and local attention matched global attention in every layer while producing major speedups, even after 100 billion tokens.
- Tokenizer experiments selected a modified OLMo tokenizer because it performed best among recent tokenizers and was expected to better support recent data and code.
E.2 Full BEIR results
The paper reports complete GLUE and BEIR evaluations with task-specific fine-tuning searches. BEIR results cover both single- and multi-vector retrieval, using average performance across selected datasets to choose final models.
- The main body reports average performance over BEIR’s 15 diverse datasets, with subset results provided separately for both retrieval settings.
- The reported GLUE scores are dev-set results, with prior-model values extracted from the literature.
- Single-vector and multi-vector retrieval results are reported in Tables 7 and 8, respectively, using BEIR nDCG@10 evaluation.
- Final retrieval models were selected by sweeping learning rates and choosing the best average result on NFCorpus, SciFact, TREC-Covid, and FiQA.
F Efficiency
ModernBERT is evaluated for inference efficiency using synthetic datasets and runtime, memory, and maximum-batch-size measurements. It achieves substantially stronger memory efficiency and faster processing than DeBERTaV3 in the reported comparisons.
- Efficiency evaluations use synthetic datasets, with token statistics reported in Table 10 and runtimes and maximum batch sizes reported in Table 11.
- ModernBERT achieves considerably higher maximum batch sizes than other models, highlighting strong memory efficiency at both evaluated model sizes.
- DeBERTaV3 uses 5-to-7 times more memory than ModernBERT on both model sizes.
- The paper releases the ModernBERT architecture, model weights, and training codebase under the Apache 2.0 license.