PaperlayerView exact revision on arXiv

Source-linked AI summary

UltraBERT: Beyond the Limit of BERT Pre-Training

Jungwoo Park, Taewhoo Lee, Jaewoo Kang

arXiv:2610.15028v1cs.CLcs.AI

TL;DR

Encoder pretraining recipes have retained small-scale MLM conventions, leaving models specialized rather than competitive across classification and retrieval. UltraBERT combines low- and high-masking regimes in one encoder with separate LM heads and low-ratio-only distillation. It achieves state of the art across both task families, including 91.0 on GLUE and 47.7 on BEIR.

  • Problem

    The inherited MLM pretraining recipe leaves encoders trained with a single configuration specialized, with no single model competitive across classification and retrieval.

    Source: §1 · p. 1

  • Method

    UltraBERT mixes 5% and 50% masked sequences with separate LM heads and restricts distillation to low-ratio sequences.

    Source: §1 · p. 2

  • Results

    UltraBERT achieves state of the art on both classification and retrieval, reaching 91.0 on GLUE and 47.7 on BEIR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results indicate that encoder specialization is a consequence of the pretraining recipe rather than an inherent constraint.

    Source: §5 · p. 10

    Because the encoders may inherit biases from web-scale training data, downstream users should evaluate them for their use case.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent bidirectional encoders have been modernized with innovations from causal large language models (LLMs). However, the pretraining recipe itself remains inherited from small-scale conventions, leaving encoders underoptimized, with no single model performing competitively across both classification and retrieval tasks. In this work, we investigate how far encoder performance can be pushed by revisiting core design choices in training, revealing three key findings: (1) Encoders saturate on downstream tasks beyond a few trillion tokens, indicating that additional compute past this point does not help. (2) Across model sizes, lower masking ratios benefit classification while higher ones benefit retrieval, contradicting prior work suggesting larger ratios for larger models. (3) Distillation with pruning improves classification at low masking ratios but degrades retrieval at high ones. Guided by these findings, we present UltraBERT, a single encoder that assigns separate LM heads to low and high masking ratios within a batch and restricts the distillation objective to the low ratio. To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR. Together, these results show how far encoder performance can be pushed and provide a principled basis for compute allocation, masking ratio design, and knowledge transfer in bidirectional models.

1 INTRODUCTION

The paper revisits encoder pretraining because the inherited MLM recipe leaves models specialized, rather than competitive across classification and retrieval. It reports that downstream performance saturates at a few trillion tokens, that the tasks favor different masking ratios, and introduces UltraBERT to combine those regimes.

1 INTRODUCTION

  • Bidirectional encoders remain useful for non-generative applications, but their inherited MLM pretraining recipe leaves single-configuration models specialized across task families.The paper frames this as a classification–retrieval trade-off.Source: §1 · p. 1
  • Downstream performance saturates around 1–2T tokens even as training and validation losses continue to decrease.The paper reports this pattern after training encoders on up to 8.8T tokens.Source: §1 · pp. 1–2
  • At saturation, lower masking ratios favor classification across model sizes, while higher ratios favor retrieval.This contrasts with prior work suggesting larger models benefit from larger masking ratios.Source: §1 · p. 2
  • Pruning and distillation from a stronger teacher improve classification at a low masking ratio but degrade retrieval at a high one.Source: §1 · p. 2
  • UltraBERT achieves state of the art on both classification and retrieval, reaching 91.0 on GLUE and 47.7 on BEIR.It combines 5% and 50% masked sequences in each batch with separate LM heads, and applies distillation only to low-ratio sequences.Source: §1 · p. 2

2 RELATED WORK

Recent encoder work has adopted architectural and scaling advances, while retaining older MLM pretraining conventions. Related work examines masking ratios, retrieval-oriented training, and distillation, but leaves important interactions with saturation and task family unresolved.

Modernized encoders

  • Recent encoders adopt LLM-era innovations, but their MLM objective and masking ratio remain inherited from BERT-era conventions.The cited modernization work includes RoPE, GLU, and alternating local and global attention (Portes et al., 2023; Nussbaum et al., 2025; Zhang et al., 2024; Warner et al., 2025; Breton et al., 2025; Weller et al., 2026).Source: §2 · p. 2

Masking ratio and pre-training objectives

  • Prior masking-ratio studies report benefits from ratios above 15%, especially for larger models, but use 10B–100B-token budgets that entangle masking effects with convergence speed.RTD is considered stronger for classification but known to degrade retrieval; prior comparisons used conventional 15% masking without varying corruption level (Clark et al., 2020; Rep et al., 2024; Warner et al., 2025).Source: §2 · p. 2

Encoders for retrieval

  • Retrieval bi-encoders encode texts as single vectors and typically combine MLM pretraining with contrastive fine-tuning, yet retrieval-focused encoders tend to lag on classification.Prior work has extensively optimized fine-tuning, while the backbone pretraining recipe is described as determining the resulting model’s upper bound (Reimers & Gurevych, 2019; Wang et al., 2022; Li et al., 2023; Nussbaum et al., 2025; Zhang et al., 2024).Source: §2 · p. 3

Knowledge distillation for language models

  • Encoder distillation has used reconstruction logits, hidden states, attention scores, and attention-vector Gram matrices, while whether logit-based cascade distillation transfers to encoders remains unexplored.Cascade Distillation initializes a small model from pruned parent weights before distilling from a stronger teacher (Muralidharan et al., 2024; Liu et al., 2026).Source: §2 · p. 3

3 EXPLORING THE PRE-TRAINING RECIPE FOR STRONGER ENCODERS

The section tests encoder pretraining choices at saturation, finding that classification and retrieval prefer opposite masking ratios. UltraBERT combines both regimes with decoupled LM heads and applies distillation only at the low ratio.

3 EXPLORING THE PRE-TRAINING RECIPE FOR STRONGER ENCODERS

  • Controlled experiments use 150M, 380M, and 1.3B encoders with a shared data mixture and tokenizer, evaluating classification and retrieval benchmarks.The section compares GLUE and SQuAD 2.0 for classification, and BEIR after MS MARCO fine-tuning for retrieval.Source: §3 · p. 3

3.1 TRAINING TOKEN SCALING AND DOWNSTREAM SATURATION

  • Downstream performance saturates around 1–2 trillion tokens even as training and validation losses continue to decline.BEIR saturates earlier, with no significant improvement beyond the first evaluation at 275B tokens.Source: Figure 2 · p. 3, §3.1 · p. 3

3.2 MASKING RATIOS AND CLASSIFICATION TASKS

  • Across model sizes, classification performance consistently degrades as masking ratios increase, contrary to prior results based on 10B–100B-token budgets.The authors attribute the difference to evaluating at saturation, which removes convergence speed as a confound.Source: Figure 4 · p. 4, §3.2 · p. 4
  • RTD’s classification advantage over MLM peaks around 15% corruption and diminishes toward both lower and higher ratios.At 50%, the generator has too little context for informative replacements; at 3%, plausible alternatives differ little from [MASK].Source: Figure 5 · p. 5, §3.2 · p. 5

3.3 MASKING RATIOS AND RETRIEVAL TASKS

  • Across model scales, 50% masking achieves the highest BEIR average nDCG@10, opposite to classification’s preference for lower ratios.Effective rank tends to rise with masking ratio, but KoLeo regularization at 6% improves BEIR less than 50% masking and also harms GLUE.Source: §3.3 · pp. 5–6, Figure 6 · p. 6

3.4 MIXING MASKING RATIOS WITH LM HEAD DECOUPLING

  • The 6% and 50% masking regimes produce clearly different LM-head geometries, while their input embeddings remain almost identical.The 6% LM head has more concentrated singular values, consistent with its lower effective rank.Source: Figure 7 · p. 6, §3.4 · p. 6, Figure 8 · p. 7
  • 89.4 on GLUE and 45.1 on BEIR with mixed 5% and 50% masking and decoupled heads, matching the best single-ratio scores of 89.5 and 45.0, respectively.Without decoupling, the mixed model scores 88.9 on GLUE and 44.7 on BEIR; the authors suggest shared heads create a conflict between masking regimes.Source: §3.4 · p. 7, Table 1 · p. 7

3.5 REVISITING PRUNING AND DISTILLATION

  • 90.2 to 90.8 on GLUE after 15% masking distillation, while 50% masking distillation lowers BEIR from 47.2 to 46.5.The authors therefore restrict distillation to the low masking ratio.Source: Table 2 · p. 8, §3.5 · p. 8

4 ULTRABERT: BEYOND THE LIMIT OF BERT PRE-TRAINING

UltraBERT combines low- and high-masking training in one encoder and achieves leading performance across classification and retrieval benchmarks at both model sizes. The evaluation covers classification, retrieval, code and math retrieval, and long-context benchmarks.

4 ULTRABERT: BEYOND THE LIMIT OF BERT PRE-TRAINING

  • UltraBERT mixes 5% and 50% masked sequences with separate LM heads, applying distillation only to the low-masking sequences.The 150M and 380M models are pruned from a 1.3B teacher; the paper also reports saturation at 250B tokens for 15% masking.Source: §4 · p. 8

4.1 EVALUATION SETUP

  • Classification evaluation includes GLUE, SQuAD 2.0, MultiCoNER v2, Few-NERD, and RewardBench, with benchmark-specific hyperparameter search.Retrieval evaluation includes BEIR, RAR-b code and math subsets, MLDR, and LongEmbed.Source: §4.1 · p. 9

4.2 RESULTS

  • Both UltraBERT scales lead on LongEmbed, while code and math retrieval gains over the second best reach 7.3 and 8.3 points at base and 8.4 and 7.7 at medium.The series natively supports 32K context length.Source: §4.2 · p. 9
  • UltraBERT is the first encoder to lead across the full benchmark range at both scales, though it does not win every individual benchmark.Base-size exceptions include MultiCoNER v2; at medium size, the exception is MLDR.Source: §4.2 · p. 9, Table 3 · p. 9

5 CONCLUSION

The paper concludes that encoder performance can be improved through large-scale pretraining choices that account for task-specific masking preferences. It reports broad benchmark gains, documents ethical considerations, and describes its AI-use and reproducibility practices.

5 CONCLUSION

  • UltraBERT combines masking ratios with decoupled LM heads and low-ratio-only distillation, achieving state-of-the-art classification and retrieval results at 150M and 380M with 32K context.The conclusion frames masking specialization as a consequence of the training recipe, not an inherent encoder trade-off.Source: §5 · p. 10

AI USE STATEMENT

  • The authors used generative AI to help formalize an author-designed proof and edit the paper, while authors wrote the training and evaluation code and checked AI-assisted work.They state that AI was not used to develop the conceptual framework, design experiments, implement methods, or interpret results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The work uses publicly available datasets and no human-subject data, but its web-scale training data may carry biases that downstream users should evaluate for their use case.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper identifies appendices for architecture, data, evaluation protocols, proofs, and per-task results, and states that evaluation code, model weights, and the pretraining data mixture are released.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 ARCHITECTURE AND OPTIMIZATION

The appendix specifies the shared encoder architecture, optimization setup, and iterative pruning and distillation procedure used to create smaller models.

A.1 ARCHITECTURE AND OPTIMIZATION

  • All models use 24 layers with RoPE, SwiGLU, and alternating local and global attention, while model size scales primarily through width.The 150M, 380M, and 1.3B models have hidden sizes of 768, 1,024, and 2,048, respectively.Source: §A.1 · p. 22
  • Training uses AdamW and a warmup-stable-decay learning-rate schedule; UltraBERT trains on 2T tokens and extends context to 32K for the final 100B tokens.Models use a 4M-token batch and are trained on 256-chip TPU pod slices.Source: §A.1 · p. 22

A.2 PRUNING AND DISTILLATION

  • Cascade Distillation iteratively prunes the 1.3B teacher to 380M and then 150M, retaining all 24 layers and selecting components using activation-based importance scores.The calibration set consists of 2,000 pretraining-mixture documents packed into 1,024 sequences of 1,024 tokens.Source: §A.2 · p. 22
  • When halving head dimension, pruning preserves RoPE frequency pairing and rescales queries; shared random orthogonal projections are used for value and output projections.The query scaling keeps attention-logit magnitude stable, while the projection preserves the value-output product in expectation.Source: §A.2 · p. 22
  • Distillation applies KL divergence on 5% masked sequences and standard MLM loss on 50% masked sequences, with both smaller models distilled from the 1.3B teacher.Source: §A.2 · pp. 22–23

A.3 DATASET MIXTURE

The paper describes a mixed pretraining corpus, reports that staged changes to its data proportions barely affected downstream fine-tuned performance, and outlines benchmark evaluation protocols.

A.3 DATASET MIXTURE

  • The pretraining mixture is dominated by web data, followed by code, academic papers, and mathematics, with instruction data accounting for 2.5%.The mixture table reports unique and trained token counts, with category ratios in parentheses.Source: §A.3 · p. 23, Table 4 · p. 23
  • The small instruction-data share supports some instruction following, including zero-shot and few-shot multiple-choice question answering through the [MASK] position.The authors report recognizing the MCQA format without tuning.Source: §A.3 · p. 23
  • Preliminary multi-stage mixture schedules barely affected downstream performance after fine-tuning, so the authors use a single mixture throughout training.They hypothesize that dataset exposure affects token-prediction distributions and apparent zero- and few-shot performance more than the linguistic knowledge used by fine-tuning.Source: §A.3 · p. 23

B EVALUATION PROTOCOLS

  • For Table 3 baselines, the authors use literature scores when available and otherwise fine-tune and evaluate models under the same protocol.Source: B EVALUATION PROTOCOLS · p. 23

B.1 CLASSIFICATION TASKS

The appendix details fine-tuning and evaluation protocols for classification, retrieval, and encoder–decoder multiple-choice question answering. Across tasks, it specifies task-specific data, scoring, and model-selection procedures.

B.1 CLASSIFICATION TASKS

  • Classification uses per-task fine-tuning grids and selects configurations by development performance, with task-specific scoring and selection procedures for GLUE, SQuAD 2.0, NER, and RewardBench.GLUE uses development scores; NER reports selected-configuration test scores, and RewardBench averages accuracy across four sections.Source: §B.1 · p. 24

B.2 RETRIEVAL TASKS

  • Retrieval uses mean-pooled single-vector models trained with cached multiple-negatives ranking loss, then evaluates MS MARCO-tuned checkpoints on BEIR and RAR-b, with further MLDR tuning for long-context evaluation.BEIR and LongEmbed report average nDCG@10; RAR-b reports nDCG@10, with code and math retrieval evaluated over large candidate sets.Source: §B.2 · p. 25

B.3 ZERO-SHOT AND FEW-SHOT MCQA

  • The MCQA evaluation covers six benchmarks, reports accuracy averaged across benchmarks, and compares encoders and decoders using masked-token and next-token answer readouts, respectively.Few-shot prompts prepend five examples; instruction-tuned encoders predict answer tokens with the MLM head, while decoders use causal LM loss.Source: §B.3 · pp. 25–26

C.1 COLA AS A CONFOUNDER

UltraBERT substantially exceeds DeBERTaV3 on retrieval averages at both compared sizes. The section also shows that RTD sharply changes CoLA relative to other GLUE tasks, motivating its exclusion from MLM–RTD comparisons.

C.1 COLA AS A CONFOUNDER

  • After 100B tokens of RTD training, CoLA rises from 61.6 to 71.5, while the average of other GLUE tasks moves by 0.5 points.The comparison starts from a 150M encoder pretrained with 50% masking.Source: §C.1 · p. 26, Table 6 · p. 26
  • Because CoLA’s acceptability judgment resembles RTD’s token-plausibility judgment, the authors exclude CoLA when comparing MLM and RTD.The stated rationale is that CoLA reflects alignment with the pretraining objective rather than general encoder quality.Source: §C.1 · p. 26

C.2 COMPARISON WITH RTD-TRAINED ENCODERS

  • UltraBERT-150M scores 75.4 on classification and 45.3 on retrieval, versus DeBERTaV3-base’s 74.2 and 20.7; UltraBERT-380M scores 78.5 and 48.7, versus DeBERTaV3-large’s 78.1 and 25.8.The reported retrieval gaps are 24.6 and 22.9 points at the two sizes.Source: §C.2 · p. 26

D WHY KOLEO INCREASES EFFECTIVE RANK

The appendix links KoLeo regularization to higher effective rank through an entropy-to-isotropy argument. It also documents baseline-score sources and reports the full per-task GLUE pattern.

D WHY KOLEO INCREASES EFFECTIVE RANK

  • KoLeo minimizes estimated divergence from the uniform representation distribution, promoting near-isotropy, flatter singular values, and maximal effective rank.The argument connects the estimated entropy of representations to the spectrum of their batch representation matrix (Garrido et al., 2023).Source: D WHY KOLEO INCREASES EFFECTIVE RANK · pp. 26–28

E.1 REPORTED BASELINE SCORES

  • For Table 3, most baseline scores come from prior literature, while the authors evaluate the NER, RewardBench, and retrieval baselines; their GLUE protocol reproduces Ettin Encoder within 0.2 points.The appendix identifies the sources of baseline scores and reports the protocol check at both sizes.Source: §E.1 · p. 28, Table 7 · p. 28

E.2 FULL GLUE RESULTS

  • UltraBERT is best on most GLUE tasks at both sizes, with specified exceptions; it remains within 1 point of the best model except for CoLA.Exceptions are QQP and CoLA at base size, and QQP, SST-2, MRPC, and CoLA at medium size.Source: §E.2 · p. 28

E.3 FULL REWARDBENCH RESULTS

UltraBERT leads most RewardBench sections at both model sizes and achieves the top score on most BEIR datasets, though other models lead on some subsets.

E.3 FULL REWARDBENCH RESULTS

  • UltraBERT leads on three RewardBench sections at base size and two at medium; its largest margin is 4.9 points on Chat Hard for UltraBERT-380M.Ettin Encoder is slightly ahead on Chat and Reasoning at medium size.Source: §E.3 · p. 28

E.4 FULL BEIR RESULTS

  • UltraBERT leads on 10 BEIR datasets at base size and 8 at medium, while LFM2.5-Encoder-350M leads on four medium-size datasets.The four are NFCorpus, SciFact, TREC-COVID, and CQADupstack.Source: §E.4 · p. 28

E.5 FULL LONGEMBED RESULTS

UltraBERT leads LongEmbed across all base tasks and most medium tasks, with particularly strong Passkey performance through 32K tokens.

E.5 FULL LONGEMBED RESULTS

  • UltraBERT is best on all six LongEmbed tasks at base size and all but Needle and 2WikiMultihopQA at medium size.LFM2.5-Encoder-230M leads on those two medium-size exceptions; the largest gains are on Passkey and Needle, which fully use 32K context.Source: §E.5 · p. 28, Table 11 · p. 31
  • UltraBERT alone retains 99% Passkey accuracy at every document length up to 32K, at both sizes.512-context baselines fail beyond 1K tokens, while 8K-context models drop sharply at 16K and 32K; LFM2.5-Encoder falls below 90% at 32K.Source: §E.5 · p. 29, Figure 11 · p. 29
Loading 2610.15028v1…