PaperlayerView exact revision on arXiv

Source-linked AI summary

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

Rian Touchent, Eric de la Clergerie

arXiv:2605.12438v1cs.CLcs.AI

TL;DR

Adapting encoders to specialized domains usually relies on MLM, but the lasting benefit of temporarily using CLM remains unclear. This paper evaluates a CLM detour followed by short MLM decay and finds improved biomedical downstream performance over matched MLM continued pretraining.

  • Problem

    The paper asks why a temporary CLM objective switch can leave lasting benefits in encoders that use bidirectional attention at inference.

  • Method

    The paper compares standard MLM continued pretraining with a two-phase CLM detour followed by MLM decay, changing only attention masking and loss computation.

  • Results

    The CLM detour improves downstream biomedical performance over matched MLM baselines, with lasting changes concentrated in low transformer layers.

  • Takeaways & Limitations

    Low-layer modification during CLM is necessary for the downstream improvement, while mid-layer modification is not.

  • Takeaways & Limitations

    The experiments cover only French and English biomedical text, so generalization to other domains or languages remains untested.

Abstract

from arXiv · show

When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.

1 Introduction

The paper proposes a temporary CLM detour before returning to MLM for domain-adaptive encoder pretraining, outperforming standard MLM continued pretraining on biomedical text and yielding released state-of-the-art English and French encoders. It attributes the lasting benefit to necessary low-layer changes during CLM that MLM does not reverse.

  • CLM detour recipe: A CLM detour temporarily switches a pretrained encoder from MLM to CLM, then returns to MLM with matched-compute training.The released ModernCamemBERT-bio and ModernBERT-bio models cover French and English biomedical text in Base and Large sizes.
  • Mechanism: >9× larger representation changes occur in low transformer layers 0–7 during CLM than expected from seed noise alone.These changes persist after returning to MLM, even when the MLM phase matches the CLM phase in length, and divergence scales with model capacity.
  • Causal evidence: Freezing low layers 0–7 during CLM eliminates the downstream benefit, whereas freezing mid layers 8–14 preserves it.The low-layer intervention drops performance to the MLM baseline level in French experiments spanning 8 tasks and 9 seeds.
  • Practical guideline: 10% of the CLM budget suffices for the return to MLM, with this practical guideline confirmed at two model scales.The detour uses a 10% MLM decay phase after CLM.

2 Related Work

Related work establishes domain-adaptive continued pretraining as effective for domain-specific language models, especially biomedical text, while highlighting debates over initialization, context length, and training objectives. Prior analyses also motivate examining layerwise representations, transfer, plasticity, and freezing interventions.

  • Domain adaptation: Domain-adaptive continued pretraining benefits biomedical language models, particularly when the target domain differs substantially from general pretraining data.Prior work also debates whether to continue from a general checkpoint or train from scratch on domain data.
  • Architectures and context: 8,192-token ModernBERT models address BERT’s 512-token limitation for long clinical documents.BioClinical-ModernBERT trains on 53B tokens in two phases with 30% then 15% MLM masking.
  • Training objectives: A biphasic CLM-then-MLM schedule outperforms pure MLM under fixed compute, with faster early convergence and reduced fine-tuning hyperparameter sensitivity.The cited models were pretrained from scratch with 210M–1B parameters on 100B tokens.
  • Representation analysis: CKA compares layerwise internal representations and has been used to show that BERT fine-tuning primarily modifies top layers while lower layers remain stable.This provides a framework for analyzing how alternative continued-pretraining objectives change representations.
  • Layer plasticity: Transfer benefits can concentrate in lower layers, while critical learning periods and loss of plasticity may shape adaptation to new distributions.Layer-freezing interventions help identify which layers support continued learning.

3 Method

The method compares standard MLM continued pretraining with a two-phase CLM detour followed by MLM decay, using ModernBERT-family encoders across French and English biomedical settings. Layer-freezing experiments and representational analyses probe which network regions carry the CLM benefit.

  • CLM detour: The proposed pipeline replaces the initial MLM phase with causal next-token prediction, then restores bidirectional attention for a short MLM decay.The decay uses 15% masking for 10% of the CLM budget, while retaining optimizer state and resetting only the learning-rate scheduler.
  • Model configuration: All models use ModernBERT architecture in Base and Large sizes, with approximately 150M and 350M parameters, respectively.The architecture combines FlashAttention, rotary positional embeddings, alternating local/global attention, and unpadding for 8,192-token sequences; French starts from ModernCamemBERT and English from ModernBERT.
  • Baseline: The MLM baseline matches the two-phase schedule, optimizer, and masking setup except for using MLM rather than CLM in Phase 1.Phase 1 uses 30% masking for the baseline and Phase 2 uses 15% masking.
  • Layer-freezing experiments: Three French Base freeze experiments test whether CLM updates in low layers are necessary, persist through decay, and are selective relative to mid-layer updates.The experiments freeze layers 0–7 during CLM, layers 0–7 during decay, or layers 8–14 during CLM, using the 10B-CLM/1B-decay setting where the gap is largest (+2.8pp).
  • Evaluation: Performance is evaluated on 8 French and 11 English biomedical tasks using macro-averaged F1, with 9 French seeds and 5 English seeds.Representational divergence is also measured layer by layer with linear CKA on held-out biomedical texts, averaged over 3 random seeds.

4 Experiments

Across French and English biomedical benchmarks, the CLM detour outperforms MLM, with larger gains in French and at larger English scale. A 10% MLM decay is optimal, while CLM also improves long-context information retention and mid-document integration.

  • French results: 61.6% average F1: CLM beats the MLM baseline by +2.8pp across all 8 French tasks for Base models.For Large, CLM reaches 64.2% versus 63.0% for MLM (+1.2pp).
  • English results: +0.8pp: CLM’s English advantage is largest at Large scale, with 7/11 task wins, versus +0.5pp at Base 10B and +0.3pp at Base 50B.CLM wins 7 of 11 tasks at each Base scale; the English effect is smaller than in French.
  • Decay-length sweep: 10% decay gives the best performance at both Base 10B and Large 25B scales on 3 French tasks.A 2.5–4% decay scores 1.0pp below optimal, while 20–50% adds no benefit.
  • Long-context evaluation: +10.7pp overall: CLM reaches 62.2% versus 51.6% for MLM in needle-in-a-haystack evaluation across context lengths and positions.MLM falls from 57% at 512 tokens to 43% at 8192, whereas CLM degrades more slowly and remains higher, especially at mid-document positions.
  • Long-context evaluation: CLM’s long-context advantage matters because clinical documents span thousands of tokens and key information can appear anywhere.Fine-tuning results indicate that documents are long enough for mid-document context integration to matter.

5 Analysis

The analysis finds that the CLM detour leaves a lasting representational imprint concentrated in low layers, which are necessary for its downstream benefit. This imprint scales with model capacity and is asymmetric across encoder and decoder architectures.

  • Persistence: 56.5% CKA divergence remains stable after 1.5B tokens of MLM decay, varying only 0.6pp over 8B additional tokens.The seed-noise control shows 49.7% divergence overall.
  • Layer localization: Low layers 0–7 show 5–44× normalized CLM–MLM divergence, whereas mid and deep layers remain near 1×.A ratio of 1 indicates no CLM-specific effect relative to seed noise.
  • Causal intervention: Freezing low layers during CLM drops F1 from 61.6% to 59.3%, while freezing mid layers preserves 61.2%.Freezing low layers during decay has negligible effect (−0.4pp, 61.2%), indicating that MLM decay preserves the CLM imprint.
  • Post-training localization: No layer transplant recovers more than 35% of the 7.1pp DiaMED gap, because already-trained layer replacements create representational mismatches.Low and late layer transplants hurt performance, unlike freeze interventions performed during training.
  • Capacity and architecture: 67.2% ± 0.3% divergence in Large exceeds 56.5% ± 0.7% in Base, while a decoder reversal retains only 26.3% ± 1.0% divergence versus 67.2% for a similar-scale encoder.The encoder–decoder comparison suggests a 2.6× asymmetry.

6 Conclusion

A temporary CLM detour during encoder continued pretraining produces lasting representational changes that improve downstream performance beyond matched MLM baselines. The effect concentrates in low transformer layers, persists through extended MLM decay, and motivates releasing biomedical encoders in Base and Large sizes.

  • Conclusion: A temporary CLM detour improves downstream performance over matched MLM baselines using identical data and training budgets.The gains are therefore not attributed to additional compute.
  • Conclusion: The CLM imprint concentrates in low transformer layers and persists through extended MLM decay without being erased by continued training.The changes are not localized to a single layer group.
  • Conclusion: +2.8pp in French compared with +0.3–0.8pp in English depending on model size.The stronger French effect suggests potential relevance to lower-resource biomedical languages, although the paper presents this as speculation.
  • Conclusion: ModernCamemBERT-bio and ModernBERT-bio are released as open biomedical encoders in Base and Large sizes with 8,192-token context.The releases target biomedical encoding.

Limitations

The study’s evidence is limited to biomedical French and English, models up to 350M parameters, and coarse freeze experiments on French Base models. Generalization across domains, languages, larger scales, architectures, and finer layer interventions remains unverified.

  • Domain and language scope: The CLM detour has not been tested beyond biomedical French and English, where gains are +2.8pp in French and +0.3–0.8pp in English.French had 8/8 task wins and English 7/11; the English gap narrows from 10B to 50B tokens.
  • Scale and architecture: The largest model is 350M parameters, so the increasing CLM imprint—56.5% for Base and 67.2% for Large—has not been verified at billion-parameter scale.The 2.6× encoder-decoder asymmetry comparison also confounds architecture, language, and pretraining data.
  • Freeze experiments: Freeze interventions used only French Base models, and replicating them on English or Large models is needed to test generalization of the causal claim.The CLM-MLM gap is largest in French Base models.
  • Freeze experiments: The freeze granularity was coarse, using 7–8 layer blocks, so finer interventions could identify the critical layers more precisely.This limitation applies to the layer-level localization of the mechanism.

A English Training Data

The English biomedical training mix is built primarily from quality-filtered PMC Open Access paragraphs, with stronger clinical upsampling for the 10B variant. Additional instruction-following and, for 50B, clinical data complete the mixture.

  • A English Training Data: 60% of the 50B mix comes from Biomed-Enriched PMC Open Access paragraphs retained at educational-value scores ≥3.0.For the 10B variant, clinical case paragraphs are upsampled 100× and other clinical documents 10×.
  • A English Training Data: 20% of the mixture is instruction-following data from five biomedical datasets, while the 10B variant uses only MediFlow.The datasets are MedS-Ins, ReasonMed, MediFlow, UltraMedical, and EHR-Ins-Reasoning.
  • A English Training Data: 20% of the 50B-only mixture is clinical data from MIMIC-III discharge summaries and clinical notes, upsampled across multiple passes.This upsampling reaches the target token count.

B Freeze Intervention Results

Table 5 presents freeze intervention results for the French Base model across eight tasks and nine seeds.

  • B Freeze Intervention Results: Table 5 reports freeze intervention results.
  • B Freeze Intervention Results: French Base is the evaluated model.
  • B Freeze Intervention Results: The evaluation covers 8 tasks and 9 seeds.

C Component Transplant Results · D Decay Ratio Sweep

The section evaluates component-level transplants on DiaMED and sweeps decay ratios across three French tasks. The decay-ratio experiments identify 10% decay as optimal at both model scales.

  • C Component Transplant Results: Component-level transplants are evaluated on DiaMED using linear probing across 3 seeds.
  • C Component Transplant Results: The transplant analysis focuses on individual components rather than only complete-model comparisons.
  • C Component Transplant Results: DiaMED is the evaluation benchmark for the component-transplant experiments.
  • D Decay Ratio Sweep: The decay-ratio sweep covers DiaMED, FrACCO-30, and FrACCO-100.
  • D Decay Ratio Sweep: Each decay-ratio condition is evaluated with 9 seeds across the three French tasks.
  • D Decay Ratio Sweep: 10% decay is optimal at both model scales.

E Evaluation Tasks … I Needle-in-Haystack Protocol

The evaluation combines biomedical fine-tuning tasks, representational analysis with linear CKA, and a synthetic French long-context retrieval probe. Protocols specify task-specific sequence lengths, optimization settings, and controlled needle-in-haystack data generation.

  • E Evaluation Tasks: Table 8 lists the evaluation tasks and their fine-tuning maximum sequence lengths.Maximum sequence length reflects the fine-tuning configuration.
  • F Raw CKA Divergence Curves: Only CLM produces large representational changes in low transformer layers 0–7, whereas both objectives heavily modify mid and deep layers.Figure 4 compares per-layer CKA divergence for CLM versus MLM and seed noise.
  • G CKA Formula: Linear CKA compares centered, mean-pooled representation matrices through their Gram matrices K = XX⊤ and L = YY⊤.The method uses n samples and model-specific representation dimensions p and q.
  • G CKA Formula: Divergence is defined as d = 1 − CKA, so larger values indicate greater representational difference; computations use float64 arithmetic.The divergence convention is applied to the reported CKA analyses.
  • H Fine-tuning Protocol: French classification and multilabel tasks use a CLS-token linear head, while NER uses a span-based BiLSTM–CRF–biaffine architecture with task-specific training schedules.Multilabel tasks use maximum sequence length 4096 and weighted BCEWithLogitsLoss; classification uses 2048.
  • H Fine-tuning Protocol: English tasks follow the BioClinical-ModernBERT protocol, using learning rate 5 × 10−5, weight decay 0.01, batch size 16, and 10 epochs for most tasks.NER tasks use 20 epochs, and all models share the same per-task hyperparameters.
  • I Needle-in-Haystack Protocol: The French needle-in-haystack task inserts a medical fact into a clinical document and tests binary query-fact presence, with negatives differing in slot values.This design targets precise retrieval rather than shallow pattern matching.
  • I Needle-in-Haystack Protocol: The probe dataset contains 1500 balanced positive/negative pairs spanning 512–8192-token haystacks and three needle positions, split 70/15/15.Encoders are frozen while a 2-layer MLP probe is trained on CLS representations for 3 epochs, selecting checkpoints by validation accuracy.
Loading 2605.12438v1…