PaperlayerView exact revision on arXiv

Source-linked AI summary

BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef, Céline Hudelot, Pierre Colombo

arXiv:2604.02045v1cs.CLcs.AI

TL;DR

Existing causal-to-bidirectional adaptation lacks consensus on effective objectives, scalable data-free adaptation, and flexible composition with specialized models. BidirLM addresses these gaps through controlled adaptation, linear merging with lightweight multi-domain data, and composition with specialized causal models, yielding encoders that outperform open-source alternatives across text, vision, and audio representation benchmarks.

  • Problem

    Existing methods lack consensus on adaptation objectives and leave causal models’ specialized knowledge underused for representation tasks.

  • Method

    The framework uses masking and contrastive adaptation, linear weight merging with lightweight multi-domain data, and composition with specialized causal models.

  • Results

    BidirLM’s five encoders outperform open-source alternatives on text, vision, and audio representation benchmarks.

  • Takeaways & Limitations

    A fully open-source recipe can transform causal decoder LLMs into bidirectional encoders spanning text, vision, and audio domains.

  • Takeaways & Limitations

    The framework’s omnimodal scaling remains incompletely studied for data composition, hard-negative mining, and scaling behavior.

Abstract

from arXiv · show

Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks.

1 Introduction

BidirLM adapts causal LLMs into bidirectional encoders while addressing unclear adaptation choices and enabling composition with specialized causal variants. The released multilingual family outperforms open-source alternatives on text, vision, and audio representation benchmarks.

  • Motivation: Causal LLMs underpin specialized code, mathematics, safety, vision, and audio models, yet representation tasks remain tied to bidirectional encoders (He et al., 2023).This motivates repurposing causal architectures and their broader ecosystem for representation learning.
  • Adaptation design: Controlled ablations disentangle adaptation choices and show that enabling bidirectional attention through a masking objective is critical for performance.The masking phase is often omitted by existing methods, which conflate training objectives with attention mechanisms.
  • Model composition: Weight merging composes adapted encoders with specialized causal variants, transferring safety, audio, and vision capabilities without full pipeline retraining.This approach uses the open-source specialization developed around the same causal backbone.
  • Released models: The released multilingual BidirLM series outperforms open-source alternatives on text, vision, and audio representation benchmarks.The family includes Gemma3-based BidirLM-270M/1B, Qwen3-based BidirLM-0.6B/1.7B, and multimodal BidirLM-Omni-2.5B, with the training corpus, checkpoints, and variants released.

2 Experimental Setup

The experiments adapt Gemma3 and Qwen3 causal models into bidirectional encoder variants using open-source corpora, then evaluate them across fine-tuning, embedding, and causal-model composition tasks. The setup spans English, multilingual, mathematical, code, image-text, and audio-text data and tests transferred safety, vision, and audio capabilities.

  • Models and adaptation objectives: The study adapts Gemma3 (270M, 1B) and Qwen3 (600M, 1.7B), using smaller models for ablations and larger models for scaling analysis.Five variants are formed by enabling bidirectional attention and applying MNTP, contrastive, or sequential MNTP-then-contrastive adaptation.
  • Adaptation corpus: Adaptation uses open-source corpora spanning English, multilingual text, mathematics, and code, with contrastive training drawing on KaLM-embedding and 89 merged datasets.Masking uses FineWeb-Edu, FineWeb2-HQ, FineMath, and Stack V2; English contrastive training uses seven hard negatives per query.
  • Adaptation corpus: Omni-Contrastive provides 1.8M mixed pairs: 65% text-text, 17.5% audio-text, and 17.5% image-text.Audio-text data comes from Laion-Audio-300M and LibriSpeech ASR, while image-text data comes from Colpali, NatCap, and MSCOCO.
  • Evaluation protocol: Evaluation combines full-parameter fine-tuning on XTREME and task-specific benchmarks with zero-shot or linear-probe embedding tests on text, image, and audio benchmarks.Fine-tuning covers information retrieval, sequence classification, and token classification; embedding evaluation uses English and Multilingual MTEB v2, MIEB lite, and MAEB beta.
  • Causal ecosystem: Post-adaptation composition transfers specialized causal-model capabilities for safety moderation, visual-textual entailment, and audio comprehension.The study uses Qwen3Guard-Gen-0.6B, Qwen3-VL-2B-Instruct, and Qwen3-ASR-0.6B, evaluated on Beaver, Safe, Aegis, e-SNLI-VE, and BoolQ.

3 Adaptation Strategies

The experiments show that bidirectional attention requires a prior MNTP adaptation phase to deliver broad gains, while masking and contrastive objectives are complementary across embedding and fine-tuning regimes.

  • Experimental setup: The study evaluates five adaptation variants on Gemma3-270M and Qwen3-0.6B using 10B masking tokens and 3M contrastive-training samples.Downstream performance is reported relative to the unmodified causal baseline, Base, in Figure 2.
  • Bidirectional attention drives performance: A prior MNTP phase enables bidirectional attention to improve performance across all evaluated tasks, including notable gains on XNLI and Seahorse.Fine-tuning with bidirectional attention alone produces mixed results, improving token classification and retrieval but degrading XNLI and Seahorse; MNTP unlocks the full benefit. Figure 2 reports Gemma gains of +0.8 on XNLI and +9.0 on Seahorse, and a Qwen XNLI gain of +2.7.
  • Masking and contrastive objectives are complementary: Contrastive training outperforms Bi+MNTP on MTEB by over 13 points across both architectures, confirming its advantage for generic embedding evaluation.The comparison covers zero-shot and linear-probing evaluation and is consistent with prior work (Gao et al., 2021; Li et al., 2023; BehnamGhader et al., 2024).
  • Masking and contrastive objectives are complementary: Contrastive-only adaptation sacrifices fine-tuning quality despite embedding gains, whereas restoring a prior MNTP phase achieves peak performance across both regimes.The controlled comparison identifies masking and contrastive objectives as complementary rather than interchangeable.

4 Scaling Adaptation Phases

Scaling adaptation from 10B to 30B tokens causes domain-specific forgetting, while combining 50% weight merging with a lightweight 20% multi-domain mixture best preserves foundational knowledge and bidirectional capabilities.

  • Scaled Adaptation Induces Forgetting: Scaling MNTP adaptation from 10B to 30B tokens on English data induces forgetting across domains: Gemma loses Arabic performance, while Qwen loses math and code performance.Gemma declines by -7.0 points on Arabic MIRACL and -2.0 on XNLI; Qwen declines by -1.5 on Math Shepherd and -2.0 on CodeSearchNet.
  • Model Merging Mitigates Forgetting and Preserves Bidirectional Capabilities: A 50% linear merge of adapted and original checkpoints substantially improves cross-domain performance while preserving bidirectional capabilities.The adapted and base models remain close in weight space, with average cosine similarities of 0.78 for Gemma and 0.97 for Qwen; merging yields +6 points on Arabic MNLI and code for Gemma and +4 points in math for Qwen.
  • Multi-domain data mixtures and weight merging yield optimal retention: The final Multilingual+Merge configuration achieves the best overall results by combining a 20% multi-domain mixture with interpolation against the original base weights.Performance plateaus at 20–30% multi-domain data, so 20% is fixed for subsequent experiments; the combined configuration yields further gains and the best overall results.
  • Multi-domain data mixtures and weight merging yield optimal retention: Together, weight merging and a lightweight multi-domain data mixture preserve the base models’ foundational knowledge and newly acquired bidirectional capabilities.The mixture distributes data equally across multilingual, math, and code domains, while merging provides a data-free complementary strategy for mitigating forgetting.

5 Frontier Performance Through Scaled Adaptation

Scaling the biphasic adaptation pipeline to Gemma3 and Qwen3 produces BidirLM encoders that redefine open-source frontiers on task-specific and generic embedding benchmarks. The same models achieve these results with classical contrastive training, without proprietary-model distillation or costly multi-run averaging.

  • Scaled adaptation: The scaled BidirLM series comprises Gemma3 models at 270M and 1B parameters and Qwen3 models at 0.6B and 1.7B, trained contrastively on 10M multi-domain samples.These final models execute the second step of the biphasic pipeline to establish general-purpose embedding capabilities.
  • Task-specific benchmarks: All BidirLM variants establish a new performance frontier on augmented XTREME under full-parameter fine-tuning; BidirLM-270M matches mmBERT-base with 10% fewer parameters, while BidirLM-0.6B exceeds EuroBERT-610m by more than 1 point.The comparisons are reported for the scaled Gemma3 and Qwen3 BidirLM variants.
  • Generic embedding benchmarks: The exact same BidirLM variants advance the open-source Pareto frontier across three of four size configurations on MTEB generic embedding benchmarks.This removes the traditional separation between generic embeddings and task-specific fine-tuned model variants.
  • Evaluation limitations: BGE-M3, KaLM, and EmbedGemma could not be evaluated because they lack architectural support for sentence or token classification.This limitation affects embedding-only models in the benchmark comparison.
  • Training and baseline scope: BidirLM achieves these embedding results using only classical contrastive training, avoiding proprietary-model knowledge distillation and costly multi-run averaging.The models are positioned as robust open-source baselines for future comparisons with closed-source systems such as Qwen3-Embedding and EmbeddingGemma.

6 Domain and Modality Specialization

Weight merging composes domain- and modality-specialized causal models into adapted bidirectional encoders, enabling rapid adaptation and strong performance even without shared prior modalities. The resulting BidirLM-Omni-2.5B sets new omnimodal benchmarks while requiring only limited additional training compute.

  • Domain Specialization: The merged domain-specialized encoder outperforms every baseline by over 1 point on average, while improving out-of-distribution generalization and training stability on Beaver, SAFE, and Aegis.Its low variance between raw and EMA-smoothed measurements indicates stable optimization.
  • Domain Specialization: Domain specialization is sample-efficient: the merged model reaches over 93% of peak performance across benchmarks within 20 steps, or 80 samples, and leads alternatives by more than 5 points at that stage.The procedure merges the encoder with Qwen3Guard-Gen and fine-tunes for 500 steps on Beaver.
  • Modality Specialization: Merged modality-specialized encoders exceed Bi+Specialist by over 1 point on vision and 15 points on audio, and surpass unmerged baselines by over 30 and 19 points, respectively.They require warm-up periods of 100 vision steps and 175 audio steps to align representations with newly introduced modality heads.
  • Modality Specialization: Weight merging succeeds even without shared modalities, with the vision specialist remaining competitive but the unimodal audio specialist degrading substantially against the merged model.The difference is attributed to the vision model’s existing text-vision capabilities versus the audio model’s speech-only training.
  • Omnimodal Composition: BidirLM-Omni-2.5B outperforms Nemotron-Omni-3B across all modalities, gaining +17 on text and +5 on image benchmarks while using nearly half the parameters.It also ranks first among baselines on MIEB and third on MAEB, establishing new Pareto frontiers across data-transparency settings.
  • Omnimodal Composition: Composing existing specialized models required only 250 additional MI250X GPU hours, allowing omnimodal encoders to be assembled incrementally instead of retraining the full pipeline.The model merges textual backbones from ASR, vision-language, and bidirectional variants, then appends their audio and visual heads before contrastive training.

7 Related Work

Related work adapts causal LLMs for generic and multimodal representations through masking-based bidirectionality or other adaptation strategies, while post-hoc weight merging offers an efficient response to catastrophic forgetting during model specialization.

  • Adapting Causal Models for Generic and Multimodal Representations: Causal LLMs have become strong backbones for text embeddings, with adaptation strategies generally organized around two paradigms.The cited work includes Ma et al. (2023), Liu et al. (2024b), and Springer et al. (2025).
  • Adapting Causal Models for Generic and Multimodal Representations: One adaptation paradigm injects bidirectionality through masking objectives, using classical masked language modeling (MLM) or the next-token variant MNTP before fine-tuning.The passage attributes MLM to Devlin et al. (2019) and MNTP to BehnamGhader et al. (2024).
  • Weight Merging for Knowledge Transfer and Specialization: Adapting models to new objectives and distributions risks catastrophic forgetting, a problem traditionally addressed through compute-intensive replay buffers or regularization.The passage cites French (1999), Rolnick et al. (2019), and Wang et al. (2024b).
  • Weight Merging for Knowledge Transfer and Specialization: Post-hoc weight merging, including Model Soups and Task Arithmetic, provides a highly efficient alternative for adapting models to new objectives and distributions.The passage cites Model Soups and Task Arithmetic (Ilharco et al., 2023).

8 Conclusion

The paper introduces a unified, fully open-source framework for transforming causal decoder LLMs into bidirectional encoders across text and multiple modalities. Systematic comparisons show that masking is critical for fine-tuning, while scaling uses linear weight merging and a lightweight multi-domain data mixture without proprietary pre-training data.

  • 8 Conclusion: The framework transforms causal decoder LLMs into bidirectional encoders spanning text and multiple modality domains.It is presented as unified and fully open-source.
  • 8 Conclusion: Systematic comparisons show that the masking phase omitted by recent contrastive-only methods is critical for fine-tuning performance.
  • 8 Conclusion: The adaptation is scaled without proprietary pre-training data through linear weight merging and a lightweight multi-domain data mixture.

Future Work

Future work will systematically study contrastive training in omnimodal settings and investigate stronger mitigation techniques beyond lightweight linear merging and data mixing. These directions include richer regularization, cross-architectural distillation, and analyses of data composition, hard negatives, scaling, and compute costs.

  • Contrastive training: Future studies should examine omnimodal contrastive training through data composition, hard-negative mining, and scaling analyses.The paper’s ablations focused on the frequently omitted masking phase, while contrastive training has already received substantial prior study (Zhang et al., 2025; Vera et al., 2025; Hu et al., 2025).
  • Additional mitigation techniques and model architectures: The framework could adopt richer regularization, especially knowledge distillation from the base model, to improve knowledge retention beyond linear merging and data mixing.Cross-architectural distillation techniques (Minixhofer et al., 2025) may strengthen retention but require additional compute.

A Base Model Architecture Details … B.2 Hyperparameters

The appendix details the diverse decoder-only base architectures and the adaptation objectives, while specifying controlled training procedures and hyperparameter-selection protocols for masking and contrastive learning.

  • A Base Model Architecture Details: Gemma3 (Gemma et al., 2025) and Qwen3 (Yang et al., 2025a) are decoder-only transformer families whose differing architectural choices test framework generality.They differ in attention patterns, normalization layers, vocabulary sizes, and pre-training configurations.
  • B.1 Loss definitions: Masked language modeling randomly masks tokens and reconstructs them using full bidirectional context.The masked input replaces selected tokens with a special [MASK] placeholder.
  • B.1 Loss definitions: Masking probability is evaluated at 10%, 20%, 30%, and 40%, while masked next-token prediction combines masked reconstruction with causal next-token prediction.MNTP predicts each masked token from the logits at the preceding position, using the MLM masking notation and hyperparameters.
  • B.1 Loss definitions: Contrastive adaptation uses InfoNCE with in-batch and explicitly mined hard negatives to align representations of semantically equivalent sequences.Sequence representations use either last-token selection or mean pooling from final-layer hidden states, with cosine similarity and temperature τ.
  • B.2 Hyperparameters: All runs use identical data order, one epoch of unique tokens or sentence pairs, and fixed seed 42 to enforce fair reproducibility.Masking processes 1B tokens, whereas contrastive adaptation processes 1M samples.
  • B.2 Hyperparameters: Learning rates are selected by grid search over 10 log-spaced values from 1 × 10−5 to 1 × 10−3 using training loss.Selection uses 1B tokens for masking and 1M samples for contrastive training; Tables 2 and 3 summarize the respective hyperparameters.

C Adaptation Data Details … F Additional Results

The appendix specifies the adaptation and evaluation data, decontamination and deduplication procedures, cross-task aggregation, fine-tuning protocols, and modular construction of the omnimodal BidirLM model.

  • C Adaptation Data Details: The adaptation mixture combines educational web, multilingual, mathematical, code, embedding, parallel, and instruction-aware retrieval data, with cited dataset sizes and language coverage.FineWeb-Edu contains 1.3T tokens; FineMath contains 54B tokens; The Stack v2 covers over 3B files across 600+ languages; FineWeb2-HQ spans 20 languages.
  • C Adaptation Data Details: Training uses single-domain batches, asymmetric or symmetric instruction placement, and mined hard negatives tailored to each query’s task structure.For retrieval and reranking, instructions are prepended only to queries; for STS and pair classification, they are applied to anchors and positives.
  • C Adaptation Data Details: MTEB decontamination removes 13 overlapping domain families, leaving 10 110 219 training pairs, while NeMo-first deduplication retains higher-quality Nemotron hard negatives over overlapping KaLM sources.The decontamination total is reported in Table 4; overlapping datasets include MIRACL, MS MARCO, TriviaQA, SQuAD, NFCorpus, GooAQ, and PAQ.
  • D Details of Evaluation; D.1 Downstream Task Evaluation; D.2 General Embeddings Evaluation: Evaluation covers downstream fine-tuning and frozen-embedding or lightweight-probe assessment across text, multilingual, image, and audio benchmarks.The general embedding suite includes MTEB English, MTEB Multilingual, MIEB image, and MAEB audio evaluations.
  • D.1 Downstream Task Evaluation: Downstream evaluation spans sequence classification, retrieval, regression, token classification, augmented XTREME tasks, and specialist safety, vision-language, and audio-text tasks.Metrics include F1, NDCG@10, Spearman correlation, and average score, with datasets such as XNLI, PAWS-X, MS MARCO, MIRACL, SeaHorse, PAN-X, and POS.
  • D.3 Aggregating Performance Across Tasks: Performance is aggregated by rescaling each task’s scores to a model-relative rank, where lower mean rank indicates consistently stronger performance across heterogeneous metrics.The best model receives rank 0 and the worst receives rank |M| −1 before arithmetic averaging across tasks.
  • D.4 Evaluation Fine-Tuning Protocol: Text fine-tuning uses batch size 32, learning-rate selection from 10 log-spaced values between 5 × 10−6 and 5 × 10−3, warmup, decay, task-specific step budgets, and early stopping.Retrieval, regression, and classification use 1k, 5k, and 10k steps respectively; small datasets use validation-based patience-one early stopping.
  • E BidirLM-Omni Model Composition; Qwen-1.7B Bi+MNTP; F Additional Results: BidirLM-Omni-2.5B is constructed by equally merging Qwen3-VL-2B, Qwen-1.7B Bi+MNTP, and Qwen3-ASR-1.7B textual backbones into a unified omnimodal representation space.Figure 10 describes a modular composition strategy using a 1/3 linear weight merge for the vision, text, and audio variants.

F.1 Masked Language Objectives and Hyperparameters … F.6 MTEB, MIEB, and MAEB (2026-03-30 Snapshot).

The appendix finds MNTP adaptation, balanced specialist merging, and weight merging to be central to robust bidirectional and omnimodal encoders. It further documents model similarities and benchmark evaluations across text, vision, and audio tasks without supplying specific leaderboard winners or scores in the provided passages.

  • F.1 Masked Language Objectives and Hyperparameters: Bi+MNTP consistently outperforms Bi+MLM across tasks and architectures except Seahorse for Gemma at 20% and 30%, with masking optima depending on objective and model.Bi+MNTP achieves higher mean performance at every corresponding masking ratio; its best average occurs at 30% for Qwen3-0.6B and 40% for Gemma3-270M, while Bi+MLM typically peaks at 20–30%.
  • F.2 Details on Model Similarities: Layer-wise cosine analysis shows bounded but architecture-dependent displacement after MNTP adaptation, with mean similarity 0.78 for Gemma3-270M versus 0.97 for Qwen3-0.6B.MLP projections deviate slightly more than Self-Attention projections; among 1.7B multimodal variants, Qwen3-VL is closer to the adapted encoder than Qwen3-ASR, with mean cosine 0.97 versus 0.96.
  • F.3 Performance by Merging Ratio with Causal Specialists: A 50% merging ratio is the strongest default for combining encoders with causal specialists, while cross-modality merges are more sensitive to unbalanced ratios.Within a common modality, 25% or 75% can remain robust; across distinct modalities, those same ratios cause significant performance drops, favoring a balanced split.
  • F.4 Effect of Merging on Omnimodal Performance: Merging the specialized causal backbones improves BidirLM-Omni performance over a non-merged baseline that uses only Bi+MNTP weights with frozen multimodal heads.The comparison covers average scores across MTEB Multilingual V2, MIEB lite, and MAEB beta after contrastive training.
  • F.5 Detailed Results Across Models and Benchmarks: Tables 5–7 report per-task-type results for four text-only BidirLM encoders and BidirLM-Omni-2.5B across MTEB, MIEB, and MAEB.The tables cover text-only MTEB performance alongside omnimodal vision and audio benchmark performance, but the supplied passages provide no cell values.
  • F.6 MTEB, MIEB, and MAEB (2026-03-30 Snapshot).: Tables 8–10 rank the paper’s models and open-data baselines by mean task performance across MTEB, MIEB, and MAEB, with best column scores bolded.The provided table captions identify leaderboard scope and zero-shot ratios for MTEB, but do not expose specific numerical winners or values.
Loading 2604.02045v1…