PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilled Rapid Embedding Transfer (DRET): Parameter-Efficient Biomedical Domain Adaptation via Priority-Based Embedding Transfer

Girish Sundaram, Daniel Berleant

arXiv:2609.02898v1cs.CL

TL;DR

Biomedical NLP requires domain knowledge that lightweight models lack, while large specialized models are costly to deploy. DRET transfers embeddings from specialized models into DistilBERT through progressively refined strategies, and achieves competitive PICO-classification performance with embedding-level evidence of transfer. The approach is positioned as an efficiency–performance trade-off for resource-constrained biomedical text mining.

  • Problem

    Large biomedical models are computationally demanding, while lightweight models lack the domain knowledge needed for accurate PICO classification and full-corpus adaptation can be infeasible.

  • Method

    DRET imports biomedical vocabulary and embeddings from domain-specific models into DistilBERT through tokenizer merging, averaging, priority-based selection, and optimization variants.

  • Results

    DRET-enhanced DistilBERT reaches PICO-classification performance comparable to domain-specific models and exceeds them on selected class-wise metrics.

  • Takeaways & Limitations

    DRET delivers competitive recall and balanced-accuracy behavior from a 66M-parameter model without source corpora or source-scale compute, supporting on-premise and edge deployment.

  • Takeaways & Limitations

    Evaluation is confined to EBM-NLP PICO classification, embedding diagnostics are descriptive rather than predictive, and source-priority ordering is fixed a priori.

Abstract

from arXiv · show

Large domain-specific language models such as BioBERT and ClinicalBERT achieve strong performance on biomedical NLP tasks, but their computational demands make them impractical for many real-world deployments. General-purpose, parameter-efficient models such as DistilBERT are lightweight yet lack the domain knowledge required for specialized tasks such as PICO (Population, Intervention, Comparison, Outcome) classification. We introduce Distilled Rapid Embedding Transfer (DRET), a knowledge-transfer paradigm that injects biomedical domain knowledge from large specialized models into a smaller general-purpose model without retraining on the original specialized corpora. DRET is developed as an iterative family of strategies: a unified tokenizer-merge strategy (DRET 1.x), hybrid embedding averaging (DRET 2.0), and a priority-based embedding-transfer mechanism (DRET 3.x) that hierarchically selects embeddings from the most authoritative source models, further combined with embedding-layer freezing, differential learning rates, label propagation, and imbalance-aware loss functions (DRET 4.x). We evaluate DRET on token-level PICO classification using the EBM-NLP corpus under severe class imbalance, across a twelve-metric battery. DRET-enhanced DistilBERT (66M parameters) attains balanced accuracy, recall, and ROC-AUC competitive with, and on several class-wise metrics exceeding, models an order of magnitude larger, while retaining DistilBERT's efficiency. We further show that transfer occurs at the embedding level through cosine-similarity, semantic-shift, and t-SNE analyses. DRET offers a scalable, resource-efficient route to near-domain-expert performance for biomedical text mining, with direct application to automated systematic literature reviews and clinical decision support.

1 Introduction

Biomedical NLP needs domain-specific knowledge, but large specialized models are costly to deploy and lightweight models lack the knowledge needed for PICO classification. DRET addresses this gap through embedding transfer and iterative efficiency-oriented variants.

  • Large biomedical models perform well, but their size and compute requirements limit deployment in resource-constrained settings.
  • DistilBERT is computationally light but lacks the biomedical knowledge needed for accurate PICO classification.
  • DRET transfers biomedical domain knowledge into a compact general-purpose model without retraining on the original specialized corpora.
  • The priority-based strategy consolidates vocabulary and embeddings from multiple biomedical source models using hierarchical precedence to avoid semantic dilution from naive averaging.
  • DRET variants progressively add embedding freezing, differential learning rates, label propagation, and imbalance-aware losses while characterizing their trade-offs.
  • A twelve-metric EBM-NLP evaluation and embedding analyses assess classification performance and whether transfer occurs in the representation space.

2 Related Work

Biomedical language models improve domain performance but remain computationally heavy, motivating efficient transfer approaches for biomedical NLP. DRET transfers embedding-level knowledge from multiple source models into a compact target without source-corpus retraining.

  • Biomedical BERT variants such as BioBERT, ClinicalBERT, BlueBERT, SciBERT, Med-BERT, and BiomedBERT are accurate but heavy.
  • DistilBERT reduces inference cost, while DRET differs from distillation and full fine-tuning by transferring embedding-level knowledge from several source models without retraining on source corpora.
  • The EBM-NLP PICO corpus is severely imbalanced, with the O class dominating and PICO-relevant classes comparatively rare.

3 Task, Dataset, and Models

The study treats PICO extraction as token-level sequence labeling and evaluates models on the imbalanced EBM-NLP corpus. Models include domain-specific embedding sources, base adaptation targets, and BLURB reference baselines.

  • PICO extraction assigns each token to I-PAR, I-INT, I-OUT, or O.
  • EBM-NLP contains roughly 5,000 annotated clinical-trial abstracts in CoNLL format and exhibits severe class imbalance.
  • The O class contains on the order of 9 × 10^5 tokens, while PICO-relevant tokens are comparatively rare.
  • The models comprise domain-specific embedding sources, base adaptation targets, and strong BLURB benchmark baselines.

4 The DRET Method

DRET adapts DistilBERT by merging biomedical vocabularies, initializing transferred embeddings, and selecting source embeddings by priority. Later variants protect transferred knowledge, improve supervision, and address dominant-class imbalance through optimization and loss changes.

  • 4 The DRET Method: DRET evolves from vocabulary merge to embedding averaging, priority-based transfer, optimization variants, and imbalance-aware losses.
  • 4.1 DRET 1.0: Unified Tokenizer-Merge Strategy: DRET 1.x merges vocabularies from six domain-specific models, expands the tokenizer, preserves existing embeddings, and randomly initializes newly added tokens.
  • 4.2 DRET 2.0: Hybrid Embedding Averaging: DRET 2.0 initializes new tokens with averaged source embeddings, providing semantic grounding but potentially diluting the strongest source signal.
  • 4.3 DRET 3.0: Priority-Based Embedding Transfer: DRET 3.0 ranks sources by biomedical competence and selects the highest-priority embedding whenever a token appears in multiple sources.
  • 4.3 DRET 3.0: Priority-Based Embedding Transfer: 30,522 original embeddings were overwritten, while the vocabulary expanded from 30,522 to 58,890 tokens; BioBERT modified 18,591 and BiomedBERT added 17,963 tokens.
  • 4.4 Optimization Variants: DRET 3.0.1 freezes transferred embeddings, while DRET 3.1 uses layer-wise learning rates and dynamic class weights during fine-tuning.
  • 4.4 Optimization Variants: DRET 3.2 propagates labels to all sub-tokens, preserving supervision that standard masking would discard.
  • 4.5 Imbalance-Aware Losses: DRET 4.x addresses dominant-O imbalance using weighted cross-entropy, manually adjusted weights, and focal loss focused on hard examples.

5 Experimental Setup

The study evaluates token-level PICO classification under severe class imbalance using 3-fold cross-validation and a twelve-metric battery, including aggregate and class-wise analyses.

  • 3-fold cross-validation evaluates token-level PICO classification on four classes: I-PAR, I-INT, I-OUT, and O.
  • 12 metrics cover accuracy, balanced accuracy, MCC, Cohen’s κ, F1, F2, specificity, precision, recall, G-mean, ROC-AUC, and PR-AUC.
  • Class-wise metrics, confusion matrices, precision–recall curves, and ROC curves supplement aggregate evaluation.

6 Results

DRET improves compact-model performance through priority-based transfer and imbalance-aware optimization, with different variants favoring different aggregate and class-wise metrics.

  • Baseline performance: BioMElectra achieves the highest baseline balanced accuracy at 0.792, while untuned DistilBERT is already close to domain-specific models on aggregate metrics.
  • Priority-based transfer: DRET 1.2.1 on BERT reaches the highest accuracy at 0.872, while DRET 1.0 improves over the DistilBERT baseline.
  • Imbalance-aware optimization: DRET 4.2 achieves the highest balanced accuracy (0.815), recall (0.815), G-mean (0.903), and ROC-AUC (0.865), with accuracy of 0.766.This reflects the reported trade-off between recall-oriented metrics and aggregate accuracy.
  • Imbalance-aware optimization: DRET 4.1 recovers accuracy to 0.815 from DRET 4.0’s 0.787 while preserving strong balanced accuracy.
  • Class-wise performance: DRET 4.2 dominates recall and ROC-AUC across most PICO classes, whereas priority-transfer variants win precision on selected classes.
  • Metric trade-offs: Across nine metrics, DRET 4.0–4.2 rise on recall, balanced accuracy, G-mean, and PR-AUC, while priority-transfer variants remain competitive on precision and MCC.
  • Class-wise performance: Embedding transfer and loss design address complementary axes: transfer improves representational separability, while imbalance-aware losses recover minority-class recall.

7 Does Transfer Actually Happen? Embedding-Level Validation

Embedding diagnostics indicate that DRET substantially repositions medical-token embeddings, supporting embedding-level knowledge transfer while leaving the interpretation of clustering diagnostics limited.

  • Embedding diagnostics: Cosine similarity between pre- and post-transfer medical-token vectors is 0.137, with an average semantic shift of 1.674 and increased pairwise token distance.
  • Embedding diagnostics: The t-SNE visualization shows domain-relevant tokens shifting toward biomedical neighbours after transfer.Examples include “chemotherapy”, “dose”, “trial”, and “patients”.
  • Interpretation: The cosine-similarity and semantic-shift results support that embeddings changed, but the decreased silhouette score is not treated as direct evidence of improved clustering.
  • Interpretation: Downstream class-wise gains are treated as the decisive evidence of useful transfer, while diagnostics predictive of those gains remain for follow-up work.

8 Discussion

The results support transferring biomedical knowledge into a smaller general-purpose model, with performance comparable to larger domain-specific models on PICO classification. DRET is best understood as an efficiency–performance trade-off whose practical value lies in resource-constrained deployment.

  • DRET supports transferring specialized biomedical knowledge into a smaller general-purpose model without retraining on the original corpora.
  • DRET-enhanced DistilBERT reaches performance comparable to domain-specific models and exceeds them on selected class-wise PICO metrics.
  • Figure 5 indicates substantial repositioning of medical-token embeddings after transfer through large semantic shifts and low before/after cosine similarity.
  • DRET’s priority-based transfer offers competitive recall and balanced-accuracy behaviour from a 66M-parameter model, although BLURB models retain the highest aggregate scores.
  • Faster, cheaper PICO extraction can benefit systematic literature review automation and clinical decision support through higher recall and lower deployment cost and energy use.
  • The evaluation is limited to EBM-NLP PICO classification, while multi-seed testing, significance analysis, learned priority ordering, and predictive embedding diagnostics remain open needs.

9 Conclusion and Future Work

The conclusion presents DRET as a compact-model adaptation approach for biomedical PICO classification without retraining on specialized corpora. It reports competitive performance and embedding-level evidence of transferred domain knowledge, while identifying broader extensions as future work.

  • DRET adapts a compact general-purpose model to biomedical PICO classification without retraining on specialized corpora.
  • Across twelve metrics, DRET-enhanced DistilBERT achieves balanced accuracy and recall competitive with far larger models.
  • Embedding-level analyses confirm that domain knowledge is imported into the target model’s representation space.
  • Future work includes retrieval-augmented generation, further compression, learned source-priority ordering, and extension to additional biomedical and multilingual tasks.

Reproducibility

The paper states that code and configuration for the DRET variants will accompany the preprint, while related augmentation data and conversion resources are released separately.

  • Code and configuration for the DRET variants will accompany the preprint.
  • RCT-derived augmentation data and the CoNLL conversion pipeline are released separately in a companion resource paper.
Loading 2609.02898v1…