TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling

• Low-resource languages and specialized domains often lack the data needed for effective NLP models; Hindi has no life sciences PLM despite being spoken by over 600M people.The introduction also notes recent life sciences models for German, Italian, and French, and similar data barriers in legal, finance, and patent domains.Source: §1 · p. 1
• The TransCorpus-trained French life sciences language model achieves state-of-the-art performance on downstream tasks in the DrBenchmark benchmark (Labrak et al., 2024b).The demonstration uses a high-resource translation direction.Source: §1 · p. 1
• TransCorpus is an open-source toolkit built on fairseq (Ott et al., 2019) for generating synthetic domain-specific corpora in up to 100 languages, with scalable API/CLI, multi-GPU and multiprocessing support, and checkpoint recovery.Source: §1 · pp. 1–2
• The releases include a 36.4GB synthetically translated French life sciences corpus and tokenizer, a pre-trained model, and reproducible pre-training and fine-tuning code.The toolkit and resources are publicly available; the authors say they are adding languages for future release.Source: §1 · pp. 1–2

2 Related Work

• Isbister et al. (2021) found that multilingual PLMs generally performed best for sentiment analysis in four low-resource Scandinavian languages, though translated data with an English PLM often beat a monolingual low-resource PLM.The strategies compared native-language monolingual fine-tuning, English translation followed by English-PLM fine-tuning, and multilingual-PLM fine-tuning on native data.Source: §2 · p. 2
• Lothritz et al. (2022) partially translated unambiguous words for Luxembourgish, and LuxemBERT improved over mBERT, though the gain was not statistically significant.The study compared Luxembourgish-only BERT, Luxembourgish-German BERT, and LuxemBERT trained on mixed corpora.Source: §2 · p. 2
• Urbizu et al. (2023) found that a Basque PLM trained only on Spanish-to-Basque synthetic data was competitive but did not beat native-only training, while adding translations to native data improved performance.This work followed the introduction of ElhBERTeu.Source: §2 · p. 2
• Phan et al. (2023) used synthetic biomedical parallel text to improve English-to-Vietnamese MT, create ViPubmed and ViMedNLI, and support ViPubMedT5’s state-of-the-art results on several biomedical NLP tasks.Their MT system outperformed strong baselines after synthetic parallel text was injected via self-training.Source: §2 · p. 2
• Ishigaki et al. (2023) found that Japanese BERT models using translated Web of Science abstracts performed better on entity and relation extraction than models trained on native abstracts.Training data combined 2.5M translated abstracts with 1.2M native Wikipedia abstracts; the study did not compare against existing PLMs.Source: §2 · p. 2

3.1 Machine Translation Framework & Model Selection

• M2M-100 (Fan et al., 2020) enables direct translation among 100 languages, while fairseq supports multilingual deployment and multi-GPU processing.Fairseq’s modularity also allows model swapping and integration of specialized models.Source: §3.1 · pp. 2–3

3.2 Model Size & Context Length

• The 12B model was excluded because quality gains entail quadratic computational cost; TransCorpus adopted the 1.2B model, whose quality advantage outweighed negligible sentence-length translation-time differences.The 418M and 1.2B models were compared, while the 12B model was not considered for the experiments.Source: §3.1 · p. 3, §3.2 · p. 3
• Because transformer attention cost rises quadratically with context and M2M-100 was trained on sentence pairs, the authors compared sentence- and document-based translation on 1,000 abstracts.No context would reduce translation to word-by-word output, while longer sequences increase time and memory demands.Source: §3.2 · p. 3
• Sentence-based translation was consistently faster, scaled linearly with data volume, and better preserved translated-document word-count distributions; document-based output showed repetition.The comparison used a sample of 1,000 life-sciences abstracts.Source: Figure 1 · p. 3, §3.2 · p. 3

3.3 Framework Translation Workflow

• The workflow distributes 22M abstracts across machines, buckets equal-length sentences to reduce padding, rejoins translated sentences to abstracts, and joins sentences shorter than 10 characters to neighbors.The toolkit uses fairseq for tokenization and concatenates subprocess outputs to reconstruct the corpus.Source: §3.3 · pp. 3–4
• The CLI supports corpus download, preprocessing, translation, and bilingual preview, alongside concurrent GPU/process work, checkpoint recovery, and configuration-based addition of domains.The life-sciences corpus was the only one ready for download at the time described.Source: §3.3 · p. 4

4 TransCorpus-bio-fr: A French Life Sciences Corpus

• DrBenchmark provides life-sciences evaluation datasets (Labrak et al., 2024b), while DrBERT supplies a French life-sciences PLM (Labrak et al., 2023), enabling practical assessment of the framework.DrBERT was pre-trained from scratch without relying on a general-domain model such as CamemBERT, as is also the case for most languages.Source: §4 · p. 4

4.1 MEDLINE/PubMed Abstracts Collection

• Filtering the 2021 MEDLINE/PubMed Baseline Repository and updates through April 2021 for citations with a PMID, title, and abstract yielded about 22M abstracts.The repository encompassed 31M citations; selected raw text was normalized by replacing each whitespace sequence with a single space.Source: §4.1 · pp. 4–5

4.2 Translated Corpus Statistics

• 36.4GB of translated text contains 221M sentences and 5.25B words, making TransCorpus about five times larger than DrBERT’s French life-sciences pretraining corpus.Translation took roughly 15 days using 32 NVIDIA Tesla V100 GPUs (about 11,520 GPU hours); the translated corpus had a density profile similar to the original, and a random manual review found no irregular events such as repetitions.Source: Table 1 · p. 5, §4.2 · p. 5

5.1 TransTokenizer Training

• SentencePiece was trained as a 32k Unigram tokenizer with 0.9995 character coverage, using whitespace-aware handling suited to diverse languages.The original implementation was used with default vocabulary-size and character-coverage values.Source: §5.1 · p. 5

5.2 Pre-training Hyperparameters

• TransBERT uses a 12-layer BERT encoder with 12 attention heads of dimension 768, trained on TransCorpus for 500k MLM steps following RoBERTa.Training used an 8k batch size, Adam, 24k warm-up steps, and a learning rate of 6e-4.Source: §5.2 · p. 5

6.1 TransTokenizer-bio-fr Training

• Tokenizer training on TransCorpus-bio-fr took approximately 12 hours on a single machine; because SentencePiece tokenizers require considerable RAM, 10M translated abstracts were randomly selected to train a domain-specific tokenizer.Source: §6.1 · pp. 5–6

6.2 CmTransBERT: Tokenizer Ablation

• The tokenizer ablation compares TransBERT-bio-fr using TransTokenizer-bio-fr with CmTransBERT using CamemBERT’s tokenizer, holding corpus and hyperparameters constant.Pseudo-perplexity per token and word was computed on 50 authentic French abstracts before fine-tuning; the paper says this confirmed successful pre-training and enabled the experimental phase.Source: §6.2 · p. 6

6.3 Pre-training Statistics

• TransBERT-bio-fr processed 4B sequences over 500k steps at batch size 8k, versus DrBERT’s 310M sequences over 78k steps at batch size 4k.TransCorpus-bio-fr is about five times larger than DrBERT’s corpus, while TransBERT-bio-fr has thirteen times more overall training data updates. CamemBERT bio has fewer updates than DrBERT but continues pre-training CamemBERT, which was already trained for 100k steps at batch size 8k.Source: §6.3 · p. 6

7.1 Baseline Models

• CamemBERT and DrBERT were selected as strong French PLM baselines; CamemBERT-bio was excluded because it derives from CamemBERT rather than being trained from scratch.The framework targets settings where domain-specific data are available in one language but resources in the target language are lacking.Source: §7.1 · p. 6

7.2 DrBenchmark: An Adaptation

• The adapted DrBenchmark added hyperparameter optimization, data cleaning, dataset merging, and iterative k-fold cross-validation for more robust evaluation.It comprised 15 tasks: five classification, six NER, two POS, and two STS tasks.Source: §7.2 · p. 6

7.3 Statistical Testing

• Significance was tested at dataset and task levels; Friedman followed by Nemenyi compared more than two models, while Wilcoxon compared two, following recommended practice.Dataset-level comparisons covered labels and folds; task-level comparisons covered labels, folds, and datasets.Source: §7.3 · p. 6

8 Results & Discussion

• Across 15 datasets, TransBERT leads in 10 cases (significant in four), CamemBERT in five (one significant), while DrBERT never leads and ranks last in 11.In DiaMed, TransBERT has the highest F1-score for 55 labels across five folds, compared with 22 for CamemBERT and 27 for DrBERT.Source: Table 2 · p. 7, §8 · p. 7
• In classification, TransBERT’s DiaMed and MorFITT advantages are significant, while CamemBERT’s two dataset wins are not.The reported distinctions on DiaMed and MorFITT are significant in both metrics and ranking.Source: §8 · p. 7
• NER results favor TransBERT across datasets; POS scores are closely matched, and STS results favor CamemBERT and TransBERT over DrBERT.NER differences are significant twice; the lowest POS weighted F1-score is 97.56, while CamemBERT’s significant POS win covers 107 tags across all five folds. In STS, CamemBERT and TransBERT achieve three and two top results, respectively, and DrBERT trails by more than 10 points in DEFT-2020/T1.Source: §8 · p. 7

8.1 Aggregated Results by Task

• Aggregated task metrics favor TransBERT in classification and NER, show a slight POS margin, and find no significant CamemBERT–TransBERT STS difference; DrBERT finishes last significantly.With the second, more precise classifier, DrBERT has the poorest results in 9 of 10 metrics. The paper suggests its 24 varied pre-training sources and sequences too short to convey document context may cause confusion; it also notes that MBR has been used with synthetic data for English biomedical pre-training.Source: §8.1 · pp. 7–8, Table 3 · p. 8

8.2 Tokenizer Ablation Study

• The controlled tokenizer ablation finds TransBERT generally outperforms CmTransBERT, with statistical significance only in NER.The study says its replication addresses unequal training durations and unclear reduced-batch-size choices in prior work, and describes itself as the first rigorous examination of tokenizer effects on domain-specific pre-trained language models.Source: §8.2 · p. 8

• TransBERT-bio-fr outperforms existing state-of-the-art models across life-sciences classification, NER, POS, and STS tasks.Source: §9 · p. 8
• The evaluation framework applies HPO, repeated training, 5-fold cross-validation, and statistical testing across 15 datasets aggregated by task.Source: §9 · p. 8
• The work releases the TransCorpus translation toolkit, TransCorpus-bio, TransBERT-bio-fr, and pre-training and fine-tuning code on GitHub.Source: §9 · p. 8

10 Future Work

• Future work would test the approach across underrepresented life-sciences languages and other domains, including multilingual models that use cross-lingual transfer to make scarce data go further.These studies would probe generalizability, constraints, and adaptability through the toolkit.Source: §10 · p. 8
• Comparisons with generative LLMs on identical datasets could assess performance, efficiency, and cost-effectiveness across life-sciences tasks and explore combining both approaches’ strengths.The comparison would clarify trade-offs between specialized domain-focused models and more general, resource-heavy LLMs.Source: §10 · pp. 8–9
• Generative LLMs could create synthetic training data for domain-specific models as an alternative to translation, with studies comparing its quality, reliability, and biases against translated data.LLM-generated data may offer more varied and nuanced datasets that capture intricate domain knowledge and linguistic patterns.Source: §10 · p. 9

10.1 Baseline Model

• Although TransBERT-bio-fr outperformed CamemBERT, DrBERT’s performance after proper pre-training and continuation of CamemBERT on TransCorpus-bio-fr remain untested.The authors also identify a comparison with CamemBERT bio as an open question.Source: §10.1 · p. 9

10.2 In-Domain/Language Generalization

• Broad benchmarks cannot establish performance across every life-sciences task, and model suitability depends on context amid continuing NLP advances.The authors call for context-specific evaluation and comparison with new architectures, pre-training techniques, and fine-tuning strategies.Source: §10.2 · p. 9

10.3 Other Domains Generalization

• Success in life sciences should not be assumed for other specialized domains, where terminology, concepts, and translation quality may differ.Finance, law, and engineering may pose distinct linguistic challenges; culture-specific or legally bound concepts may be difficult to translate, so broader applicability needs further research.Source: §10.3 · pp. 9–10

10.4 Other Languages Generalization

• Transfer to other languages depends on MT quality and language-pair characteristics; low-resource translation can hallucinate (Guerreiro et al. (2023)).Differences in syntax, morphology, and writing systems can hinder preservation of semantic and domain-specific nuance, while cultural or scientific context may be lost or biased; suitability across languages requires further evaluation.Source: §10.4 · p. 10

A.1 Example of an English Abstract; A.2 Corpus Statistics

• The appendix pairs an English abstract and an MBR citation example with English life-science corpus statistics, whose token counts use a cased BERT tokenizer.The supplied captions identify the examples and the statistic, but give no corpus values.Source: §A.1 · p. 13, Figure 3 · p. 13, Table 5 · p. 13

A.3 Example of a Translation with Repetition; A.4 Example of a Tokenized Abstract

• Illustrations show a translation example with repetition and an abstract tokenized at both sentence and word levels.The translation example is captioned as 418M, by abstract, with repetition.Source: Figure 4 · p. 13, Figure 5 · p. 14

A.5 Example of a Translated Citation

• A translated MBR citation example presents a French title and abstract, with McPhie (1975) identified in the figure caption.Source: §A.5 · p. 14, Figure 6 · p. 14

A.6 Translation Examples Compared to True French Abstracts

• Three paired examples compare original and translated French abstracts on xenobiotic hepatotoxicity, tuberculosis diagnostics, and patient therapeutic education.The passages provide the French texts but do not report a formal quality assessment.Source: §A.6 · pp. 15–16

A.7 Tokenizers Comparison Example

• A tokenizer example indicates that TransTokenizer uses fewer tokens than CamemBERT to encode the same sequence.Source: Figure 7 · p. 16

A.8 Pseudo-Perplexity Comparison Across Models

• A table caption identifies a cross-model pseudo-perplexity comparison and says the highest uncertainty is highlighted, but supplies no values.Source: Table 7 · p. 16

A.10 Statistical Testing

• The statistical-testing appendix describes Friedman testing with Nemenyi post-hoc comparisons for more than two models, and Wilcoxon testing for two.Source: Figure 8 · p. 16

A.9 Downstream Tasks Summary

• The DrBenchmark adaptation evaluates models by cross-validation on 15 datasets spanning varied tasks; its label key distinguishes classes, label counts, tags, and entities.The table caption also defines CC as Clinical Cases and says datasets are listed alphabetically.Source: Table 8 · p. 17