Enriching Biomedical Knowledge for Low-resource Language Through Large-Scale Translation

1 Introduction

• The authors introduce ViPubmedT5, a pretrained encoder-decoder Transformer trained on synthetic translated Vietnamese biomedical text.The paper also describes this as the first encoder-decoder Transformer model pretrained on large-scale synthetic translated biomedical data.Source: §1 · p. 2
• ViMedNLI is a Vietnamese medical NLI task translated from MedNLI and refined by human experts.The source dataset is MedNLI (Romanov and Shivade, 2018).Source: §1 · p. 2
• ViPubmedT5 outperforms general-domain and health-specific Vietnamese pretrained models on biomedical tasks.The compared models are Nguyen and Nguyen (2020), Phan et al. (2022), and Minh et al. (2022).Source: §1 · p. 2
• The work presents a state-of-the-art English–Vietnamese translation model for medical and general domains.Source: §1 · p. 2
• The authors release model checkpoints, datasets, and source code for future low-resource-language studies.Source: §1 · p. 2

2 Related Works

• Biomedical machine translation has not adequately served crucial healthcare domains, and prior biomedical MT work did not target low-resource languages.The paper positions its use of machine translation to produce Vietnamese biomedical datasets as a response to this gap.Source: §2 · pp. 2–3

2.1 Pubmed and English Biomedical NLP Studies

• PubMed abstracts underpin many biomedical pretrained models and benchmark datasets, and all top 10 BLURB models use them for pretraining.The paper describes PubMed as containing more than 34 million biomedical abstracts and being updated yearly.Source: §2.1 · p. 3

2.2 English-Vietnamese Translation

• VietAI curated 4.2M high-quality training pairs and its translation model exceeded existing English–Vietnamese models by more than 2% BLEU.English–Vietnamese translation remains low-resource because high-quality parallel data are limited.Source: §2.2 · p. 3

3 Improvements on Biomedical English-Vietnamese Translation through Self-training

• Self-training leverages domain-specific English monolingual data, but generated Vietnamese targets may be low-quality and affect translation performance.Source: §3 · p. 3
• Self-training uses bTA to translate 1M English–Vietnamese biomedical abstract pairs, expanding the combined MTet and PhoMT bitext from 6.2M to 7.2M sentence pairs.The new pairs come from PubMed abstracts and are added to existing high-quality translation datasets.Source: §3 · p. 3
• 6.61 BLEU (38.69->45.61) on MTet Medical and 0.53 BLEU (45.47->46.01) on PhoMT General mark self-training gains across both test sets.The authors report that the model achieves state-of-the-art results on the PhoMT general-domain test set and that gains extend beyond biomedical translation.Source: Table 1 · p. 4, §3 · p. 4

4 ViPubmed

• The authors apply bTB to generate the first large-scale synthetic translated biomedical corpus for Vietnamese.The preceding section describes bTB as the English–Vietnamese translation system developed for biomedical translation.Source: §4 · p. 4
• They use the newest PubMed collection, with approximately 34 million English biomedical abstracts, to include up-to-date research.The abstracts are parsed from structured XML.Source: §4 · p. 4
• Abstracts longer than 512 tokens are filtered out, and a 20M-abstract subset is translated using the bTB model.The authors choose the subset for a fair size comparison with other health-related Vietnamese pretrained-model datasets and leave a larger subset for future release.Source: §4 · p. 4

5 ViMedNLI

• ViMedNLI was created as a translated and human-refined NLI benchmark, chosen because NLI is less prone to biomedical entity translation errors than NER or relation extraction.The corpus creation process is shown in Figure 2.Source: §5 · p. 4, Figure 2 · p. 4

5.1 MedNLI

• MedNLI uses premise–hypothesis pairs labeled as entailment, contradiction, or neutral by two board-certified radiologists, and the authors translated its 14,522 released samples across the same splits.MedNLI is grounded in patients’ medical histories and draws premises from MIMIC-III.Source: §5.1 · p. 5, §5.2 · p. 5

5.2 Dataset Challenges

• Clinical notes contain abbreviations, inconsistent punctuation, and misspellings that challenge neural machine translation and complicate production of a high-quality Vietnamese dataset.The paper identifies these as features of the clinical-note sublanguage.Source: §5.2 · p. 5

5.3 Human Refining

• Human refining used detailed annotation guidelines and checked abbreviation expansions against Vietnamese medical websites, blogs, and dictionaries, while correcting spelling and mistranslated meanings.Examples include correcting “herm” to “term” and changing a mistranslation of “post-term delivery” that reversed its meaning.Source: §5.3 · p. 6

6 ViPubmedT5

• The authors use unlabeled translated ViPubmed data and the ViMedNLI benchmark to pretrain and finetune a Transformer model for Vietnamese biomedical language.This section describes the model and its pretraining settings.Source: §6 · p. 6

6.1 Model Architecture

• ViPubmedT5 uses a Transformer encoder–decoder, ViT5 checkpoints, and the T5 framework; the reported base setting has 220 million parameters.Larger model sizes are left for future work.Source: §6.1 · p. 6

6.2 Pretraining

• The model is continuously pretrained from Vietnamese ViT5 checkpoints for another 500k steps on 20GB of translated biomedical data.The paper states that this approach aims to learn biomedical context while maintaining Vietnamese language representation.Source: §6.2 · pp. 6–7
• Pretraining uses span masking: random text spans are replaced with sentinel tokens, and the target concatenates those sentinels with the original masked spans.The objective follows Raffel et al. (2019).Source: §6.2 · p. 7

7.1 Benchmark dataset

• The benchmark datasets are acrDrAid, FAQSum, and ViMedNLI, used to fine-tune and evaluate ViPubmedT5.The paper says dataset statistics are given in Table 3.Source: §7.1 · p. 7
• ac rDrAid uses Vinmec radiology reports for acronym disambiguation; three expert radiologists annotated its 135 acronyms and 424 expansion texts.The task is to identify an acronym’s expansion from its report context.Source: §7.1 · p. 7
• FAQ Summarization pairs questions from healthcare-site FAQ sections with their titles as target summaries.Source: §7.1 · p. 7

7.2 Baseline

• The baseline set includes PhoBERT, a general Vietnamese model pretrained on a 20GB word-level news corpus, and ViT5, trained on general-domain CC100-vi.The paper describes PhoBERT as following the RoBERTa architecture and ViT5 as a state-of-the-art Vietnamese model for generation and classification.Source: §7.2 · p. 7
• ViHealthBERT provides a health-specific baseline, initialized from PhoBERT and trained on 25M health sentences mined from different sources.Source: §7.2 · p. 7

• ViPubmedT5 achieves state-of-the-art results in medical and clinical contexts but performs slightly worse than ViT5 on healthcare topics.The authors connect its stronger medical and clinical results to learning biomedical context from synthetic translated data.Source: §8 · p. 7
• On FAQSum, ViPubmedT5 approaches the current state-of-the-art results (60.6 and 61.3) while outperforming other models by a large margin (43.85).The authors associate the remaining gap to ViT5 with differences between scientific PubMed abstracts and patient–doctor dialogues.Source: §8 · pp. 7–8
• ViPubmedT5 outperforms other models on medical and clinical datasets, improving from ViT5’s 88 to 89.04 on acrDrAid and 77.85 to 81.65 on ViMedNLI.The authors attribute these gains to biomedical knowledge in the translated PubMed corpus.Source: §8 · p. 8
• On ViMedNLI, health and biomedical models outperform general-domain models, supporting its use as a robust baseline for Vietnamese BioNLP research.The authors describe the translated and human-refined dataset as high-quality with robust biomedical contexts.Source: §8 · p. 8

• Using a state-of-the-art neural machine translation system to create synthetic translated medical data is presented as scalable to many low-resource languages.The authors point to recent improvements in low-resource translation systems as making the approach more practical.Source: §9 · p. 8
• The approach is not limited to Vietnamese and is intended to enrich biomedical NLP research knowledge in multiple low-resource languages.Source: §9 · p. 8
• Recent state-of-the-art translation systems are described as making synthetic translated biomedical data more practical to produce.Source: §9 · p. 8

10 Conclusion

• ViPubMedT5 is a T5-style encoder-decoder pretrained on a large-scale translated biomedical corpus and reports state-of-the-art results on biomedical inference and acronym disambiguation.Source: §10 · p. 8
• ViMedNLI adds a machine-translated, human-expert-refined natural language inference benchmark to Vietnamese biomedical data and benchmarks.Source: §10 · p. 8

11 Limitations

• For other low-resource languages, the approach depends heavily on neural machine translation quality, and the minimum useful performance remains an open question.The authors report that the English–Vietnamese system was strong enough for their experiments.Source: §11 · p. 8

A Results for En-Vi Translation with Self-training

• The experiment tests self-training to expand biomedical bitexts and evaluates translation on Law, Religion, and News as well as Medical.The self-training model uses an extra 1M synthetic English–Vietnamese biomedical abstracts.Source: A Results for En-Vi Translation with… · p. 12
• Table 5 reports BLEU scores for English–Vietnamese translation on the MTet multi-domain test set.Source: Table 5 · p. 12
• Biomedical bitext expansion does not improve every MTet domain: results improve in Medical and Law but worsen in Religion and News.The authors attribute this pattern to the formal, academic style of PubMed abstracts, which aligns more closely with Law than with News or Religion.Source: A Results for En-Vi Translation with… · p. 12, Table 5 · p. 12