PaperlayerView exact revision on arXiv

Source-linked AI summary

En-ViMedNER: An English-Vietnamese Parallel Biomedical Corpus with UMLS Semantic Type Annotations

Nhu Vo, Phuong Nguyen, Nu Uyen Phuong Le, Inigo Jauregi Unanue, Dung D. Le, Massimo Piccardi, Wray Buntine

arXiv:2608.29890v1cs.CL

TL;DR

Vietnamese biomedical NER lacks a large parallel corpus with UMLS semantic annotations comparable to English resources. En-ViMedNER addresses this gap through a hybrid corpus-construction pipeline and evaluates Vietnamese and cross-lingual NER, finding that target-language supervision and fine-tuned encoder-decoder models outperform their alternatives in the respective settings.

  • Problem

    Vietnamese biomedical NER resources remain limited in scale and entity-type coverage, while no comparable parallel corpus with UMLS semantic annotations exists for Vietnamese.

  • Method

    The paper constructs En-ViMedNER from MedMentions using translation, expert post-editing, LLM-assisted label projection, and human verification, then benchmarks Vietnamese and English-input/Vietnamese-output NER systems.

  • Results

    Vietnamese-supervised encoders outperform English-supervised multilingual encoders and prompt-based LLMs, while fine-tuned encoder-decoder models outperform prompt-based LLMs for cross-lingual NER.

  • Takeaways & Limitations

    En-ViMedNER provides parallel UMLS semantic annotations and supervision for Vietnamese biomedical NER and cross-lingual learning.

  • Takeaways & Limitations

    Sentence-level annotation can make context-dependent translations less natural and create surface-form ambiguity, limiting reliable use on clinical notes, patient-facing text, or safety-critical applications without additional validation.

Abstract

from arXiv · show

Biomedical Named Entity Recognition (NER) is fundamental to healthcare AI applications, including clinical decision support and medical information extraction. While corpora with Unified Medical Language System (UMLS) annotations, such as MedMentions, have driven progress in English biomedical NER, no comparable resource exists for Vietnamese. This paper presents En-ViMedNER, the first English-Vietnamese parallel biomedical NER corpus annotated with UMLS semantic types, which are language-neutral codes providing a shared cross-lingual label space and ensuring direct comparability with existing UMLS-based resources. The corpus contains 4,392 PubMed abstract pairs, 44,892 English-Vietnamese sentence pairs, and 202,949 aligned entity-mention pairs across 21 semantic types adapted from the MedMentions ST21pv dataset. To balance quality and scalability, we have constructed the corpus through automatic translation, expert post-editing, LLM-assisted label projection, and human verification and adjudication. We characterize En-ViMedNER as a large-scale silver-standard corpus with a human-audited and consensus-corrected mini-test subset. We evaluate En-ViMedNER in two settings: (i) Vietnamese-input/Vietnamese-output biomedical NER and (ii) English-input/Vietnamese-output cross-lingual NER. For Vietnamese NER, we benchmark Vietnamese-supervised encoder models, English-supervised multilingual encoder models, and prompt-based LLMs. The best model achieves an F1 score of 52.70 on the test set and 53.78 on the mini-test set. For cross-lingual NER, we benchmark encoder-decoder models and prompt-based LLMs. The best model achieves an F1 score of 45.44 on the mini-test set. We publicly release our corpus, corpus construction pipeline, and baseline models to facilitate future Vietnamese biomedical NLP research.

1 Introduction

Vietnamese biomedical NER lacks large, ontology-grounded resources comparable to English corpora. En-ViMedNER addresses this gap with a parallel UMLS-annotated corpus, broad benchmarks, and released resources.

  • Research gap: Vietnamese biomedical NER resources remain limited in scale, entity-type coverage, and ontology grounding.Existing resources include task-specific or spoken-data label sets rather than broad UMLS semantic types.
  • Corpus contribution: En-ViMedNER is presented as the first English-Vietnamese parallel biomedical NER corpus with UMLS-based semantic types for Vietnamese text.The corpus projects semantic type annotations across aligned English-Vietnamese sentence pairs.
  • Corpus contribution: The corpus contains 4,392 PubMed abstract pairs, 44,892 sentence pairs, and 202,949 aligned entity-mention pairs across 21 semantic types.It is derived from MedMentions and constructed through translation, post-editing, label projection, verification, and adjudication.
  • Quality and scope: En-ViMedNER is a large-scale silver-standard resource, while exhaustive independent verification and consensus adjudication are limited to a 450-sentence mini-test subset.The corpus supports both Vietnamese biomedical NER and English-input/Vietnamese-output cross-lingual NER.
  • Evaluation: The paper benchmarks Vietnamese-supervised encoders, English-supervised multilingual encoders, prompt-based LLMs, and cross-lingual encoder-decoder systems.Cross-lingual experiments compare direct generation with cascaded translation-and-NER pipelines.
  • Evaluation and reproducibility: The authors introduce an LLM-as-a-judge protocol and publicly release the corpus, construction pipeline, and benchmark setup.The evaluation protocol targets semantic equivalence when exact span matching is too restrictive for translated or paraphrased entities.

2 Background and Related Work

The work builds on UMLS and MedMentions while addressing the limited availability of UMLS-annotated Vietnamese biomedical NER data. It combines translation-based corpus expansion with cross-lingual NER evaluation.

  • UMLS and semantic types: UMLS integrates biomedical vocabularies and uses semantic types as standardized label categories for biomedical NER.Its Semantic Network contains 127 semantic types and 54 semantic relations.
  • MedMentions: MedMentions contains 4,392 PubMed abstracts with expert-linked UMLS concept annotations spanning 126 of 127 semantic types.The corpus includes 352,496 mentions, 34,724 unique concepts, and reported inter-annotator agreement of 97.3%.
  • MedMentions: MM-ST21pv filters MedMentions to 21 clinically relevant semantic types, providing a more tractable label space while retaining broad biomedical coverage.En-ViMedNER translates these abstracts and projects their semantic types onto Vietnamese text.
  • Vietnamese biomedical NER: Vietnamese biomedical NER resources include terminology and medical-speech datasets, but no existing corpus provides UMLS semantic-type annotations.The paper addresses this gap by deriving the first such resource from MedMentions ST21pv.
  • Cross-lingual transfer: Cross-lingual transfer is useful because biomedical annotations are concentrated in English, but language distance, terminology variation, and span-boundary differences can affect performance.English-Vietnamese biomedical NER must accommodate changes in terms and entity spans across languages.
  • Cross-lingual transfer: En-ViMedNER extends translation-based corpus expansion by preserving ontology labels and aligning entity spans after expert post-editing.The paper evaluates multilingual encoder transfer and compares prompt-based generation with fine-tuned encoder-decoder systems in direct and cascaded settings.

3 Corpus Construction

En-ViMedNER is constructed by translating and post-editing English biomedical sentences, projecting labels into Vietnamese, and applying automated and human quality checks. The resulting data are split into standard partitions, with a randomly sampled mini-test subset audited by annotators.

  • Translation and post-editing: 44,892 English sentences are translated into Vietnamese and post-edited by Vietnamese medical experts for terminology, fluency, and biomedical meaning.Experts correct mistranslations and unnatural phrasing using medical terminology references and English–Vietnamese dictionaries.
  • Label projection: LLM-assisted projection maps each English entity to its Vietnamese span and assigns the same semantic label.The projected annotations are stored in templates retaining English spans and labels while filling Vietnamese annotation fields.
  • Quality control: Automatic consistency checks flag missing entities, offset mismatches, overlaps, and duplicate annotations before human review.Verified spans are converted into BIO-tagged data for experiments.
  • Quality control: 450 test sentences are randomly sampled as a mini-test set for human evaluation rather than treated as an additional split.The sample represents approximately 5% of the full test set and is intended to estimate aggregate projection quality.
  • Quality control: Cohen’s κ is 0.83 for pooled sentence-level judgments, while 98.32% exact-match accuracy measures pre-correction projected annotations.Disagreements and confirmed errors are resolved by consensus in the final mini-test annotations.

4 Experiments

The experiments evaluate En-ViMedNER for Vietnamese-input NER and English-input/Vietnamese-output cross-lingual NER. Each task compares supervised model families with prompt-based LLMs using task-specific metrics and evaluation protocols.

  • Task 1 - Vietnamese NER: Task 1 evaluates Vietnamese biomedical NER using Vietnamese-supervised encoders, English-supervised multilingual encoders, and prompt-based LLMs.The comparison tests whether Vietnamese supervision provides benefits beyond English-supervised transfer and prompting.
  • Task 2 - Cross-lingual NER: Task 2 generates Vietnamese NER-tagged translations from plain English sentences using fine-tuned encoder-decoder models and prompt-based LLMs.Encoder-decoder systems compare direct generation with translation-and-NER cascades.
  • Evaluation metrics: Task 1 reports entity-level precision, recall, and F1, while Task 2 additionally measures translation quality with Sacre-BLEU.Task 2 reports Exact-F1 for exact surface and label matching and Judge-F1 for semantically matched entities.
  • Evaluation metrics: The LLM-as-a-judge protocol addresses valid Vietnamese translations or paraphrases that exact span matching would reject.The judge matches predicted and gold entities and evaluates span and label correctness, missed entities, and extra predictions.
  • Evaluation metrics: Human–LLM agreement is assessed on 100 randomly sampled NLLB-M1 outputs using the same entity-matching criteria.Cohen’s κ is used to measure consistency between human judgments and the LLM judge.

5 Results

Vietnamese supervision outperforms English-supervised transfer and prompting for Vietnamese NER, while fine-tuned encoder-decoder models lead cross-lingual NER. Cross-lingual results also show that translation quality and NER quality can diverge.

  • Task 1 - Vietnamese NER: 52.70 F1 is achieved by ViPubMedDeBERTa-base on the full test set, exceeding XLM-R-large by 16.16 points.On the mini-test set, ViPubMedDeBERTa-xsmall reaches 53.78 F1 versus 37.95 F1 for XLM-R-large.
  • Task 1 - Vietnamese NER: 36.47 F1 is reached by gemini-3.1-flash-lite-preview with 3-shot prompting, up from 12.70 F1 zero-shot, but prompting remains below fine-tuning.The best prompted model remains 17.31 points below the best Vietnamese-supervised encoder on the mini-test set.
  • Task 2 - Cross-lingual NER: Cohen’s κ = 0.87 supports the reliability of the LLM-as-a-judge protocol for cross-lingual evaluation.Table 3 reports precision, recall, Judge-F1, Exact-F1, and invalid-output counts on the mini-test set.
  • Task 2 - Cross-lingual NER: 45.44 Judge-F1 is achieved by NLLB-M1, outperforming the best closed-source LLMs by 2.80 and 3.89 points.NLLB-M3 instead achieves the highest Exact-F1 at 15.98, above the best closed-source LLM result of 10.40.
  • Task 2 - Cross-lingual NER: Direct generation performs best under Judge-F1, whereas NER→Trans. performs best under Exact-F1 for both encoder-decoder models.NLLB-M1 reaches 45.44 Judge-F1, while NLLB-M3 reaches 15.98 Exact-F1.
  • Task 2 - Cross-lingual NER: Higher BLEU does not always imply better NER because sentence-level translation quality may not preserve entity boundaries and labels.For both NLLB and umT5, M2 has the highest BLEU while M1 has the best F1.

6 Conclusion and Future Work

The paper introduces En-ViMedNER as a parallel English-Vietnamese biomedical NER corpus with UMLS semantic annotations and evaluates its value for Vietnamese and cross-lingual NER. Future work targets additional Southeast Asian languages and expanded model and pipeline settings.

  • Conclusion: En-ViMedNER contains 4,392 PubMed abstracts and 202,949 aligned entity-mention pairs across 21 UMLS semantic types.Its construction combines machine translation, expert post-editing, LLM-assisted projection, and human verification.
  • Conclusion: Vietnamese-supervised encoders outperform English-supervised multilingual encoders and prompt-based LLMs, while fine-tuned encoder-decoder models outperform prompting cross-lingually.The parallel annotations support learning translation, entity-boundary preservation, and biomedical label transfer.
  • Future work: Future work will examine portability to Indonesian and Thai and investigate open-source LLM fine-tuning, judge reliability, and cascaded medical translation-NER pipelines.These directions use emerging medical NER resources and extend the evaluated modeling settings.

Limitations

En-ViMedNER has limitations involving context, surface-form consistency, and domain coverage. Its use in clinical or safety-critical settings requires additional validation.

  • Context and representation: Sentence-level annotation can make some Vietnamese translations appear unnatural when their meaning depends on abstract-level context.The authors give “animals” translated as laboratory animals as an example requiring broader context.
  • Context and representation: Inconsistent translation of repeated surface forms may confuse NER models and reduce surface-level learnability.Repeated forms can be translated selectively, such as one occurrence of HCC remaining unchanged while another is translated.
  • Domain coverage: Because the corpus uses formal PubMed abstracts, models trained on it may underperform on clinical notes and patient-facing health communication.
  • Downstream use: Models trained on En-ViMedNER should not be assumed reliable for clinical notes, patient-facing text, or safety-critical medical applications without additional validation.

Ethical considerations

En-ViMedNER follows the licensing conditions of MedMentions and UMLS. It redistributes translated data while limiting included UMLS information to semantic type codes.

  • Licensing: The dataset can be redistributed because MedMentions is released under CC0, subject to the licensing conditions of both MedMentions and UMLS.
  • Licensing: The release retains UMLS semantic type codes but excludes UMLS concept names, definitions, and hierarchy information.Users needing detailed UMLS content must access it through their own UMLS accounts under the official UMLS license.
  • Data sources: The corpus is derived from PubMed abstracts and MedMentions rather than clinical notes or patient records, reducing personally identifying health-information risk.

Use of AI tools

The paper uses LLM-assisted label projection within a sentence-level annotation pipeline, followed by automated consistency checks and human verification in Label Studio.

  • Label projection: LLM-assisted projection maps English entity labels onto post-edited Vietnamese translations at the sentence level.The input JSON contains aligned sentences and English entity spans, while Vietnamese annotation fields are initially left blank.
  • Automatic checking: Automatic checks flag missing entities, incorrect offsets, unexpected overlaps, duplicates, and suspicious boundary modifiers.
  • Automatic checking: Shared entity identifiers link English and Vietnamese mentions so projected annotations can be compared consistently.
  • Human verification: Flagged sentence pairs enter a review queue, where annotators compare source text, translations, spans, and labels before correcting annotations with domain-expert consultation.
  • Human verification: Label Studio displays aligned English and Vietnamese sentences with projected labels for checking span boundaries, entity text, and UMLS semantic types.

B.3 Projection quality evaluation

Projection quality is evaluated through mention text and boundary criteria on an audited mini-test, with agreement measured across annotator judgments. Exact-match accuracy is high before correction, while disagreements concentrate on boundary and text-matching cases.

  • Projection criteria: Projection quality is measured using text match, start index, end index, and exact match, which requires correct text and both boundaries.
  • Agreement procedure: 450 mini-test sentences were judged by annotators for errors across the four projection criteria, with pooled Cohen’s kappa aggregating paired ratings.
  • Projection results: 99.04% start-index accuracy and 98.27% end-index accuracy were obtained across the audited mentions.
  • Projection results: 98.32% exact-match accuracy was achieved across 2,085 mentions before correction.The reported figure measures the original GPT-projected annotations rather than the final corrected mini-test annotations.
  • Agreement and errors: The overall Cohen’s κ was 0.8315, indicating strong agreement, while most confirmed errors were boundary-related.The consensus audit identified 35 confirmed projection errors across 30 audited sentences, with boundary errors comprising 57.1%.

E Implementation Details

The implementation combines fine-tuned encoders, encoder-decoder models, and prompted LLMs across Vietnamese and cross-lingual NER settings, with task-specific selection and postprocessing procedures. Experiments also document projection-error categories and corpus label distributions.

  • Model training: Encoder models are fine-tuned for token-level sequence labeling, while encoder-decoder models generate text with inline entity tags.Encoder training uses validation F1 for checkpoint selection; NER-only encoder-decoder models use validation entity-level F1, while multi-task models use validation BLEU.
  • Prompting setup: Prompt-based LLM experiments compare zero-shot, 1-shot, and 3-shot settings using demonstrations retrieved only from the training split.Vietnamese-input settings retrieve Vietnamese examples, whereas cross-lingual settings retrieve parallel examples using English-side dense similarity.
  • Error analysis: Projection-error tables organize errors by category, semantic type, and boundary subtype.The reported categories include over-projection and under-projection or under-extension.
  • Experimental resources: The corpus label distribution is explicitly documented across its 21 entity tags, while hardware and API settings constrain reproducibility details.Fine-tuning uses two NVIDIA L40 48GB GPUs, and closed-source LLM evaluations allow up to two retries for transient API failures.

F Other results

Additional analyses show that semantic-type frequency and model size do not fully determine Vietnamese biomedical NER performance. The appendix also specifies annotation conversion, conflict filtering, LLM postprocessing, and prompting templates.

  • Label distribution: Label frequency does not fully determine F1, because rare semantic types can exceed macro-F1 while a frequent type performs worst among reported labels.T031 scores 61.19 and T005 scores 59.07, exceeding macro-F1 49.01; frequent T033 performs worst among the ten reported labels.
  • Model characteristics: Model size alone does not determine performance: ViPubMedDeBERTa-base achieves the best Micro-F1 despite being smaller than XLM-R-large.The passage also reports that ViPubMedDeBERTa-xsmall remains competitive with much larger general-purpose encoders.
  • Annotation conversion: Nested entities are reduced to outermost spans, while 128 unresolved non-nested conflicts are excluded from the current experimental setting.The excluded cases account for less than 0.3% of the full dataset.
  • Prompt templates: LLM prompting templates cover BIO conflict resolution, Vietnamese NER, and English-to-Vietnamese cross-lingual NER with optional few-shot examples.The cross-lingual task combines translation, Vietnamese tokenization, and one BIO label per token.
  • Prompt templates: Figure prompts distinguish zero-shot and few-shot configurations by including or excluding the few-shot examples section.Separate templates are provided for Vietnamese and cross-lingual experiments.
Loading 2608.29890v1…