Source-linked AI summary
MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER
Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cambria, Luo Si, Chunyan Miao
TL;DR
Low-resource NER faces scarce labeled data and token-label misalignment from augmentation methods. MELM conditions masked entity generation on injected labels and extends it with code-mixing; experiments report gains across monolingual, cross-lingual, and multilingual settings.
Problem
Low-resource NER has limited annotated data, while augmentation methods for token-level tasks can misalign entity tokens and labels.
Source: §1 · p. 1Method
MELM fine-tunes a masked language model to replace entity tokens conditioned on injected labels, and combines with code-mixing and similarity-based entity selection for multilingual augmentation.
Source: §1 · p. 2, §2.5 · p. 5Results
Takeaways & Limitations
Novel entities are intended to provide entity regularity knowledge, and code-mixing offers an additional augmentation route when multilingual training data are available.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Data augmentation is an effective solution to data scarcity in low-resource scenarios. However, when applied to token-level tasks such as NER, data augmentation methods often suffer from token-label misalignment, which leads to unsatsifactory performance. In this work, we propose Masked Entity Language Modeling (MELM) as a novel data augmentation framework for low-resource NER. To alleviate the token-label misalignment issue, we explicitly inject NER labels into sentence context, and thus the fine-tuned MELM is able to predict masked entity tokens by explicitly conditioning on their labels. Thereby, MELM generates high-quality augmented data with novel entities, which provides rich entity regularity knowledge and boosts NER performance. When training data from multiple languages are available, we also integrate MELM with code-mixing for further improvement. We demonstrate the effectiveness of MELM on monolingual, cross-lingual and multilingual NER across various low-resource levels. Experimental results show that our MELM presents substantial improvement over the baseline methods.
1 Introduction
Because labeled NER data are scarce and token-level augmentation can misalign labels, the introduction motivates entity-focused augmentation. It presents MELM as label-conditioned entity generation and extends it with code-mixing for multilingual low-resource settings.
1 Introduction
- Limited annotated data across many languages and domains motivates low-resource NER, while manual annotation is described as expensive.The paper situates data augmentation as a way to enlarge training sets through label-preserving transformations.Source: §1 · p. 1
- Existing augmentation methods can misalign labels: word modifications may replace entities with label-mismatched alternatives, while back-translation relies on error-prone external word alignment.The cited approaches were effective on sentence-level tasks but face this issue on token-level tasks such as NER.Source: §1 · p. 1
- Prior entity substitution preserves labels but does not increase entity diversity and may produce contextually unsuitable entities; context-only augmentation leaves entities unchanged.The introduction also reports marginal improvement from context augmentation on pretrained-LM-based NER models.Source: §1 · p. 1, Figure 1 · p. 2
- MELM masks entity tokens and conditions their replacement on injected entity labels, alleviating token-label misalignment while generating diverse entities.The label tokens are inserted around entity tokens and treated as context during masked language modeling.Source: §1 · p. 2
- For multilingual low-resource NER, MELM is combined with code-mixing and bilingual-embedding similarity search to select semantically similar same-type entities across languages.Language markers are added to code-mixed data before MELM is fine-tuned to generate further augmented samples.Source: §1 · p. 2
2 Method
MELM augments low-resource NER by conditioning entity-token generation on labels, then filtering generated data before training the tagger. For multilingual settings, it adds similarity-based code-mixing and language markers.
2 Method
- MELM linearizes labeled sentences, fine-tunes on them, generates entity-diverse augmented data, and combines it with the original set to train the NER model.For multilingual settings, the framework also applies entity-similarity code-mixing to gold and code-mixed data.Source: §2 · pp. 2–3
2.1 Labeled Sequence Linearization
- Entity labels are inserted before and after entity tokens as context, with label-token embeddings initialized from semantically related words.This labeled sequence is used to fine-tune MELM so predictions are conditioned on the inserted labels.Source: §2.1 · p. 3
2.2 Fine-tuning MELM
- MELM fine-tuning masks entity tokens and maximizes their prediction probabilities while reconstructing the linearized sequence.The model uses both sentence context and label information to predict masked entities.Source: §2.2 · p. 3
2.3 Data Generation
- For generation, MELM replaces masked entities by random sampling from the top k candidates, removes label tokens, and repeats generation R rounds per sentence.Dynamic masking rates vary across rounds to produce more varied augmented data.Source: §2.3 · pp. 3–4
2.4 Post-Processing
- A NER model trained on gold data filters augmented sentences, retaining only those whose predicted labels match the original labels.The retained augmented set is combined with gold data to train the final tagger.Source: §2.4 · p. 4
2.5 Extending to Multilingual Scenarios
- For multilingual code-mixing, MELM replaces entities with the most semantically similar entity of the same class from a sampled target language.Entity similarity uses aligned MUSE embeddings; language markers are prepended to entity tokens before MELM augmentation to distinguish languages.Source: §2.5 · pp. 4–5, Figure 3 · p. 5
3 Experiments
The experiments evaluate MELM on monolingual, zero-shot cross-lingual, and multilingual low-resource NER. MELM outperforms the baselines across these settings, while ablations and comparisons examine label linearization and code-mixing.
3.1 Dataset
- Experiments use CoNLL data in English, German, Spanish, and Dutch, with 100, 200, 400, or 800 sampled training sentences per language.The multilingual setting uses 100, 200, or 400 samples from each language.Source: §3.1 · p. 5
3.3 Baseline Methods
- The comparisons include Gold-Only, Label-wise Substitution, MLM-Entity, DAGA, and MulDA as data-augmentation baselines.MLM-Entity masks entity tokens without MELM fine-tuning or labeled sequence linearization; DAGA and MulDA generate augmented data with language models.Source: §3.3 · pp. 5–6
3.4.1 Monolingual and Cross-lingual NER
- Removing labeled sequence linearization causes a considerable performance drop, supporting its role in helping MELM generate entities compatible with their labels.The comparison fine-tunes MELM on masked sentences without linearization.Source: §3.4.1 · p. 6
- At 100 and 200 samples, DAGA underperforms MELM by a large margin; the authors attribute this to disfluent, ungrammatical scratch-generated sentences, whereas MELM leaves context unchanged.DAGA is competitive at 400 and 800 samples.Source: §3.4.1 · p. 6
- MELM achieves the best averaged results across monolingual and cross-lingual low-resource levels, with absolute gains over the best baselines of 6.3, 1.6, 1.3, and 0.38 at 100, 200, 400, and 800 samples.With 100 Dutch training samples, Gold-Only almost fails to converge while MELM reaches 66.6 monolingual F1.Source: §3.4.1 · p. 6, Table 1 · p. 7
3.4.2 Multilingual NER
- MELM-gold improves substantially over Gold-Only in multilingual NER, and combining MELM with Code-Mix-ess further improves results.Both random code-mixing and entity-similarity-based Code-Mix-ess improve over Gold-Only; Code-Mix-ess gains are more significant and consistent across low-resource levels.Source: §3.4.2 · pp. 6–7, Table 2 · p. 7
4 Further Analysis
The further analysis examines whether MELM's predictions respect entity labels and whether its augmentation adds valid unseen entities. It reports that label guidance improves compatibility and MELM introduces more unseen entities than the compared methods.
4.1 Case Study
- Pretrained MLM predictions tend toward high-frequency words, and most generated words do not belong to the original entity class.The case study compares top-five predictions from MLM, MELM without linearization, and MELM.Source: §4.1 · p. 7, Table 3 · p. 8
- MELM without linearization generates more entity-related tokens than pretrained MLM, but ambiguous context alone can still yield label-mismatched predictions.The example notes that both “Pompeo” (PER) and “Reuters” (ORG) fit the context, illustrating the ambiguity.Source: §4.1 · p. 8
- MELM considers both context and entity labels, producing more entities that fit the context and align with the original label than the compared methods.The case study notes that MELM can generate real-world entities absent from the original NER dataset, including “Greenpeace” and “Amnesty.”Source: §4.1 · p. 8
4.2 Number of Unique Entities
- MELM introduces significantly more unseen entities than the compared methods after an oracle NER model filters entities whose labels do not match its assignments.The analysis counts unique entities with labels matching those assigned by an oracle trained on the full CoNLL dataset.Source: §4.2 · p. 8, Figure 4 · p. 8
5 Related Work
Prior NER augmentation approaches either modify words or generate text, but token-label alignment remains a central challenge. Existing methods address it through entity substitution, context-only modification, or generation from labeled sequences.
5 Related Work
- Word-level augmentation methods used for sentence-level tasks can cause token-label misalignment in NER, often requiring post-processing to remove noisy augmented samples.The cited approaches include synonym replacement, language models, and constituent-based tagging schemes.Source: §5 · p. 8
- Prior work addresses token-label misalignment by replacing entities with existing entities of the same class, or modifying only context while leaving entities and aspect terms unchanged.Examples cited are Dai and Adel (2020) and Li et al. (2020a).Source: §5 · p. 8
- Other approaches train or fine-tune generative language models on linearized labeled sequences to create augmented data.The cited works are Ding et al. (2020) and Liu et al. (2020).Source: §5 · pp. 8–9
- Back-translation preserves overall sentence semantics but depends on external word-alignment tools for token-level label propagation, which are often error-prone.The cited works include Sennrich et al. (2016) and Tsai et al. (2016).Source: §5 · p. 9
6 Conclusion
The paper presents MELM for low-resource NER, using labeled sequence linearization to condition masked entity prediction on entity labels. It reports gains across monolingual, cross-lingual, and multilingual settings, with code-mixing integrated for multilingual improvement.
6 Conclusion
- MELM uses labeled sequence linearization to condition masked entity-token prediction on labels, alleviating token-label misalignment while generating novel entities from pretrained knowledge.For multilingual settings, MELM is integrated with code-mixing for further performance gains.Source: §6 · p. 9
- Experiments report performance gains for MELM across monolingual, cross-lingual, and multilingual low-resource NER at various low-resource levels.The conclusion characterizes the gains as encouraging.Source: §6 · p. 9
A.1 Hyperparameter Tuning
The authors tune MELM’s fine-tuning mask rate and generation masking parameter by grid search, then select three augmentation rounds based on dev-set F1.
A.1 Hyperparameter Tuning
- To determine optimal fine-tune mask rate η and generation masking parameter µ, the authors grid-search both over [0.3, 0.5, 0.7], fine-tuning MELM.Source: §A.1 · p. 12
- Dev-set F1 peaks at η = 0.7 and µ = 0.5, the masking settings used in the remaining experiments.The authors grid-search both parameters over [0.3, 0.5, 0.7] using English CoNLL augmented data.Source: Table 4 · p. 12, §A.1 · p. 12
- Dev-set F1 rises through R = 3 augmentation rounds and drops beyond, so the authors use R = 3 in all experiments.The authors attribute the decline beyond the optimum to increasing noise as additional augmented data is added.Source: Table 5 · p. 12, §A.1 · p. 12
A.2 Statistics for Reproducibility
The reproducibility statistics report MELM validation F1, fine-tuning time, and parameter counts across monolingual, cross-lingual, and multilingual NER settings.
A.2 Statistics for Reproducibility
- The reported validation F1 is averaged across three runs and covers monolingual, cross-lingual, and multilingual NER settings.The corresponding statistics are presented in Tables 6, 7, and 8, respectively.Source: §A.2 · p. 12, Table 6 · p. 12, Table 7 · p. 12
- The reproducibility statistics also include estimated MELM fine-tuning time and the number of parameters used.The passage does not provide the specific time or parameter values.Source: §A.2 · p. 12
A.3 Computing Infrastructure
The experiments were conducted on an NVIDIA V100 GPU.
A.3 Computing Infrastructure
- All experiments were conducted on an NVIDIA V100 GPU.Source: §A.3 · p. 12