Source-linked AI summary
BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, Chao Zhang
TL;DR
Open-domain NER needs scalable supervision, but knowledge-base distant labels are incomplete and noisy. BOND adapts pre-trained language models with distant labels, then self-trains using pseudo soft-labels. Across five benchmark datasets, it consistently outperforms distantly supervised NER competitors.
Problem
Open-domain NER lacks affordable, reliable token-level supervision because knowledge-base distant labels are incomplete and noisy.
Method
BOND uses a two-stage framework that first adapts a pre-trained language model with distant labels, then applies teacher-student self-training with pseudo soft-labels.
Results
BOND consistently achieves the best F1 score, precision, and recall across all five datasets, outperforming the strongest distant-supervision baselines by 0.66–21.91 F1 points.
Takeaways & Limitations
Pre-trained language models can provide semantic information and reduce label noise in open-domain NER under distant supervision.
Takeaways & Limitations
Results could not be reproduced for KALM and ConNET because their knowledge-base and crowd annotations were unavailable.
Abstract
from arXiv · showhide
We study the open-domain named entity recognition (NER) problem under distant supervision. The distant supervision, though does not require large amounts of manual annotations, yields highly incomplete and noisy distant labels via external knowledge bases. To address this challenge, we propose a new computational framework -- BOND, which leverages the power of pre-trained language models (e.g., BERT and RoBERTa) to improve the prediction performance of NER models. Specifically, we propose a two-stage training algorithm: In the first stage, we adapt the pre-trained language model to the NER tasks using the distant labels, which can significantly improve the recall and precision; In the second stage, we drop the distant labels, and propose a self-training approach to further improve the model performance. Thorough experiments on 5 benchmark datasets demonstrate the superiority of BOND over existing distantly supervised NER methods. The code and distantly labeled data have been released in https://github.com/cliang1453/BOND.
1 Introduction
Open-domain NER lacks affordable, reliable token-level annotations because distant supervision produces incomplete and noisy labels. BOND uses pre-trained language models in a two-stage framework to refine those labels and improve distantly supervised NER.
- Motivation: Open-domain NER annotation is expensive, time-consuming, and error-prone because it requires token-level labels.
- Motivation: Knowledge-base matching provides cheap distant labels, but limited coverage leaves entity mentions unmatched and creates false-positive labels.
- Motivation: Distant labels are also noisy because the same entity mention can map to multiple entity types.
- Method: BOND leverages pre-trained language models to provide semantic information and reduce label noise for open-domain NER with distant supervision.
- Method: BOND uses two stages: iterative distant-label refinement with a language model, followed by teacher-student training with pseudo soft-labels.
- Results: Across 5 datasets, BOND significantly outperforms state-of-the-art distantly supervised NER competitors, with significant margins on 4 datasets.
2 Preliminaries
NER assigns entity-category labels to tokens, while distant supervision replaces unavailable gold annotations with labels obtained by matching text against external knowledge resources. Pre-trained language models supply contextual representations that support NER adaptation, but open-domain distant labels remain noisy and incomplete.
- NER Formulation: NER locates entity spans and assigns BIO sequence labels, using B-X for the first token, I-X for continuation tokens, and O for non-entities.
- Supervised NER: Fully supervised NER trains on sentences with token-level annotations and minimizes cross-entropy loss.
- Distantly Supervised NER: Distantly supervised NER replaces true labels with labels generated by matching unlabeled sentences against external gazetteers or knowledge bases.
- Distantly Supervised NER: Open-domain distant labels are often noisy and incomplete because general-purpose knowledge resources have limited matching quality.
- Pre-trained Language Models: Pre-trained language models use bidirectional transformers trained on open-domain unlabeled data to capture deep contextual information.
3 Two-Stage Framework: BOND
BOND uses a two-stage framework to address noisy and incomplete distant supervision for open-domain NER. It adapts a pre-trained language model with early stopping, then performs teacher-student self-training using progressively refined pseudo-labels and confidence-aware soft labels.
- Distant Label Generation: BOND first generates distant labels by identifying potential entities and querying Wikidata for their types, supplemented by gazetteers.Ambiguous entities are assigned type O during Wikidata matching.
- Stage II: Teacher-Student Self-Training: Stage II initializes teacher and student models from Stage I; the student fits teacher-generated pseudo-labels while the teacher is iteratively updated by the early-stopped student.The framework discards previous-iteration pseudo-labels and progressively refines the labels during self-training.
- Stage I: BERT-Assisted Learning: Stage I adapts pre-trained BERT to distantly supervised NER, using token-wise contextual embeddings and early stopping to limit overfitting.The pre-trained model supplies semantic and syntactic knowledge, while early stopping helps preserve generalization under noisy, incomplete labels.
- Stage II: Teacher-Student Self-Training: Self-training with progressive pseudo-label refinement can reduce pseudo-label noise and improve model fitting without retaining pseudo-labels from earlier iterations.The student is trained on labels generated at the current iteration, combined with early stopping to avoid overfitting.
- Stage II: Re-weighted High-Confidence Soft Labels: BOND uses confidence-reweighted soft labels and selects high-confidence tokens so the student retains class-probability information while becoming more robust to low-confidence tokens.The soft-label objective uses KL divergence, and the selected-token threshold is ε ∈ (0,1).
4 Experiments
Experiments evaluate BOND on five NER benchmarks against multiple baseline groups and analyze how its two training stages and components affect performance. BOND consistently leads distantly supervised methods, while ablations show benefits from pre-trained representations, early stopping, soft labels, and confidence selection.
- Experimental Setup: The experiments compare BOND with KB matching, fully supervised NER, distantly supervised methods, and methods using different settings or additional data.The benchmark suite includes CoNLL03, Twitter, OntoNotes5.0, and two other datasets.
- Main Results: BOND achieves the best F1, precision, and recall on all five datasets under distant supervision.It exceeds the strongest distantly supervised baselines by 11.74, 21.91, 0.66, 14.35, and 12.53 F1 points across the five datasets.
- Ablation Study: Removing pre-trained language models reduces Stage I F1 from 75.61 to 36.66 on CoNLL03 and from 51.55 to 18.31 on Wikigold.The ablation attributes this difference to transferred semantic and contextual information from pre-trained RoBERTa.
- Ablation Study: Early stopping, soft labels, and high-confidence selection improve the reported ablation outcomes, whereas mean teacher and virtual adversarial training provide marginal or inconsistent gains.Early stopping limits overfitting to incomplete labels; soft labels improve F1 and recall; high-confidence selection improves F1 on both datasets.
- Ablation Study: Re-initialization improves precision and recall with hard labels but deteriorates both with soft labels, where the authors attribute the change to underfitting.The authors explain that soft labels retain sufficient transferred and learned information, making re-initialization unnecessary.
5 Related Work and Discussion
Low-resource NER research leverages cross-lingual information or bilingual dictionaries, while semi-supervised methods use partially labeled data and unlabeled data for regularization.
- Low-resource NER methods transfer information from closely related languages or bilingual dictionaries to improve performance in a target language.
- Semi-supervised NER methods such as Teacher and Virtual Adversarial Training assume partially labeled data and use unlabeled data for regularization.
A.1 External Knowledge Bases
BOND generates distant labels by matching corpus entities against Wikidata and multi-source gazetteers, with POS tagging, category queries, and heuristic rules supporting the process.
- External Knowledge Bases: Wikidata supplies structured entity knowledge and a SPARQL query service for obtaining entity relationships.
- External Knowledge Bases: For each dataset and entity type, BOND builds gazetteers by collecting names from multiple online sources.
- External Knowledge Bases: BOND first identifies potential entities with POS tagging before matching them through Wikidata queries.
- External Knowledge Bases: The Wikidata procedure traverses parent categories until a matching entity type is found, discarding ambiguous entities as type O.
- External Knowledge Bases: Gazetteer matching assigns an entity type when the entity appears in that type’s gazetteer.
- External Knowledge Bases: Hand-crafted stamp-word rules classify unmatched entities, such as using “Inc.” to identify organization names.
- External Knowledge Bases: For Twitter, BOND uses Godin et al.’s baseline system rather than building a multi-source gazetteer.
B Baseline Settings
The baseline settings include an implemented LSTM-CNN-CRF system and officially released implementations of AutoNER and LRNT.
- Baseline Settings: LSTM-CNN-CRF uses 100-dimensional GloVe embeddings, 30-dimensional character embeddings, a width-3 CNN, tuned output dimensions, and early stopping.The model is trained for 50 epochs using SGD with momentum m = 0.9.
- Baseline Settings: AutoNER and LRNT are evaluated using their officially released author implementations.
C Implementation Details of BOND
All implementations are based on the Huggingface Transformer codebase.
- Implementation Details of BOND: The implementation uses the Huggingface Transformer codebase.
C.1 Adapting RoBERTa to the NER task
The NER model uses pre-trained RoBERTa-base with a linear classification layer, while Stage II masks pseudo-labels assigned to padded tokens during training and loss computation.
- RoBERTa-base serves as the backbone for the NER model, with a linear classification layer built on top.
- Figure 9 illustrates the NER model built on pre-trained RoBERTa-base.
- Label masking ignores pseudo-labels assigned to padded tokens during Stage II training and loss computation.
C.3 Parameter Settings
The implementation specifies dataset-dependent training schedules and learning-rate choices, includes multiple student-model re-initializations, and can combine MT or VAT losses with the Stage II objective.
- Training schedules vary by dataset, while T2 stops training when total training reaches 50 epochs for all datasets.The passage lists T1 and T3 values separately for CoNLL03, Tweet, OntoNotes5, Webpage, and Wikigold.
- Multiple re-initialization restarts Stage II self-training from pre-trained RoBERTa-base when the student model no longer improves.
- MT or VAT can be combined with BOND by minimizing a weighted MT or VAT loss together with Eq. (7).The loss weight is selected from [10,1,10^-1,10^-2,10^-3] using the development set.