Source-linked AI summary
Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu
TL;DR
NER often lacks document-level context in practical settings. This paper retrieves semantically relevant external texts, then uses Cooperative Learning to improve both context-enriched and original input views, achieving state-of-the-art results across eight datasets in five domains.
Problem
Document-level contexts are unavailable in many NER applications, limiting access to useful contextual knowledge for recognizing ambiguous entities.
Method
The method retrieves and reranks external texts for a sentence, then applies Cooperative Learning to align token representations or label distributions across two input views.
Results
The approach achieves state-of-the-art accuracy across NER datasets spanning five domains, while Cooperative Learning further improves both input views.
Takeaways & Limitations
External retrieved contexts and Cooperative Learning improve NER performance, with unlabeled data enabling further improvement.
Takeaways & Limitations
The method uses six retrieved contexts, a setting selected as reasonable based on preliminary experiments.
Abstract
from arXiv · showhide
Recent advances in Named Entity Recognition (NER) show that document-level contexts can significantly improve model performance. In many application scenarios, however, such contexts are not available. In this paper, we propose to find external contexts of a sentence by retrieving and selecting a set of semantically relevant texts through a search engine, with the original sentence as the query. We find empirically that the contextual representations computed on the retrieval-based input view, constructed through the concatenation of a sentence and its external contexts, can achieve significantly improved performance compared to the original input view based only on the sentence. Furthermore, we can improve the model performance of both input views by Cooperative Learning, a training method that encourages the two input views to produce similar contextual representations or output label distributions. Experiments show that our approach can achieve new state-of-the-art performance on 8 NER data sets across 5 domains.
1 Introduction
The paper addresses NER settings where document-level context is unavailable by retrieving related texts and combining them with the input sentence. It further proposes Cooperative Learning to improve both contextualized input views, achieving strong results across domains.
- Motivation: Document-level contexts can boost NER accuracy, but they are often unavailable for search queries, tweets, and short comments.Annotators commonly use search engines to find domain knowledge for disambiguating ambiguous entities in these settings.
- Method: The method retrieves and reranks texts related to an input sentence, then concatenates selected external contexts with the sentence as a retrieval-based input view.The resulting sequence is passed through pretrained contextual embeddings before named entity prediction.
- Method: Cooperative Learning jointly improves the original and retrieval-based input views by encouraging similar representations or predictions.The paper proposes constraints based on L2 distances between token representations or KL divergence between prediction distributions.
- Results: Including retrieved external contexts significantly improves NER accuracy on 8 datasets spanning 5 domains.The paper reports that its approaches outperform previous state-of-the-art approaches in each domain.
2 Framework
The framework retrieves and re-ranks external texts for a sentence, then uses them in a retrieval-based NER view alongside the original view. Cooperative Learning aligns the two views so the model can benefit from external context while supporting settings where retrieval is unavailable or too slow.
- 2.2 NER Model: The NER model encodes either the original sentence or the concatenated retrieval-based view with transformer embeddings and predicts labels through a CRF layer.The retrieval-based view appends external contexts to the sentence before contextual encoding.
- 2.1 Re-ranking: The framework retrieves k related texts, re-ranks them by semantic relevance, and selects the top l texts as external contexts.It uses BERTScore-based relevance and concatenates the selected texts with the input sentence.
- 2.3 Cooperative Learning: The retrieval-based view favors accuracy-oriented offline prediction, whereas online serving is constrained by longer inputs and search latency.Cooperative Learning transfers useful information from the retrieval-based view to the original view when external contexts are unavailable or undesirable.
- 2.3 Cooperative Learning: Cooperative Learning constrains the two input views to produce similar internal representations or label distributions.The framework uses L2 distance for token representations and KL divergence for label distributions, with marginal distributions used to approximate the CRF-level divergence.
- 2.3 Cooperative Learning: The training objective combines negative log-likelihood losses for the input views with one or both Cooperative Learning losses.The architecture computes LNLL and LNLL-EXT together with an L2- or KL-based consistency loss.
3 Experiments
Experiments evaluate the retrieval-based NER approach across 8 datasets and 5 domains, including supervised, cross-domain, and semi-supervised settings. External contexts and Cooperative Learning improve performance across input views and achieve state-of-the-art results.
- Settings: Experiments cover 8 NER datasets across 5 domains, including social media, news, biomedical, science and technology, and E-commerce.The E-commerce dataset includes 300,000 unlabeled sentences for semi-supervised training.
- Settings: The study compares no-context and context-enhanced models, CL-L2, and CL-KL against prior state-of-the-art approaches using entity-level F1.Evaluation includes both inputs without external contexts and inputs with retrieved contexts, with results averaged over 5 runs.
- Results: With external contexts, CL models outperform previous state-of-the-art approaches on most datasets and significantly outperform the no-context baseline.The authors also report that their approaches outperform LUKE in all compared cases.
- Results: Cooperative Learning improves accuracy on both input views, including the view without external contexts.The experiments report further improvement from CL and identify CL-KL as relatively stronger than CL-L2 in most cases.
- Cross-Domain Transfer: In cross-domain transfer from CoNLL-03 to CBS SciTech News, CL improves both views and reduces the performance gap between them.External contexts help the context-enhanced view during evaluation, while CL improves transfer for both views.
- Semi-Supervised Learning: Semi-supervised Cooperative Learning uses unlabeled data and further improves both input views, especially the view without external contexts.Training alternates labeled-data loss with the CL loss for unlabeled data.
4 Analysis
The analysis examines context quality, re-ranking choices, and Cooperative Learning through comparisons and ablations. Semantic re-ranking helps, while CL mitigates the loss incurred when models trained with contexts must handle inputs without them.
- Context Quality: The context-quality analysis compares document-level, generated, randomly selected, and retrieved external contexts.The analysis uses WNUT-17 to evaluate performance under four context conditions.
- Comparison of Re-ranking Approaches: BERTScore re-ranking performs best among the evaluated approaches on WNUT-17, indicating that semantic relevance helps performance.The comparison includes search-engine ranking, fuzzy matching, and BERTScore with tf-idf weighting.
- Comparison of Re-ranking Approaches: BERTScore with tf-idf weighting significantly lowers accuracy, possibly because rare-word weighting favors irrelevant texts.The reported drop is statistically significant with p < 0.05.
- Ablation Study: Using both CL losses provides no improvement over using a single CL loss.The ablation compares one-view training, joint training without CL, and training with one or both CL losses.
- Ablation Study: Training with external contexts causes accuracy to drop when predicting inputs without external contexts.Joint training without CL slightly improves the no-context view but lowers performance on the context view, whereas adding CL addresses this trade-off.
5 Related Work
Prior NER work improves token representations with pretrained contextual embeddings and document-level context, while retrieval and multi-view methods provide related foundations for this paper’s approach.
- Pretrained contextual embeddings, including BERT, Flair, and LUKE, are a major source of recent NER accuracy improvements.
- Encoding document-level contexts with pretrained embeddings achieves state-of-the-art accuracy on CoNLL 2002/2003 NER datasets.
- Improving Models through Retrieval: Related-text retrieval has been applied to neural machine translation, text generation, and semantic parsing, with BERTScore used here to measure semantic correlations.
- Multi-View Learning: Multi-view learning trains models from multiple input subsets, while co-training and co-regularization require independent views and unlabeled data.
- Multi-View Learning: Compared with CVT, Cooperative Learning improves both input views, adds token-representation distance minimization, and uses external contexts.
- Knowledge Distillation: Unlike knowledge distillation, the proposed approaches train one unified model rather than transferring knowledge between two models.
6 Conclusion
The paper retrieves related search-engine contexts for NER and adds Cooperative Learning to improve robustness when those contexts are unavailable. It reports strong performance across five domains and effectiveness in semi-supervised training.
- The method retrieves related contexts from a search engine and applies Cooperative Learning to improve NER robustness without external contexts.
- Cooperative Learning enforces consistency between two input views through token-representation or label-distribution constraints.
- The approach significantly outperforms baseline and previous state-of-the-art models across datasets spanning five domains.
- Cooperative Learning is effective in a semi-supervised training setting.
A Retrieved Contexts Versus Document-level contexts on CoNLL-03
Retrieved contexts are competitive with some document-context approaches on CoNLL-03, but a performance gap remains relative to document-level contexts. Cooperative Learning reduces that gap.
- The comparison evaluates retrieved contexts against document-level contexts on CoNLL-03 datasets.
- Retrieved-context approaches are competitive with, and stronger than some, previous methods using maximal document-level contexts on CoNLL-03.
- A gap remains between retrieved contexts and document-level contexts, but Cooperative Learning reduces the gap.