Source-linked AI summary
Learning Named Entity Tagger using Domain-Specific Dictionary
Jingbo Shang, Liyuan Liu, Xiang Ren, Xiaotao Gu, Teng Ren, Jiawei Han
TL;DR
The paper tackles the need for large, costly manual annotations in domain-specific NER by learning from noisy dictionary-based distant supervision. It proposes Fuzzy-LSTM-CRF and AutoNER with a Tie or Break scheme, and finds that AutoNER performs best with dictionaries alone while remaining competitive with fully supervised benchmarks.
Problem
Domain-specific neural NER requires substantial manual annotation, while dictionary-based distant supervision produces noisy labels that are difficult to learn from effectively.
Method
The paper proposes Fuzzy-LSTM-CRF with a modified IOBES scheme and AutoNER, which detects token ties to form entity spans before assigning their types.
Results
AutoNER achieves the best F1 scores without additional human effort and is competitive with supervised benchmarks using full human annotation.
Takeaways & Limitations
Dictionary-only distant supervision can support competitive NER performance, while refining supervision with high-quality corpus phrases and corpus-tailored dictionaries improves performance in ablations.
Takeaways & Limitations
With all gold annotations, the supervised benchmark performs better than AutoNER, which the authors attribute to AutoNER lacking more advanced distant-supervision techniques.
Abstract
from arXiv · showhide
Recent advances in deep neural models allow us to build reliable named entity recognition (NER) systems without handcrafting features. However, such methods require large amounts of manually-labeled training data. There have been efforts on replacing human annotations with distant supervision (in conjunction with external dictionaries), but the generated noisy labels pose significant challenges on learning effective neural models. Here we propose two neural models to suit noisy distant supervision from the dictionary. First, under the traditional sequence labeling framework, we propose a revised fuzzy CRF layer to handle tokens with multiple possible labels. After identifying the nature of noisy labels in distant supervision, we go beyond the traditional framework and propose a novel, more effective neural model AutoNER with a new Tie or Break scheme. In addition, we discuss how to refine distant supervision for better NER performance. Extensive experiments on three benchmark datasets demonstrate that AutoNER achieves the best performance when only using dictionaries with no additional human effort, and delivers competitive results with state-of-the-art supervised benchmarks.
1 Introduction
The paper addresses costly manual annotation for domain-specific NER by learning from dictionary-based distant supervision, despite noisy and incomplete labels. It introduces Fuzzy CRF and AutoNER, with experiments showing AutoNER performs best using dictionaries alone and remains competitive with fully supervised benchmarks.
- Motivation: Domain-specific NER requires expensive or slow expert annotation, while neural methods typically need many manually labeled sentences.
- Motivation: Dictionary-based distant supervision reduces human effort but limited dictionary coverage can create false-negative labels for unmatched entity tokens.Existing systems often tag unmatched tokens as non-entities, which can miss phrases such as “prostaglandin synthesis.”
- Contributions: Fuzzy CRF extends the traditional CRF framework to accommodate tokens with multiple possible labels without sacrificing computing efficiency.It revises the IOBES-based LSTM-CRF design for noisy distant supervision.
- Contributions: AutoNER predicts whether adjacent tokens are tied within an entity, then identifies each resulting span’s type to better tolerate boundary noise.The Tie or Break scheme relies on inner entity ties being less affected by distant-supervision mismatches.
- Contributions: AutoNER achieves the best performance using only dictionaries with no additional human effort and is competitive with supervised benchmarks.The paper also explores adding high-quality phrases to reduce false negatives and reports ablation experiments.
2 Overview
The paper learns an NER tagger using only dictionaries, extending them with high-quality phrases when coverage is limited. Dictionary matching produces labels that may be known, unknown, or non-entity, which motivates two specialized neural models.
- The goal is to learn a named entity tagger using only dictionaries, whose entries contain surface names, synonyms, and entity types.
- High-quality corpus phrases are added as potential entities with unknown types to compensate for limited dictionary coverage.
- Exact string matching generates entity labels, resolving conflicting matches by maximizing the total number of matched tokens.
- Each token is categorized as belonging to a known-type entity, an unknown-type entity, or a non-entity.
- Fuzzy-LSTM-CRF and AutoNER are designed to learn from labels containing unknown and multiple types.
3 Neural Models
The paper introduces Fuzzy-LSTM-CRF and AutoNER to learn NER from distant supervision containing unknown or multiple possible labels. Fuzzy CRF preserves sequence labeling while AutoNER uses Tie or Break decisions to better exploit noisy dictionary supervision.
- Fuzzy-LSTM-CRF with Modified IOBES: Fuzzy CRF extends conventional CRF to support tokens with multiple possible labels under a modified IOBES scheme.Unknown-typed tokens can receive all IOBES tags across available entity types, while matched tokens receive their known types and non-entities receive O.
- Fuzzy-LSTM-CRF with Modified IOBES: The fuzzy objective maximizes the total probability of all label sequences compatible with the distant-supervision labels.It enumerates possible IOBES tags and matched entity types rather than optimizing only one valid sequence.
- AutoNER with Tie or Break: AutoNER predicts whether adjacent tokens are tied within one entity mention or broken into separate spans, separating span detection from type prediction.A binary classifier predicts Tie or Break, and candidate spans are subsequently assigned entity types, including None for non-entity spans.
- AutoNER with Tie or Break: Tie or Break is designed to remain robust when distant supervision mismatches entity boundaries because inner token ties are often unaffected.Matched unigram false positives are represented by Break labels around the unigram, while Unknown positions are skipped during training.
- AutoNER with Tie or Break: AutoNER omits the CRF layer and Viterbi decoding, making inference more efficient than Fuzzy-LSTM-CRF.The model uses sigmoid-based span-boundary prediction followed by span-type prediction.
4 Distant Supervision Refinement
The paper refines dictionary-based distant supervision by reducing false positives through corpus-tailored dictionaries and false negatives through high-quality out-of-dictionary phrases.
- Refinement overview: Ablation experiments verify the effectiveness of two techniques for refining distant supervision.The techniques target better named entity taggers.
- Reducing false positives: Full dictionaries can create false-positive labels when aliases match entities outside the corpus.For example, “Wednesday” may be wrongly marked as a person because of the unrelated dictionary entry “Wednesday Addams.”
- Reducing false positives: The original dictionary is tailored by excluding entities whose canonical names never appear in the corpus.Canonical names appear in 88.12% of BC5CDR entity mentions and 95.07% of NCBI entity mentions.
- Reducing false negatives: Limited dictionary coverage makes unmatched spans unreliable negative labels because out-of-dictionary high-quality phrases may be entities.This creates false-negative labels during distant supervision.
- Reducing false negatives: AutoPhrase mines high-quality multi-word and single-word phrases from unlabeled in-domain text and a domain dictionary.The method uses example thresholds of 0.5 for multi-word phrases and 0.9 for single-word phrases.
- Reducing false negatives: Out-of-dictionary phrases are added as unknown-type dictionary entries, leaving only unmatched spans labeled non-entity.This extends dictionary coverage before generating distant-supervision labels.
5 Experiments
Experiments evaluate the proposed models and distant-supervision refinements across three benchmark datasets. AutoNER achieves the strongest dictionary-only results, remains competitive with supervised systems, and benefits from more data and refined dictionaries.
- Experimental Setup: Experiments compare Fuzzy-LSTM-CRF and AutoNER with other distantly supervised and supervised methods on three benchmark datasets.The datasets cover biomedical chemical and disease recognition and technical-review aspect-term recognition.
- NER Performance Comparison: AutoNER achieves the best performance without extra human effort, while Fuzzy-LSTM-CRF improves over Dictionary Match but remains worse than AutoNER.The comparison reports F1, precision, and recall across the datasets.
- NER Performance Comparison: AutoNER outperforms SwellShark in almost all cases despite SwellShark using substantially more expert effort for entity-span extraction.The only reported outlier is on NCBI-Disease when SwellShark’s matcher is carefully tuned.
- NER Performance Comparison: AutoNER beats Distant-LSTM-CRF on LaptopReview and approaches supervised performance, with an F1 score 2.16% away from the BC5CDR supervised benchmark.Distant-LSTM-CRF was the previous state-of-the-art distantly supervised model on LaptopReview.
- Distant Supervision Explorations: Tailored dictionaries improve F1 and precision while retaining similar recall, and adding unknown-typed high-quality phrases significantly enhances AutoNER scores, especially recall.On NCBI-Disease, tailoring raises precision from 53.14% to 77.30% while recall changes from 63.54% to 58.54%.
- Test F1 Scores vs. Corpus Size: Test F1 rises substantially at first as distantly supervised sentences increase, then grows more slowly; with gold data, sufficient distant supervision can surpass supervised benchmarks.The authors attribute this to emphasis on matchable mentions and potentially missed matchable mentions in gold annotations.
- Comparison with Gold Supervision: With only 50 annotated BC5CDR articles, the supervised benchmark reaches 74.29% F1, while matching AutoNER’s 84.8% requires at least 300 annotated articles.The reported comparable supervised score is 83.91%.
- Comparison with Gold Supervision: When trained with all annotations, the supervised benchmark performs better than AutoNER, which the authors connect to AutoNER’s limited handling of distant supervision.Further improvements are left for future work.
6 Related Work
Prior work reduces annotation demands through distant supervision and domain resources, but existing distantly supervised NER systems rely on heuristic span detection. AutoNER is presented as a dictionary-only approach requiring no additional human effort.
- Neural NER models reduce feature engineering but typically require large amounts of manually annotated training data.
- Distant supervision uses knowledge bases or dictionaries to reduce human effort across several NLP tasks, while AutoPhrase extracts high-quality phrases without assigning typed entities contextually.
- Existing distantly supervised NER approaches detect entity spans with heuristic combinations of dictionary matching, POS tags, and regular expressions.
- SwellShark models noise across supervision sources for biomedical entity typing but still delegates span detection to heuristic dictionary and POS-based rules.
- AutoNER is described as the most effective model for learning NER using only dictionaries without additional human effort.
7 Conclusion and Future Work
The paper concludes that AutoNER achieves the best dictionary-only F1 scores across three benchmark datasets and remains competitive with fully supervised benchmarks. Future work targets broader languages, domains, sequence-labeling tasks, and more complex entity structures.
- Conclusion: AutoNER achieves the best F1 scores across three benchmark datasets without additional human effort and is competitive with fully supervised benchmarks.
- Future Work: Future work will examine AutoNER across languages and domains and extend the framework to noun phrase chunking.
- Future Work: Future research will also address nested and multiple typed entity recognition beyond the classical NER setting.