PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Deep Learning for Named Entity Recognition

Jing Li, Aixin Sun, Jianglei Han, Chenliang Li

arXiv:1812.09449v3cs.CL

TL;DR

NER supports downstream applications, while deep learning has advanced NER performance with reduced reliance on handcrafted features. This survey consolidates NER resources, organizes deep-learning approaches by input representations, context encoders, and tag decoders, reviews applications and settings, and identifies challenges and future directions.

  • Problem

    NER supports applications including information retrieval, question answering, and machine translation, motivating a survey of the growing deep-learning literature and its performance factors.

  • Method

    The survey consolidates NER corpora and tools, organizes deep-learning approaches along three architectural axes, and reviews methods for applied NER settings.

  • Results

    Deep learning reports state-of-the-art NER performance, including 93.5% on CoNLL03 and 92.07% on OntoNotes5.0, while noisy data remains challenging.

  • Takeaways & Limitations

    Pre-trained language-model embeddings are becoming a paradigm for neural NER and are associated with significant performance improvements, especially on formal documents.

  • Takeaways & Limitations

    Supervised NER requires substantial annotated data, making annotation time-consuming and expensive for resource-poor languages and specialized domains.

Abstract

from arXiv · show

Named entity recognition (NER) is the task to identify mentions of rigid designators from text belonging to predefined semantic types such as person, location, organization etc. NER always serves as the foundation for many natural language applications such as question answering, text summarization, and machine translation. Early NER systems got a huge success in achieving good performance with the cost of human engineering in designing domain-specific features and rules. In recent years, deep learning, empowered by continuous real-valued vector representations and semantic composition through nonlinear processing, has been employed in NER systems, yielding stat-of-the-art performance. In this paper, we provide a comprehensive review on existing deep learning techniques for NER. We first introduce NER resources, including tagged NER corpora and off-the-shelf NER tools. Then, we systematically categorize existing works based on a taxonomy along three axes: distributed representations for input, context encoder, and tag decoder. Next, we survey the most representative methods for recent applied techniques of deep learning in new NER problem settings and applications. Finally, we present readers with the challenges faced by NER systems and outline future directions in this area.

1 INTRODUCTION

NER identifies mentions of rigid designators by predefined semantic types and supports many NLP applications. This survey reviews deep-learning NER research because recent work has advanced performance while existing reviews remain limited.

  • NER identifies mentions of rigid designators belonging to predefined semantic types such as person, location, and organization.
  • NER supports information extraction and applications including text understanding, information retrieval, summarization, question answering, machine translation, and knowledge-base construction.
  • NER commonly distinguishes generic entities, such as persons and locations, from domain-specific entities, such as proteins, enzymes, and genes.The survey mainly focuses on generic NER in English and does not claim exhaustive coverage.
  • Deep-learning NER systems have expanded since Collobert et al., using minimal feature engineering and successively advancing state-of-the-art performance.
  • The survey addresses a review gap by organizing deep-learning NER methods and resources for researchers and practitioners.Earlier reviews covered technique trends, challenges, opportunities, or narrower domains and entity types.
  • Its taxonomy organizes approaches by input representations, context encoders, and tag decoders, alongside resources, applications, challenges, and future directions.

2 BACKGROUND

NER locates and classifies entity mentions into predefined categories, with task variants differing in granularity and allowed types. The section also introduces datasets and tools used for English NER research.

  • 2.1 What is NER?: NER outputs entity spans and types by locating and classifying named entities in token sequences.A formal output is a list of tuples containing start index, end index, and entity type.
  • 2.1 What is NER?: Coarse-grained NER uses a small set of entity types and one type per mention, whereas fine-grained NER uses more types and may assign multiple types.
  • 2.1 What is NER?: NER supports downstream applications such as information retrieval, question answering, and machine translation.About 71% of search queries contain at least one named entity, motivating entity-aware search applications.
  • 2.2 NER Resources: Datasets and Tools: The survey introduces widely used NER datasets and off-the-shelf tools, focusing on English and the general domain.
  • 2.2 NER Resources: Datasets and Tools: English NER datasets increasingly cover Wikipedia, conversations, tweets, YouTube comments, and StackExchange posts, beyond earlier news articles.
  • 2.2 NER Resources: Datasets and Tools: Dataset tag counts range from 1 in NCBI-Disease to 36 in GENIA, while HYENA contains 505 entity types.

2.3 NER Evaluation Metrics

NER evaluation compares system outputs with human annotations using exact-match or relaxed-match criteria. Precision, recall, and F-score summarize recognition quality across entities and types.

  • 2.3 NER Evaluation Metrics: NER evaluation compares system outputs against human annotations using exact-match or relaxed-match criteria.
  • 2.3.1 Exact-match Evaluation: Exact-match evaluation requires both entity boundaries and types to be correct simultaneously.
  • 2.3.1 Exact-match Evaluation: True positives, false positives, and false negatives provide the counts used to compute precision, recall, and F-score.
  • 2.3.1 Exact-match Evaluation: Precision measures the percentage of returned entities recognized correctly, while recall measures the percentage of total entities recognized correctly.
  • 2.3.1 Exact-match Evaluation: F-score is the harmonic mean of precision and recall, and macro- and micro-averaged F-scores summarize performance across entity types.Macro-F-score averages type-specific scores; micro-F-score aggregates counts across types and can be strongly influenced by large entity categories.
  • 2.3.2 Relaxed-match Evaluation: Relaxed-match evaluation can credit correct types despite boundary differences when spans overlap, but complex procedures reduce score comparability and complicate error analysis.

2.4 Traditional Approaches to NER

Traditional NER approaches comprise rule-based, unsupervised, and feature-based supervised learning systems. They differ in supervision and representation, with supervised methods relying heavily on engineered features.

  • 2.4 Traditional Approaches to NER: Traditional NER approaches are broadly classified as rule-based, unsupervised learning, and feature-based supervised learning.
  • 2.4.1 Rule-based Approaches: Rule-based systems use hand-crafted gazetteers and syntactic-lexical patterns, including domain-specific resources such as synonym dictionaries.
  • 2.4.1 Rule-based Approaches: Rule-based systems can achieve high precision and low recall when dictionaries are incomplete, and domain-specific rules limit transfer across domains.
  • 2.4.2 Unsupervised Learning Approaches: Unsupervised systems use clustering, lexical resources, patterns, and corpus statistics to infer entities without hand-labeled training examples.
  • 2.4.3 Feature-based Supervised Learning Approaches: Supervised NER casts recognition as classification or sequence labeling and learns from annotated examples represented by engineered features.
  • 2.4.3 Feature-based Supervised Learning Approaches: Supervised feature sets include word-level, gazetteer lookup, document, and corpus features, used with algorithms such as HMMs, decision trees, maximum entropy, SVMs, and CRFs.

3 DEEP LEARNING TECHNIQUES FOR NER

Deep learning has become a dominant approach for NER, automatically discovering useful representations from raw input and achieving state-of-the-art results.

  • Deep-learning NER models automatically discover representations needed for classification or detection from raw input in an end-to-end manner.
  • Recent deep-learning NER models have become dominant and achieve state-of-the-art results.

3.1 Why Deep Learning for NER?

Deep learning is motivated for NER by its ability to learn complex features automatically and support end-to-end modeling, organized here around input representations, context encoding, and tag decoding.

  • Deep learning uses multiple processing layers to learn representations with multiple levels of abstraction.Its neural layers apply weighted sums and nonlinear functions, while backpropagation computes gradients for training.
  • Nonlinear transformations let deep-learning NER models learn complex features that linear models such as HMMs and CRFs cannot represent as readily.
  • Deep-learning models reduce the feature-engineering burden by automatically learning useful representations and underlying factors from raw data.
  • End-to-end gradient-descent training enables the design of potentially complex NER systems.
  • The survey organizes DL-based NER models into distributed input representations, context encoders, and tag decoders.The taxonomy separates word- and character-level inputs, context-dependence modeling, and token-tag prediction.

3.2 Distributed Representations for Input

NER models use distributed word, character, contextual, and hybrid representations to encode semantic, syntactic, subword, contextual, and auxiliary information before context encoding.

  • Distributed representations encode words as low-dimensional dense vectors whose dimensions capture latent semantic and syntactic properties.
  • Pre-trained word embeddings are learned from large text collections and may remain fixed or be fine-tuned during NER training.Examples include Word2Vec, GloVe, fastText, and SENNA.
  • Character-level representations capture explicit prefix and suffix information and naturally handle out-of-vocabulary words.CNNs and RNNs are widely used to extract these representations.
  • Character-level word representations can be produced with CNNs, convolutional-highway layers, or LSTMs before context encoding.
  • Contextual string embeddings assign a word different representations according to its surrounding sentential context.Forward and backward character-level language models provide the contextual information.
  • Hybrid representations combine word and character embeddings with auxiliary information such as gazetteers, lexical similarity, dependencies, or visual features.This combination may improve NER performance but carries an associated cost.

3.3 Context Encoder Architectures

Deep-learning NER systems encode context with CNNs, RNNs, recursive networks, and language-model architectures, including efficient ID-CNNs and contextual Transformer embeddings.

  • Convolutional Neural Networks: CNN context encoders extract local features around words and combine them into a global sentence representation.
  • Convolutional Neural Networks: ID-CNNs use stacked dilated convolutions to handle larger context and structured prediction more efficiently than recurrent models.With four width-3 dilated convolutions, they achieve 14-20x test-time speedups over Bi-LSTM-CRF while retaining comparable accuracy.
  • Recurrent Neural Networks: Bidirectional RNNs incorporate both past and future information, making them standard context encoders for deep context-dependent text representations.
  • Recursive Neural Networks: Recursive neural networks classify nodes in constituency structures by recursively computing hidden representations for their subtrees.This explicitly incorporates phrase structures into NER.
  • Neural Language Models: Auxiliary language-modeling objectives train sequence taggers to predict surrounding words alongside the current tag, encouraging richer reusable representations.
  • Deep Transformer: Contextual language-model embeddings combine forward and backward information and can replace traditional embeddings or be fine-tuned for NER.Transformer-pre-trained embeddings are described as a new NER paradigm, including BERT-based task formulations.

3.4 Tag Decoder Architectures

NER tag decoders transform context-dependent representations into tag sequences through four main architectures: MLP+Softmax, CRF, RNN, and pointer networks. These designs differ in whether they predict tags independently, model transitions or segments, or decode sequentially.

  • Tag decoders take context-dependent representations and produce a sequence of tags corresponding to the input sequence.
  • MLP + Softmax: MLP+Softmax casts sequence labeling as independent multiclass classification, predicting each word’s tag without considering neighboring tags.
  • CRF: CRF decoders globally condition on the observation sequence and are the most common tag-decoder choice in neural NER.
  • CRF: Semi-Markov CRFs address CRF limitations by modeling segments directly and combining word-level and segment-level information for segment scoring.
  • RNN: RNN decoders greedily generate tags by feeding each predicted tag and decoder state into the next time step.
  • Pointer Network: Pointer networks repeatedly identify a chunk or segment and then label it, using RNNs to point to positions in the input sequence.

3.5 Summary of DL-based NER

The survey finds that neural NER architectures increasingly rely on pretrained input representations and that performance varies with encoder, decoder, data, and domain conditions. It also highlights trade-offs involving external knowledge, Transformer pretraining, and sequential decoding.

  • Architecture Summary: BiLSTM-CRF is the most common deep-learning architecture for NER, while pretrained bidirectional Transformers and BERT-based models achieve reported state-of-the-art results.The cited results are 93.5% on CoNLL03 for a pretrained bidirectional Transformer and 92.07% on OntoNotes5.0 for BERT with dice loss.
  • Architecture Summary: Pretrained language-model embeddings substantially improve reported NER performance, but noisy data such as W-NUT17 remains challenging.
  • Architecture Comparison: External knowledge can boost NER performance, but acquiring it may be labor-intensive or computationally expensive and can reduce end-to-end generality.
  • Architecture Comparison: Transformers outperform LSTMs when pretrained on huge corpora, fail without pretraining under limited data, and are faster than recurrent layers when n < d.Self-attention has complexity O(n^2 · d), whereas recurrent layers have complexity O(n · d^2).
  • Architecture Comparison: RNN and pointer-network decoders hinder parallelization through greedy decoding, while CRFs can be expensive with many entity types and may not outperform softmax with contextualized embeddings.
  • Architecture Comparison: Architecture selection depends on data and domain: transfer strategies are suggested for scarce data, while domain-specific contextualized models can be fine-tuned for specialized domains.
  • NER for Different Languages: The survey mainly focuses on English, general-domain NER, while other work addresses languages and cross-lingual settings.

4 APPLIED DEEP LEARNING FOR NER

Applied deep learning for NER extends beyond standard architectures through multitask, transfer, active, reinforcement, adversarial, and attention-based learning. These methods target related-task supervision, low-resource adaptation, annotation efficiency, robustness, and selective information use.

  • Multi-task Learning: Multitask learning jointly models related tasks such as POS, chunking, NER, semantic role labeling, entity-relation extraction, and entity segmentation.
  • Transfer Learning: Transfer learning adapts knowledge from source domains or tasks to target NER settings, including low-resource, cross-domain, and cross-lingual scenarios.
  • Transfer Learning: Parameter-sharing transfer models can share representations or CRF layers across tasks, with experiments reporting significant improvements under low-resource conditions.
  • Active Learning: Active learning seeks to reduce annotation effort by letting the model select data, while incremental training avoids retraining the deep classifier from scratch after each labeling round.
  • Reinforcement Learning: Reinforcement learning models sequential extraction decisions through environment interaction and rewards, including query selection and noisy-sentence selection.
  • Adversarial Learning: Adversarial NER adds bounded perturbations or cross-domain adversarial examples and trains on original-plus-adversarial data to improve generalization.
  • Attention Mechanism: Attention mechanisms let NER models focus on informative inputs, including dynamically weighting character- and word-level components and modeling self-attention.
  • Attention Mechanism: Co-attention extends attention to multimodal NER by modeling interactions between visual and textual information in tweets.

5 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies challenges in annotation, informal and unseen entities, scalability, transfer, and tooling, then outlines directions for more robust and usable NER systems.

  • 5.1 Challenges: Supervised deep-learning NER requires large annotated datasets, but annotation is costly and difficult for resource-poor languages and specialized domains.Domain experts are often needed to perform annotation.
  • 5.1 Challenges: Annotation inconsistencies can prevent models trained on one dataset from working well on another, even within the same domain.The survey cites disagreements over entity types and boundaries as sources of inconsistency.
  • 5.1 Challenges: Nested entities occur frequently, with 17% of GENIA entities embedded within another entity and 30% of ACE sentences containing nested entities.The survey calls for annotation schemes covering nested and fine-grained entities.
  • 5.1 Challenges: On informal user-generated text, the best WUT-17 F-scores are slightly above 40%, reflecting challenges from short, noisy, and domain-specific content.The survey also highlights the difficulty of recognizing previously unseen entities in emerging discussions.
  • 5.2 Future Directions: Future work should decouple boundary detection from type classification to support shared boundary solutions and domain-specific type classification.The survey states that correct boundaries can alleviate error propagation in entity linking.
  • 5.2 Future Directions: Jointly performing NER and entity linking could let subtasks benefit from partial outputs and reduce error propagation in pipeline settings.The proposed joint setting may include boundary detection, type classification, and linking.
  • 5.2 Future Directions: Auxiliary resources such as user-language gazetteers are considered often necessary for understanding informal or domain-specific text.The survey identifies obtaining and effectively using matching resources as an open question.
  • 5.2 Future Directions: Scaling deep NER remains difficult because some models require massive computation, motivating work on complexity, compression, pruning, and scalability.ELMo training cited in the survey used 32 GPUs for 5 weeks, while BERT representations used 64 cloud TPUs.

6 CONCLUSION

The survey reviews deep-learning NER research, resources, methods, applications, challenges, and future directions. Its organization is intended to help researchers understand the field and design NER models.

  • 6 CONCLUSION: The survey consolidates tagged corpora and off-the-shelf NER systems, focusing on general-domain English NER resources.Resources are presented in tabular form with links for access.
  • 6 CONCLUSION: It introduces NER preliminaries, evaluation metrics, traditional approaches, and basic deep-learning concepts.
  • 6 CONCLUSION: It maps deep-learning NER studies to a new taxonomy and surveys representative applications in new problem settings.
  • 6 CONCLUSION: The survey summarizes NER applications and presents challenges and future research directions as a reference for designing deep-learning NER models.
Loading 1812.09449v3…