PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Named Entity Recognition with Neural Character Embeddings

Cicero Nogueira dos Santos, Victor Guimarães

arXiv:1505.05008v2cs.CL

TL;DR

State-of-the-art NER systems often depend on handcrafted features and outputs from other NLP tasks. This paper evaluates CharWNN, which combines word- and character-level embeddings for language-independent sequential NER classification. It achieves state-of-the-art results in Portuguese and Spanish, including substantial HAREM I gains over the prior state-of-the-art system.

  • Problem

    Existing state-of-the-art NER systems rely heavily on costly handcrafted features and outputs from other NLP tasks, while deep-learning NER had not used character-level embeddings.

  • Method

    The paper applies CharWNN, a deep neural network that jointly uses word-level and character-level embeddings for language-independent sequential NER classification.

  • Results

    CharWNN achieves state-of-the-art results for Portuguese and Spanish NER, using the same configuration as for POS tagging and only automatically learned features.

  • Takeaways & Limitations

    Neural character embeddings contribute to NER performance, and CharWNN defines new state-of-the-art results for Portuguese and Spanish NER.

Abstract

from arXiv · show

Most state-of-the-art named entity recognition (NER) systems rely on handcrafted features and on the output of other NLP tasks such as part-of-speech (POS) tagging and text chunking. In this work we propose a language-independent NER system that uses automatically learned features only. Our approach is based on the CharWNN deep neural network, which uses word-level and character-level representations (embeddings) to perform sequential classification. We perform an extensive number of experiments using two annotated corpora in two different languages: HAREM I corpus, which contains texts in Portuguese; and the SPA CoNLL-2002 corpus, which contains texts in Spanish. Our experimental results shade light on the contribution of neural character embeddings for NER. Moreover, we demonstrate that the same neural network which has been successfully applied to POS tagging can also achieve state-of-the-art results for language-independet NER, using the same hyperparameters, and without any handcrafted features. For the HAREM I corpus, CharWNN outperforms the state-of-the-art system by 7.9 points in the F1-score for the total scenario (ten NE classes), and by 7.2 points in the F1 for the selective scenario (five NE classes).

1 Introduction

The paper presents a language-independent NER system that replaces costly handcrafted features and auxiliary NLP-task outputs with automatically learned word- and character-level representations. Experiments on Portuguese and Spanish corpora show state-of-the-art performance, including large gains on HAREM I.

  • NER identifies names in text and classifies them into predefined categories such as person, organization, location, and time.
  • The study addresses the reported absence of deep-learning NER approaches using character-level embeddings while avoiding handcrafted features.
  • CharWNN jointly uses word-level and character-level embeddings in a deep neural network for sequential classification.Its convolutional layer extracts character-level features from words of any size.
  • The experiments use annotated Portuguese HAREM I and Spanish SPA CoNLL-2002 corpora to compare embedding combinations and assess unsupervised word-embedding pre-training.
  • 7.9 points in F1-score for the total scenario and 7.2 points in F1 for the selective scenario separate CharWNN from the HAREM I state-of-the-art system.The total scenario covers ten named-entity classes, while the selective scenario covers five.

2 CharWNN

CharWNN combines word and character embeddings, processes local context through neural layers, and predicts structured tag sequences with Viterbi inference. Character representations are built by convolution and max pooling, while word-context scores are integrated with tag-transition scores.

  • 2.1 Word- and Character-level Embeddings: CharWNN converts each word into joint word-level and character-level embeddings, then uses a fixed-sized word window to score tags.Word embeddings capture syntactic and semantic information, whereas character embeddings capture morphological and shape information.
  • 2.1 Word- and Character-level Embeddings: A convolutional layer extracts local features around each character and applies max pooling to produce a fixed-sized character-level embedding.The same process is applied to every word, regardless of word size.
  • 2.1 Word- and Character-level Embeddings: Character embeddings are obtained from a learned character embedding matrix, while word embeddings are retrieved from a learned word embedding matrix.
  • 2.2 Scoring and Structured Inference: The word-context vector passes through two neural layers to compute a score for every tag.The context-window size and hidden-layer size are user-selected hyperparameters.
  • 2.2 Scoring and Structured Inference: Viterbi inference selects the predicted tag sequence after word-level scores are computed.The structured score combines tag scores with transition and starting-tag scores.
  • 2.2 Scoring and Structured Inference: The model is trained by minimizing negative conditional log-likelihood with stochastic gradient descent and backpropagation.

3 Experimental Setup

The experiments evaluate CharWNN and related variants on Portuguese and Spanish NER corpora using specified preprocessing, tagging, initialization, and training settings.

  • 3.1 Unsupervised Learning of Word Embeddings: Word embeddings are initialized through unsupervised skip-gram pre-training, while character embeddings are randomly initialized from a uniform distribution.Spanish pre-training uses a processed Wikipedia corpus containing around 450 million tokens.
  • 3.2 Corpora: Portuguese experiments use HAREM I, with First HAREM for training, MiniHAREM for testing, and 5% of training data for development.The corpus contains ten named entity categories.
  • 3.2 Corpora: Spanish experiments use SPA CoNLL-2002, which provides annotated training, development, and test sets across four entity categories.The categories are Person, Organization, Location, and Miscellaneous.
  • 3.3 NER Setup: NER is formulated as sequential classification using IOB2 tags that distinguish outside, beginning, and inside tokens for each entity type.The setup uses O, B-X, and I-X labels.
  • 3.3 Training Settings: Most experiments reuse POS-tagging hyperparameters, except for SPA CoNLL-2002's learning rate, set to 0.005 to avoid divergence.Development sets select six epochs for HAREM and sixteen for SPA CoNLL-2002.
  • 3.4 Compared Architectures: CharWNN is compared with CharNN and WNN to assess joint versus separate character-level and word-level representations, alongside capitalization and suffix features added to WNN.CharNN removes word embeddings, while WNN removes character embeddings.

4 Experimental Results

Experiments across Spanish and Portuguese NER compare CharWNN with neural variants and handcrafted-feature systems. CharWNN performs strongly with jointly learned word- and character-level information, including state-of-the-art comparisons.

  • Spanish NER: CharWNN achieves the best precision, recall and F1 on both development and test sets for SPA CoNLL-2002.
  • Spanish NER: 3 points: CharWNN’s test-set F1 exceeds WNN’s despite WNN using suffix and capitalization features.The result suggests character-level embeddings are at least as effective as those two handcrafted character-level features.
  • Neural architecture comparisons: Jointly using word-level and character-level embeddings outperforms using either word embeddings or character-level embeddings alone.
  • Spanish NER: CharWNN achieves state-of-the-art results for SPA CoNLL-2002 using only automatically learned features, without the compared system’s gazetteers and other handcrafted features.The comparison system uses AdaBoost, decision trees, gazetteers, and outputs from POS tagging and noun-phrase chunking.
  • Portuguese NER: CharWNN outperforms ETLCMT by a large margin in both total and selective HAREM I scenarios while using automatically learned features only.The total scenario scores ten categories, whereas the selective scenario scores five chosen categories.
  • Portuguese NER: 21 points: CharWNN improves recall for the ORG entity over ETLCMT in the selective HAREM I scenario.CharWNN also produces much better recall for LOC and PER.
  • Unsupervised pre-training: 13.2 points: unsupervised pre-training improves HAREM I selective F1, compared with 4.3 points for SPA CoNLL-2002.The authors associate the larger HAREM I impact with its much smaller training set.

5 Related Work

Earlier deep-learning NER systems addressed related tasks or relied on additional handcrafted information. This work distinguishes itself through neural character embeddings, automatic features, and experiments across Portuguese and Spanish.

  • Prior deep-learning approaches: Earlier systems used deep learning for NER but often simplified the task or added handcrafted features such as gazetteers.Chen et al. assumed entity boundaries were already identified, while Collobert et al. added gazetteer information and Passos et al. used handcrafted features.
  • Prior deep-learning approaches: Chen et al. performed named entity categorization after entity boundaries had been identified, using character-level information on Chinese text.Their system achieved state-of-the-art results for that categorization task.
  • Prior deep-learning approaches: Collobert et al. used a WNN-equivalent neural network for English NER and achieved state-of-the-art results by adding gazetteer information.
  • Prior deep-learning approaches: Passos et al. combined phrase embeddings with a linear-chain CRF and handcrafted gazetteer features for state-of-the-art English NER results.They evaluated the approach on CoNLL 2003 and Ontonotes NER.
  • This work: This work uses neural character embeddings to perform the full task with automatically learned features across two languages, whereas previous work focused on one language.The experiments cover Portuguese and Spanish NER.

6 Conclusions

The paper concludes that CharWNN can achieve state-of-the-art language-independent NER using automatically learned word- and character-level representations. It also examines character embeddings and reports conclusions across Portuguese and Spanish NER.

  • Conclusions: CharWNN uses word- and character-level embeddings for sequential classification in language-independent NER.
  • Conclusions: The same DNN previously used for POS tagging achieves state-of-the-art NER results with the same hyperparameters and no handcrafted features.
  • Conclusions: The experiments examine neural character embeddings and define new state-of-the-art results for Portuguese and Spanish NER.
Loading 1505.05008v2…