PaperlayerView exact revision on arXiv

Source-linked AI summary

Bidirectional LSTM-CRF Models for Sequence Tagging

Zhiheng Huang, Wei Xu, Kai Yu

arXiv:1508.01991v1cs.CL

TL;DR

Sequence tagging supports tasks such as POS, chunking, and NER, motivating models that capture contextual inputs and sentence-level tag information. This paper introduces BI-LSTM-CRF models and finds state-of-the-art or near-state-of-the-art accuracy across benchmark datasets, with robustness and reduced dependence on word embeddings.

  • Problem

    Sequence tagging for POS, chunking, and NER is important because its outputs support downstream applications such as search and advertising.

  • Method

    The paper systematically compares LSTM, BI-LSTM, LSTM-CRF, and BI-LSTM-CRF models for sequence tagging.

  • Results

    BI-LSTM-CRF achieves state-of-the-art or near-state-of-the-art accuracy across POS, chunking, and NER datasets and is robust with less dependence on word embeddings.

  • Takeaways & Limitations

    The model can achieve accurate sequence tagging without relying on word embeddings.

Abstract

from arXiv · show

In this paper, we propose a variety of Long Short-Term Memory (LSTM) based models for sequence tagging. These models include LSTM networks, bidirectional LSTM (BI-LSTM) networks, LSTM with a Conditional Random Field (CRF) layer (LSTM-CRF) and bidirectional LSTM with a CRF layer (BI-LSTM-CRF). Our work is the first to apply a bidirectional LSTM CRF (denoted as BI-LSTM-CRF) model to NLP benchmark sequence tagging data sets. We show that the BI-LSTM-CRF model can efficiently use both past and future input features thanks to a bidirectional LSTM component. It can also use sentence level tag information thanks to a CRF layer. The BI-LSTM-CRF model can produce state of the art (or close to) accuracy on POS, chunking and NER data sets. In addition, it is robust and has less dependence on word embedding as compared to previous observations.

1 Introduction

The introduction frames POS tagging, chunking, and NER as longstanding sequence-tagging tasks with downstream applications. It reviews statistical and neural approaches before presenting LSTM, bidirectional LSTM, and CRF-based models as the paper’s focus.

  • Motivation: Sequence tagging covers POS tagging, chunking, and NER, and its outputs support downstream applications such as identifying product spans in search queries.The passage gives product-ad targeting as an example application of named entity recognition.
  • Related approaches: Existing sequence-tagging models include linear statistical methods such as HMMs, MEMMs, and CRFs.These models are described as the predominant existing approaches.
  • Related approaches: Recent neural approaches include convolutional networks paired with a CRF layer, denoted Conv-CRF, which have produced promising sequence-tagging results.The passage also mentions recurrent neural networks in the speech-language-understanding community.
  • Contributions: The paper proposes and systematically compares LSTM, BI-LSTM, LSTM-CRF, and BI-LSTM-CRF models on NLP tagging data sets.It identifies the first application of BI-LSTM-CRF to NLP benchmark sequence-tagging data sets.

2 Models

The paper develops LSTM-based sequence-tagging models, extending recurrent networks with memory cells, bidirectional processing, and CRF-based sentence-level tag modeling. The resulting BI-LSTM-CRF combines past and future input features with sentence-level tag information.

  • LSTM: LSTMs replace standard RNN hidden-layer updates with purpose-built memory cells to better exploit long-range dependencies.The LSTM cell uses input, forget, and output gates together with a cell vector.
  • BI-LSTM: BI-LSTMs use forward states for past features and backward states for future features at each time frame.The networks are trained using back-propagation through time.
  • CRF: CRFs model tag dependencies at the sentence level rather than predicting tags independently at individual positions.CRF layers use state-transition parameters and can efficiently incorporate past and future tags during prediction.
  • LSTM-CRF: LSTM-CRF combines past input features from an LSTM layer with sentence-level tag information from a CRF layer.Dynamic programming efficiently computes optimal tag sequences during inference.
  • BI-LSTM-CRF: BI-LSTM-CRF adds future input features to the LSTM-CRF architecture, while retaining past input features and sentence-level tag information.The paper states that these additional features can improve tagging accuracy.

3 Training procedure

All models use a generic stochastic-gradient forward-and-backward training procedure, illustrated with BI-LSTM-CRF. Training proceeds epoch by epoch over batches, alternating model and CRF passes before parameter updates.

  • Training procedure: The paper uses a generic SGD forward-and-backward training procedure for all models, illustrating it with the BI-LSTM-CRF model.The procedure is presented in Algorithm 1.
  • Forward pass: For each batch, the BI-LSTM-CRF performs forward passes through both forward and backward LSTM states, producing scores for all tags at all positions.The resulting output is denoted fθ([x]T 1 ).
  • CRF optimization: The CRF layer then performs forward and backward passes to compute gradients for network outputs and state-transition edges.These gradients support subsequent backpropagation from output to input.
  • Parameter update: Errors are backpropagated through both LSTM directions, after which the transition matrix [A]i,j∀i, j and bidirectional LSTM parameters θ are updated.The algorithm repeats these steps for every batch in every epoch.

4 Experiments · 4.1 Data

The experiments evaluate five sequence-tagging models across PTB POS tagging, CoNLL 2000 chunking, and CoNLL 2003 named entity tagging. The data section describes dataset splits and task-specific tagging schemes, using BIO2 for chunking and NER.

  • 4.1 Data: The study tests LSTM, BI-LSTM, CRF, LSTM-CRF, and BI-LSTM-CRF models on three NLP tagging tasks.The tasks are Penn TreeBank POS tagging, CoNLL 2000 chunking, and CoNLL 2003 named entity tagging.
  • 4.1 Data: Table 1 reports sentence, token, and label counts for the training, validation, and test sets.
  • 4.1 Data: POS tagging assigns each word a unique tag indicating its syntactic role.
  • 4.1 Data: Chunking tags each word with its phrase type, such as B-NP for a word starting a noun phrase.
  • 4.1 Data: NER tags each word as other or as one of four entity types: Person, Location, Organization, or Miscellaneous.
  • 4.1 Data: The experiments use the BIO2 annotation standard for both chunking and named entity recognition.

4.2 Features

The models use spelling and context features alongside word features, with dataset-specific feature counts and n-gram configurations. Direct connections from spelling and context features to outputs accelerate training while maintaining similar tagging accuracy.

  • Feature types and sizes: Features are grouped into spelling and context features, totaling 401K for POS, 76K for chunking, and 341K for NER.No extra data is used for POS and chunking, except Senna embeddings.
  • Spelling features: Spelling features include capitalization, letter patterns, prefixes and suffixes, punctuation, apostrophes, and word-pattern mappings.These features are extracted in addition to lowercase word features.
  • Context features: Word features use unigrams and bigrams across datasets, with trigrams additionally used for selected POS and POS–CHUNK features.Trigrams are used for POS features in CoNLL2000 and POS and CHUNK features in CoNLL2003.
  • Word embeddings: Senna embeddings provide a 130K-word vocabulary with 50-dimensional vectors that replace one-hot word representations.The embedding is downloaded and used as an alternative word representation.
  • Feature integration: Direct connections from spelling and context features to network outputs accelerate training and produce very similar tagging accuracy.All reported tagging accuracy uses this connection.

4.3 Results

The experiments compare LSTM-based, CRF, and hybrid models under random or Senna embeddings across POS, chunking, and NER. BI-LSTM-CRF generally performs best, achieves 97.55% POS accuracy, and depends less on pretrained embeddings than Conv-CRF.

  • Experimental setup: Models are trained with identical feature sets under random and Senna word-embedding initialization, isolating differences attributable to network architecture.Training uses a 0.1 learning rate and 300-unit hidden layers; all three tasks converge in fewer than 10 epochs.
  • Overall results: BI-LSTM-CRF generally delivers the best tagging performance, except on randomly initialized POS data, where LSTM-CRF is best.LSTM is the weakest baseline; CRF provides strong baselines, while BI-LSTM-CRF further improves LSTM-CRF.
  • Embedding robustness: BI-LSTM-CRF shows smaller random-versus-Senna performance differences than Conv-CRF: 0.12%, 0.33%, and 4.57% for POS, chunking, and NER.Conv-CRF differences are 0.92%, 3.99%, and 7.20% on the same tasks, indicating heavier dependence on Senna embeddings.
  • Feature robustness: Removing spelling and context features is used to evaluate robustness to engineered features, with degradation reported relative to models using those features.The supplied results include a 0.06 degradation for one compared model.
  • POS: 97.55% test accuracy is achieved on POS, significantly outperforming other systems at the 95% confidence level.BI-LSTM-CRF also reaches good POS accuracy without Senna embeddings.
  • NER: 90.10% F1 is achieved on NER with Senna embeddings and gazetteer features, while BI-LSTM-CRF slightly exceeds Conv-CRF with the same Senna embedding.The reported comparison is 90.10% versus 89.59%.

5 Discussions

The work is positioned relative to prior neural sequence-tagging models using convolutional or LSTM architectures. It argues that bidirectional LSTM-CRF models improve tagging accuracy over comparable single-CRF models with identical feature sets.

  • Related work: Compared with Collobert et al. (2011), this work uses bidirectional LSTM networks rather than convolutional neural networks for sequence tagging.
  • Related work: Compared with prior LSTM tagging work, this model adds bidirectionality and CRF layers, whose absence may hurt tagging accuracy.The cited prior studies include Hammerton (2003) and Yao et al. (2014); Hammerton’s performance was described as not impressive.
  • Related work: The bidirectional LSTM-CRF model consistently achieved better tagging accuracy than a single CRF model with identical feature sets.

6 Conclusions

The paper systematically compares LSTM-based sequence-tagging models and presents the first application of BI-LSTM-CRF to NLP benchmark data. BI-LSTM-CRF achieves state-of-the-art or near-state-of-the-art accuracy while remaining robust and less dependent on word embeddings.

  • The paper systematically compares the performance of LSTM-based models for sequence tagging.
  • The authors present the first application of BI-LSTM-CRF to NLP benchmark sequence-tagging data.
  • BI-LSTM-CRF achieves state-of-the-art or near-state-of-the-art accuracy on POS, chunking, and NER datasets.
  • The model is robust and achieves accurate tagging with less dependence on word embeddings.It can achieve accurate tagging without resorting to word embeddings.
Loading 1508.01991v1…