Source-linked AI summary
Semi-supervised sequence tagging with bidirectional language models
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power
TL;DR
Sequence taggers need context-sensitive representations, but their recurrent components are commonly trained with limited labeled data. The paper introduces pretrained bidirectional language-model embeddings as a semi-supervised augmentation, achieving state-of-the-art results on NER and chunking. It also finds benefits from backward language models and robustness across domains and labeled-data settings.
Problem
Sequence tagging models need context-sensitive word representations, while their recurrent networks are typically trained only on relatively little labeled data.
Method
The method pretrains neural language models on large unlabeled corpora, extracts contextual embeddings, and adds them to supervised sequence tagging models.
Results
The approach achieves state-of-the-art results on CoNLL 2003 NER and CoNLL 2000 Chunking, including 91.93 mean F1 for NER and 96.37 mean F1 for Chunking.
Takeaways & Limitations
Backward LM embeddings consistently improve performance, and the method remains robust when pretraining data comes from another domain or the baseline uses many labeled examples.
Takeaways & Limitations
The study leaves attention-like alternatives to simple concatenation for future work, and a similar backward-LM experiment did not improve Chunking F1, making that conclusion task dependent.
Abstract
from arXiv · showhide
Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks. However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data. In this paper, we demonstrate a general semi-supervised approach for adding pre- trained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks. We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.
1 Introduction
The paper addresses the limitation that sequence taggers need context-sensitive word representations, while their bidirectional recurrent networks are usually trained only on labeled data. It proposes using context embeddings from language models pretrained on unlabeled text and achieves state-of-the-art results on NER and chunking.
- Motivation: Sequence tagging requires representing words in context because the same word can belong to different entity types in different phrases.“Central” is used as part of an Organization in one phrase and a Location in another.
- Motivation: Current sequence tagging models typically use bidirectional recurrent networks to encode token sequences into context-sensitive representations.These representations support token-specific predictions.
- Research gap: The parameters of these bidirectional recurrent networks are typically learned only from labeled data, despite pretrained word embeddings.Prior work instead explored jointly learning with supplemental labeled data from other tasks.
- Approach: The proposed semi-supervised approach pretrains a neural language model on a large unlabeled corpus and adds its contextual encoding to a supervised sequence tagger.The resulting LM embeddings are intended to capture semantic and syntactic roles in context.
- Results: 91.93% F1 on CoNLL 2003 NER and 96.37% F1 on CoNLL 2000 Chunking establish state-of-the-art performance on both tasks.The NER result improves from 90.87% to 91.93% F1, while the Chunking result is reported as a new state of the art.
- Results: Using both forward and backward language-model embeddings improves performance over a forward-only language model, while domain-specific pretraining is not necessary.The paper applies a language model trained in the news domain to scientific papers.
2 Language model augmented sequence taggers (TagLM)
TagLM augments a hierarchical bidirectional RNN sequence tagger with context embeddings from separately pre-trained forward and backward language models. The resulting architecture combines token-level representations, bidirectional sequence modeling, and CRF decoding for sequence labeling.
- TagLM architecture: TagLM extracts word and language-model embeddings from large unlabeled corpora before using both in a supervised sequence tagging model.The pipeline pre-trains embeddings and a neural language model, extracts representations for each input token, and feeds them into tagging.
- Baseline sequence tagging model: The baseline represents each token by concatenating a character-based representation with a fine-tuned token embedding.Character representations capture morphological information and may be produced by a CNN or RNN.
- Baseline sequence tagging model: Multiple bidirectional RNN layers combine past and future information to produce context-sensitive token representations.Each layer concatenates forward and backward hidden states; the experiments use two RNN layers with GRU or LSTM units depending on the task.
- Baseline sequence tagging model: The final RNN representations feed tag scores and a sentence-level CRF that models dependencies between successive labels.Joint decoding addresses constraints such as invalid BIOES tag transitions.
- Bidirectional language model: Bidirectional LM embeddings concatenate independent forward and backward LM states, encoding context from both preceding and following tokens.The forward LM predicts the next token from preceding context, while the backward LM predicts the previous token from future context.
- Combining LM with sequence model: The combined system concatenates LM embeddings with the first sequence-model RNN layer, which performed best among tested insertion points.The paper notes alternative nonlinear and attention-like integration mechanisms but leaves them for future work.
3 Experiments
Experiments on CoNLL NER and chunking evaluate TagLM against prior systems and analyze language-model direction, size, domain, and task-specific components. TagLM achieves state-of-the-art results, while backward context and task-specific sequence modeling materially affect performance.
- Overall system results: TagLM establishes new state-of-the-art results on both CoNLL 2003 NER and CoNLL 2000 Chunking using bidirectional language models.The system uses a forward CNN-BIG-LSTM and backward LSTM-2048-512.
- Overall system results: 91.93 mean F1 on CoNLL 2003 NER significantly exceeds the previous best 91.62 ±0.33 using gazetteers.The reported increase is statistically significant at 95% with p = 0.021.
- Overall system results: 96.37 mean F1 on CoNLL 2000 Chunking exceeds prior results without additional labeled data by more than 1% absolute F1.It also significantly exceeds the previous best 95.77 system trained jointly with Penn Treebank POS tags.
- Overall system results: Adding LM embeddings improves average absolute F1 by 1.06 points for NER and 1.37 points for Chunking.These gains are reported for the two evaluated sequence-tagging tasks.
- Language-model analysis: Backward LM embeddings consistently outperform forward-only embeddings, improving F1 by 0.22–0.27%.This holds even with the relatively small backward LSTM-2048-512 language model.
- Language-model analysis: Replacing forward LSTM-2048-512 with CNN-BIG-LSTM improves F1 by 0.26–0.31%, approximately matching the gain from adding a backward LM.The corresponding test perplexities on the 1B Word Benchmark are 47.7 and 30.0.
4 Related work
The related work situates LM embeddings among semi-supervised, neural language-model, encoder, and sequence-tagging approaches, while noting a task-dependent limitation in one comparison.
- LM embeddings extend semi-supervised sequence tagging by adding context-sensitive representations learned from unlabeled data to supervised models.
- A similar unlabeled-data experiment improved one task but not Chunking, making the conclusion task dependent.
- Unlike sentence and document encoders, this approach learns token-in-context embeddings from unlabeled language-model objectives for downstream sequence tagging.
- TagLM uses neural language models to encode input words, rather than their more traditional role of scoring candidate translations.
- Bidirectional language models had received limited prior attention, and TagLM uses separate forward and backward models without shared parameters.
- LM embeddings could also serve as features in other sequence-tagging models, although sufficient model complexity may be necessary to use them effectively.
5 Conclusion
The paper proposes a general semi-supervised sequence-tagging method that augments token representations with pre-trained neural language models. It reports state-of-the-art performance across NER and Chunking, with benefits from backward LMs, cross-domain data, and larger labeled sets.
- TagLM augments token representations in sequence-tagging models with pre-trained neural language models.
- TagLM significantly outperforms current state-of-the-art models on two popular NER and Chunking datasets.
- Adding a backward LM consistently improves performance over traditional forward LMs alone.
- The method remains robust when the LM uses unlabeled data from another domain or the tagger has many labeled examples.