Source-linked AI summary
Semi-supervised Multitask Learning for Sequence Labeling
Marek Rei
TL;DR
Sequence labeling objectives focus on sparse token labels, leaving limited incentive to learn general language features. The paper jointly predicts labels and surrounding words with forward and backward language-modeling components, improving performance consistently across benchmarks, including a 3.9% absolute gain on learner-text error detection. The framework requires no additional annotated or unannotated data.
Problem
Sparse sequence-labeling labels mean many tokens contribute little to training, limiting the incentive to learn general language features.
Method
The model jointly predicts each token’s label and surrounding words, training opposite halves of a bidirectional LSTM as forward and backward language models.
Results
The objective improved performance consistently on every benchmark, including a 3.9% absolute improvement over the previous best architecture for learner-text error detection.
Takeaways & Limitations
Language modeling lets the framework make better use of sparse training data across error detection, named entity recognition, chunking, and POS-tagging.
Takeaways & Limitations
Future work is needed to extend the architecture to additional unannotated in-domain resources and improve pre-training of compositional modules.
Abstract
from arXiv · showhide
We propose a sequence labeling framework with a secondary training objective, learning to predict surrounding words for every word in the dataset. This language modeling objective incentivises the system to learn general-purpose patterns of semantic and syntactic composition, which are also useful for improving accuracy on different sequence labeling tasks. The architecture was evaluated on a range of datasets, covering the tasks of error detection in learner texts, named entity recognition, chunking and POS-tagging. The novel language modeling objective provided consistent performance improvements on every benchmark, without requiring any additional annotated or unannotated data.
1 Introduction
Sequence labeling models can discover task-specific features, but sparse labels provide little incentive to learn broader language patterns. The paper adds a language-modeling objective to use each word more effectively and reports consistent gains across benchmarks.
- Sequence labeling applications include named entity recognition, POS tagging, error detection, and shallow parsing.
- 17% of CoNLL 2003 tokens are entities, while only 14% of FCE tokens are annotated errors.These sparse labels leave most words contributing little to training.
- Language modeling predicts the next word from plain text, requiring no task-specific annotation.Its recurrent architecture also resembles common neural sequence labeling frameworks.
- Predicting surrounding words offers more varied targets, encouraging richer and more general language features.
- The proposed architecture jointly assigns token labels and predicts surrounding words using forward- or backward-moving language models.
- Across 10 datasets spanning NER, POS-tagging, chunking, and error detection, the objective consistently improved performance, with the largest gains on error detection.It outperformed the previous state-of-the-art architecture on error detection.
2 Neural Sequence Labeling
The baseline is a bidirectional LSTM sequence labeler that builds contextual representations from word and character information before predicting token labels. It can use softmax or CRF outputs to model labels independently or as structured sequences.
- The baseline takes tokenized sentences and assigns each token a label using a bidirectional LSTM.
- Opposing LSTMs construct context-dependent representations by processing word embeddings in forward and backward directions.Their hidden representations are combined for each word.
- A feedforward layer maps the concatenated directional representations into a joint space for learning context-based features.The layer uses tanh as its nonlinear activation and Wd as a weight matrix.
- The model predicts token labels with either a softmax distribution over labels or a CRF that searches for the optimal label sequence.
- Softmax training minimizes categorical crossentropy, equivalent to minimizing the negative log-probability of the correct labels.
- A character-level bidirectional LSTM supplies alternative word representations that capture character patterns and help handle unseen words.These representations are combined with word embeddings through dynamic weighting.
- Figure 1 unfolds the network over the sentence “Fischler proposes measures,” with input tokens below and expected labels above.Directional arrows indicate whether components move forward or backward.
3 Language Modeling Objective
The framework adds bidirectional language-modeling objectives to sequence labeling so the model can learn broader semantic and syntactic features from otherwise weakly informative tokens. These objectives are combined with label prediction during training and omitted during testing.
- Motivation: Sparse labels leave most tokens weakly informative for the sequence-labeling objective, motivating a supplementary objective that uses more of the training data.In CoNLL 2003 NER, 83% of tokens have label O; in error detection, 86% contain no errors.
- Architecture: The model jointly predicts token labels and surrounding words, using forward and backward language models with bidirectional context for labeling.At each position, language-model components predict the previous or next word while label prediction uses context from both directions.
- Language Modeling Objective: Language modeling is implemented through separate transformations and softmax layers that predict preceding and following words from directional hidden representations.The transformations extract language-model-specific features while the LSTM is optimized for both objectives.
- Training Objective: The combined cost adds the forward and backward language-model losses to the sequence-labeling objective, with γ controlling their relative importance.The paper uses γ to balance the language-modeling and sequence-labeling objectives.
- Test-Time Cost: Language-modeling components are ignored during testing, so sequence-labeling computational complexity and resource requirements remain equal to the baseline.The authors note that training efficiency could be improved with noise-contrastive estimation or hierarchical softmax.
4 Evaluation Setup
The architecture was evaluated across ten sequence-labeling datasets using established splits, repeated random seeds, and task-specific output layers. Development experiments selected regularization and objective-weight settings before test evaluation.
- Datasets: The evaluation covers 10 sequence-labeling datasets spanning error detection, named entity recognition, chunking, and POS-tagging.Word embeddings were initialized from publicly available pretrained vectors, using domain-specific sources for biomedical datasets.
- Evaluation Protocol: Experiments use established train, development, and test splits, with each configuration trained using 10 random seeds and averaged results reported.The development set selected the best model after early stopping, which was then evaluated on the test set.
- Error Detection Results: Table 1 reports Precision, Recall, and F0.5 for alternative architectures on error-detection datasets, adding dropout and LMcost incrementally to the baseline.The table compares the baseline with architectures that incorporate the two modifications.
- Model Configuration: Error detection uses a softmax output, while all other datasets use CRF because development experiments found CRF did not improve the sparse two-label error-detection task.The baseline is the publicly available sequence-labeling system by Rei et al. (2016), with embedding dropout applied during training.
- Objective Weight: The language-modeling objective uses a static γ value of 0.1 across experiments, performing comparably to or better than gradually decreasing γ.The experiments indicate that language modeling helps learn general-purpose features useful for sequence labeling even later in training.
5 Error Detection
Error detection benefits strongly from the added language modeling objective, especially because error labels are sparse and language modeling uses information from all words. The combined architecture improves FCE performance and sustains gains during training.
- Dropout alone decreases error-detection performance, but improves generalization when combined with the language modeling objective.
- 3.9% absolute improvement on the FCE test set was achieved by adding language modeling and word-embedding dropout to the baseline.One FCE training batch increased from 112 seconds on the baseline to 133 seconds with the language modeling objective.
- The language modeling objective gives the model a more varied training signal by predicting surrounding words alongside the two error-detection labels.
- The language modeling system outperforms other configurations early in training and sustains its results in later epochs.
6 NER and Chunking
The language modeling objective consistently improves named entity recognition and chunking performance across general-domain and biomedical benchmarks. Results are generally comparable to state-of-the-art systems, while remaining below a specialized CoNLL-03 result.
- The evaluation covers general-domain CoNLL-2003 NER, biomedical CHEMDNER and JNLPBA NER, and CoNLL-2000 chunking.
- NER and chunking retain majority-label structure, with roughly 83% of NER tokens tagged O and 53% of chunking tokens carrying NP.
- Dropout consistently improves NER and chunking, and adding the language modeling objective further improves performance on every benchmark.
- On JNLPBA, the system achieves 73.83%, compared with 72.55% from Zhou and Su and 72.70% from Rei et al.
- On CoNLL-03, the system reaches 86.26% F1, outperforming Huang et al.'s similar architecture at 84.26% but trailing Lample et al.'s 90.94%.
7 POS tagging
POS-tagging datasets provide relatively balanced and varied label distributions, leaving less room for improvement. Nevertheless, language modeling produces small but consistent gains across domains, languages, and datasets.
- The POS-tagging evaluation uses four datasets, including PTB, GENIA, and Finnish and Spanish datasets.
- POS-tagging labels are relatively balanced, with no single label covering over 50% of tokens, and include 48 PTB labels and 42 GENIA labels.
- Performance improvements are small but consistent across all POS-tagging domains, languages, and datasets.
- Dropout produces a more robust model, while the language modeling cost improves performance further.
- Predicting surrounding words provides additional general-purpose features even when POS labels already supply a varied training objective.
8 Related Work
The work extends multitask learning for sequence labeling by using language modeling as an auxiliary objective rather than switching to additional supervised tasks or datasets. It is designed to integrate with bidirectional sequence labeling frameworks and provide a general training signal.
- Prior multitask sequence-labeling research includes weight sharing between networks optimized for different supervised tasks.
- Cheng et al. predict the next word for out-of-vocabulary name detection, whereas this work integrates language modeling with a bidirectional network.
- Plank et al. jointly predict word frequency and POS tags, while this work uses language modeling as a more general training signal.
- Other multitask approaches switch to auxiliary datasets, whereas the proposed language modeling objective requires no additional data.
9 Conclusion
The framework adds bidirectional language modeling to sequence labeling, encouraging richer feature learning without additional data. Across multiple tasks, this objective improves labeling accuracy, with the largest benefit on sparse, unbalanced learner-text error detection; future work could extend it to unannotated resources.
- 9 Conclusion: The framework predicts surrounding words as a secondary objective while combining forward and backward context to predict each token's label.A bidirectional LSTM uses one half for forward language modeling and the other for backward language modeling.
- 9 Conclusion: The additional objective encourages features that capture language distribution and composition patterns, improving sequence labeling accuracy.Language modeling is auxiliary rather than the system's main goal.
- 9 Conclusion: The architecture was evaluated on datasets covering learner-text error detection, named entity recognition, chunking and POS-tagging.These evaluations span several sequence labeling tasks.
- 9 Conclusion: 3.9% absolute improvement over the previous best architecture was achieved on learner-writing error detection.This was the largest benefit, in a task with sparse and unbalanced labels.
- 9 Conclusion: Future work could extend the architecture to additional unannotated resources, including large amounts of unlabeled in-domain text.The paper identifies pre-training or co-training advanced compositional modules as a limited research area.