Source-linked AI summary
Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss
Barbara Plank, Anders Søgaard, Yoav Goldberg
TL;DR
The paper addresses limited evidence about how bi-LSTM POS taggers depend on input representation, language, training-data size, and label noise. It evaluates word, character, and byte representations across languages and introduces an auxiliary-loss model for rare words, achieving strong multilingual results and gains for morphologically complex languages.
Problem
The study addresses limited evidence about bi-LSTMs’ sensitivity to input representations, target languages, training-data size, and label noise in POS tagging.
Method
The paper evaluates word, character, and Unicode byte bi-LSTM representations across 22 languages and introduces a multi-task auxiliary loss for token frequency.
Results
The auxiliary loss improves rare-word accuracy, while subtoken representations and hierarchical word-plus-character models support strong multilingual POS tagging.
Takeaways & Limitations
Subtoken information is necessary for state-of-the-art POS tagging, and auxiliary frequency prediction is especially useful for rare words and morphologically rich languages.
Abstract
from arXiv · showhide
Bidirectional long short-term memory (bi-LSTM) networks have recently proven successful for various NLP sequence modeling tasks, but little is known about their reliance to input representations, target languages, data set size, and label noise. We address these issues and evaluate bi-LSTMs with word, character, and unicode byte embeddings for POS tagging. We compare bi-LSTMs to traditional POS taggers across languages and data sizes. We also present a novel bi-LSTM model, which combines the POS tagging loss function with an auxiliary loss function that accounts for rare words. The model obtains state-of-the-art performance across 22 languages, and works especially well for morphologically complex languages. Our analysis suggests that bi-LSTMs are less sensitive to training data size and label corruptions (at small noise levels) than previously assumed.
1 Introduction
The paper evaluates bi-LSTM POS taggers across languages, representation granularities, training-data conditions, and label noise. It also introduces an auxiliary-loss model targeting rare words and reports gains for morphologically rich languages.
- The study evaluates bi-LSTM POS taggers across 22 languages and compares word, character, and byte representations.Previous evaluations focused on one or a small number of languages, and a comparative representation study was missing.
- It investigates whether bi-LSTMs are more sensitive than standard POS taggers to training-data size and label noise.
- The proposed auxiliary-loss bi-LSTM jointly predicts POS tags and token log frequency to distinguish rare from common words.The auxiliary objective is intended to make representations predictive of frequency.
- The auxiliary-loss model improves performance on rare and out-of-vocabulary words, with gains transferring to morphologically rich languages.
- The paper’s contributions are representation evaluation, multilingual and robustness comparisons, and the LOGFREQ auxiliary-loss model.
2 Tagging with bi-LSTMs
The tagging architecture builds contextual representations with bidirectional recurrent networks and combines word-level and subtoken-level bi-LSTMs. Its auxiliary objective jointly predicts POS tags and token frequency to improve rare-token handling.
- Bidirectional recurrent representations: A sequence bi-RNN reads an input sequence in both directions and concatenates the forward and reverse encodings.A context bi-RNN additionally represents each sequence position together with its sequential context.
- Bidirectional recurrent representations: LSTMs replace recurrent cells with units designed to prevent vanishing gradients, and bidirectional LSTMs provide the corresponding bidirectional architecture.
- Tagging architecture: The basic tagger uses a context bi-LSTM over word embeddings, while a lower-level sequence bi-LSTM encodes characters or Unicode bytes.
- Tagging architecture: The subtoken representation is concatenated with learned word embeddings before being passed to the higher-level context bi-LSTM.The paper also tests byte-plus-character and single-subtoken representations.
- Auxiliary loss: FREQBIN jointly predicts the POS tag and the token’s frequency class at every time step.
- Auxiliary loss: The auxiliary objective combines POS-tag and log-frequency cross-entropy losses to encourage distinct representations for common and rare words.The frequency label is computed from the token’s training-data frequency.
3 Experiments
The experiments compare bi-LSTM representations and taggers across languages, data sizes, rare-word regimes, and label noise. Character-informed and auxiliary-loss models perform strongly, while data size and noise affect model comparisons differently.
- Experimental setup: The study compares bi-LSTM, TNT, and CRF taggers across 22 Universal Dependencies languages, with additional WSJ evaluation.The multilingual data use UD v1.2, 17 POS tags, canonical splits, and languages with at least 60k tokens.
- Results: The multi-task bi-LSTM FREQBIN achieves the best results on 12/22 languages and performs well on out-of-vocabulary tokens, especially in Arabic, Farsi, Hebrew, and Finnish.FREQBIN combines word+character inputs with Polyglot initialization for word embeddings.
- Rare words: Rare-token improvements from sub-token representations are largest in morphologically complex Slavic and non-Indoeuropean languages.Figure 2 plots absolute bi-LSTM (⃗w +⃗c) improvements over TNT against mean log frequency.
- Data set size: TNT is better with little data, whereas bi-LSTM is better with more data and already performs surprisingly well after 500 training sentences.For non-Indoeuropean languages, the bi-LSTM is on par with or above the other taggers with 100 sentences.
- Label noise: At low label-noise rates, bi-LSTMs and TNT lose similar accuracy; above 30% corrupted labels, bi-LSTMs show larger drops across language families.The noise experiments artificially corrupt training labels.
4 Related Work
Prior POS-tagging work applied character representations and bi-LSTM models to limited languages and tasks. These studies established strong results for character-based approaches, while later work broadened the comparison.
- Character representations: Early character embeddings were applied to language modeling and text classification before extending to structured prediction tasks.For POS tagging, early work used convolutional neural networks on English and Portuguese.
- Bi-LSTM POS tagging: Ling et al. showed that bi-LSTM character representations outperform the CNN approach of Santos and Zadrozny for language modeling and POS tagging.Subsequent work evaluated character embeddings for German and reported additional bi-LSTM POS-tagging results.
5 Conclusions
Across 22 languages, the paper evaluates token and subtoken representations and introduces a multi-task bi-LSTM with auxiliary loss. Subtoken and hierarchical representations are especially important, while the tagger matches traditional baselines with little data but becomes less robust to higher label noise.
- The study evaluates token- and subtoken-level representations across 22 languages and proposes a multi-task bi-LSTM with auxiliary loss.
- Subtoken representations are necessary for state-of-the-art POS tagging, with character embeddings particularly helpful for non-Indoeuropean and Slavic languages.
- Combining subtoken and word embeddings hierarchically provides the best representation.
- The bi-LSTM tagger matches CRF and HMM taggers with as little as 500 training sentences but is less robust to label noise at higher noise rates.