Source-linked AI summary
Multi-Task Cross-Lingual Sequence Tagging from Scratch
Zhilin Yang, Ruslan Salakhutdinov, William Cohen
TL;DR
Sequence tagging requires models that generalize across tasks and languages without handcrafted features or language-specific pipelines. The paper uses a hierarchical recurrent tagger with shared parameters for multi-task and cross-lingual training, achieving strong results across benchmark tasks and languages while focusing mainly on morphological similarities.
Problem
The paper addresses limited evidence about whether feature-free sequence-tagging models can benefit from joint training across tasks and languages.
Method
The model stacks recurrent layers, applies a CRF over word-level hidden states, and shares architecture and parameters across tasks and languages.
Results
The model achieves state-of-the-art results on several POS tagging, chunking, and NER benchmarks, including 95.41% CoNLL 2000 chunking and 91.20% CoNLL 2003 English NER.
Takeaways & Limitations
Joint training can improve model performance in various multi-task and cross-lingual cases.
Takeaways & Limitations
The work mainly leverages morphological similarities for cross-lingual training, leaving cross-lingual word semantics with parallel data and low-resource settings for future study.
Abstract
from arXiv · showhide
We present a deep hierarchical recurrent neural network for sequence tagging. Given a sequence of words, our model employs deep gated recurrent units on both character and word levels to encode morphology and context information, and applies a conditional random field layer to predict the tags. Our model is task independent, language independent, and feature engineering free. We further extend our model to multi-task and cross-lingual joint training by sharing the architecture and parameters. Our model achieves state-of-the-art results in multiple languages on several benchmark tasks including POS tagging, chunking, and NER. We also demonstrate that multi-task and cross-lingual joint training can improve the performance in various cases.
1 Introduction
The paper develops a feature-free sequence-tagging model for multi-task and cross-lingual joint training, sharing representations across tasks and languages. It evaluates this approach across several tagging tasks and languages, reporting strong benchmark performance and improvements from joint training.
- Sequence tagging predicts a linguistic tag for each word and supports tasks including POS tagging, chunking, and NER.
- Multi-task joint training shares language-specific regularities across different sequence-tagging tasks in one language.
- Cross-lingual joint training studies the same tagging task across languages, while avoiding reliance on parallel corpora or word alignments.
- The model combines character- and word-level gated recurrent units with a conditional random field, and shares architecture and parameters across tasks and languages.
- The evaluation covers POS tagging, chunking, and NER in English, Dutch, and Spanish, with state-of-the-art results on several benchmarks.
2 Related Work
Prior work explored multi-task sequence tagging, multilingual resources, and neural architectures, but important gaps remained. This paper distinguishes itself by testing cross-lingual joint training and extending a neural tagger beyond NER to multiple tagging tasks.
- Earlier multi-task frameworks improved several tagging tasks, but remained behind state-of-the-art results on individual tasks.
- Cross-lingual sequence-tagging approaches commonly use multilingual resources such as parallel corpora, word alignments, or cross-lingual features.
- Previous neural taggers combined recurrent or convolutional architectures with character- and word-level representations, often targeting particular tasks or feature settings.
- Compared with the most similar prior approach, this work studies cross-lingual joint training, generalizes beyond NER, and evaluates multi-task joint training.
3 Model
The model is a deep hierarchical GRU architecture that encodes character-level morphology and word-level context before using a CRF for structured sequence tagging.
- Hierarchical GRU: Deep bidirectional GRU layers are stacked separately at character and word levels to build hierarchical sequence representations.Each recurrent layer receives the previous layer's hidden states, while forward and backward states are concatenated.
- Character-Level GRU: Character-level GRUs encode morphology from each word's character sequence, combining prefix and suffix information into a word representation.The first backward hidden state and last forward hidden state capture character-level morphology before word-level processing.
- Word-Level GRU: The word-level GRU processes word representations to learn N-gram patterns, neighbor dependencies, and word-sequence context without handcrafted features.Its hidden states provide inputs to subsequent tagging layers.
- Conditional Random Field: A CRF layer predicts tag sequences by combining position-specific tag scores with first-order transition scores between adjacent tags.The transition matrix assigns scores to tag pairs, and the model uses a START token for the initial tag.
- Training: Training maximizes a cost-augmented conditional objective using tag-wise Hamming loss, while dynamic programming normalizes and decodes the CRF.Mini-batch AdaGrad and end-to-end backpropagation train the network with character and word embeddings fine-tuned.
- Joint Training: Figure 2 depicts joint-training architectures in which red components share architecture and parameters, whereas blue components remain task- or language-specific.The figure distinguishes English, Spanish, character, and embedding components.
4 Multi-Task and Cross-Lingual Joint Training
The model is extended to jointly train multiple tasks or languages by sharing selected architecture components and parameters while retaining task-specific parameters. Joint optimization alternates updates across task or language objectives, with different sharing schemes targeting shared language regularities or morphology.
- The architecture and parameters are shared across tasks and languages, while task-specific parameters remain separate.The joint model optimizes multiple objective functions for different tasks or languages.
- Optimization: Joint training optimizes the average of all task objectives by sampling batches from each task and taking gradient-descent updates.The same algorithm extends to cross-lingual training by replacing tasks with languages.
- Multi-task joint training: Multi-task training shares all parameters below the CRF layer, including word embeddings, to learn language-specific regularities across tasks.
- Cross-lingual joint training: Cross-lingual training shares the character-level GRU to capture morphological similarities between languages.
5 Experiments
The experiments evaluate the model across five benchmark datasets, sequence-tagging tasks, and languages, then assess standalone performance, component contributions, and joint training. The model reaches strong benchmark results, while joint training helps most in selected tasks and low-label settings.
- Datasets and settings: The evaluation covers POS tagging, chunking, and NER across English, Dutch, and Spanish benchmark datasets.The datasets are PTB POS tagging, CoNLL 2000 chunking, CoNLL 2003 English NER, CoNLL 2002 Dutch NER, and CoNLL 2002 Spanish NER.
- Standalone performance: The model achieves state-of-the-art results on English NER, Dutch NER, Spanish NER, and English chunking, while obtaining the second-best English POS result.It surpasses previously reported results on Dutch NER and English chunking by 2.35 and 0.95 points, respectively, and its English POS score is 0.23 points below the state of the art.
- Joint training: Joint training improves Spanish NER by 1.08 points, Dutch NER by 0.19 points, and English chunking by 0.75 points, with no significant gains on English POS or English NER.The experiments jointly train English POS, chunking, and NER for multi-task learning, and English, Dutch, and Spanish NER for cross-lingual learning.
- Standalone performance: 97.55% is the reported PTB POS tagging result, while the reimplemented comparison model reaches 97.41% with the same SENNA initialization.The reimplemented result is 0.14 points worse than the paper’s model; the authors attribute the main difference to different pre-trained embeddings.
- Component analysis: Word embeddings contribute more than the character-level GRU, although character-level morphology remains a smaller but critical contributor.The comparison removes word embeddings and the character-level GRU separately.
- Joint training: 1.99 points is the maximum reported gain when the target task has a 0.1 labeling rate, and gains are generally larger with less labeled data.The improvement at a 0.1 labeling rate averages 0.37 points more than at 0.3.
- Joint training: The character-level GRU captures cross-lingual morphology: corresponding English and Spanish country names are nearest neighbors in the original embedding space.The authors use a 2-dimensional t-SNE visualization of character-level GRU outputs to show this pattern.
6 Conclusion
The paper presents a GRU- and CRF-based sequence-tagging model and explores joint training by sharing parts of the architecture and parameters. It reports state-of-the-art results across tasks and languages, while limiting cross-lingual training mainly to morphological similarity.
- The model combines gated recurrent units with conditional random fields for sequence tagging.
- Multi-task and cross-lingual joint training share parts of the network architecture and model parameters.
- The model achieves state-of-the-art results on POS tagging, chunking, and NER in multiple languages.
- Cross-lingual joint training mainly leverages morphological similarities, leaving cross-lingual word semantics with parallel data for future work.
- Applying the joint-training approach to low-resource tasks and languages remains future work.