Source-linked AI summary
Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
Zhilin Yang, Ruslan Salakhutdinov, William W. Cohen
TL;DR
Neural sequence taggers are general, but it is unclear whether they work well when target tasks have limited annotations. The paper develops hierarchical recurrent transfer models that jointly train source and target tasks across domains, applications, and languages, and reports improvements under low-resource conditions and on several benchmarks.
Problem
It is unclear whether general neural sequence taggers can transfer useful representations to sequence-tagging tasks with limited target annotations.
Method
The paper jointly trains source and target tasks with a deep hierarchical recurrent network that shares representations and selected parameters across transfer settings.
Results
The approach consistently improves non-transfer results, especially at lower labeling rates, and achieves new state-of-the-art results on most considered benchmark datasets.
Takeaways & Limitations
Transfer performance depends crucially on target-label abundance, source-target relatedness, and the number of parameters that can be shared.
Takeaways & Limitations
Cross-lingual transfer is focused on languages with similar alphabets because transfer between disparate alphabets is difficult without additional resources.
Abstract
from arXiv · showhide
Recent papers have shown that neural networks obtain state-of-the-art performance on several different sequence tagging tasks. One appealing property of such systems is their generality, as excellent performance can be achieved with a unified architecture and without task-specific feature engineering. However, it is unclear if such systems can be used for tasks without large amounts of training data. In this paper we explore the problem of transfer learning for neural sequence taggers, where a source task with plentiful annotations (e.g., POS tagging on Penn Treebank) is used to improve performance on a target task with fewer available annotations (e.g., POS tagging for microblogs). We examine the effects of transfer learning for deep hierarchical recurrent networks across domains, applications, and languages, and show that significant improvement can often be obtained. These improvements lead to improvements over the current state-of-the-art on several well-studied tasks.
1 INTRODUCTION
The paper asks whether neural sequence taggers can transfer learned representations across tasks, especially when target annotations are scarce. It proposes joint source-target training with a hierarchical recurrent network and studies transfer across domains, applications, and languages.
- Motivation: Transfer learning addresses the challenge of improving a target sequence-tagging task using plentiful labels from a related source task.The setting is motivated by low-resource languages and domains, though multi-task transfer can also help when labels are relatively abundant.
- Motivation: The paper tests whether neural networks’ general representations can improve task performance through shared parameters and feature representations.This question follows from end-to-end neural taggers that avoid task-specific feature engineering and can serve multiple applications and languages.
- Approach: The proposed approach jointly trains source and target tasks with a deep hierarchical recurrent neural network that shares hidden representations and selected parameters.The framework combines task objectives and uses gradient-based optimization.
- Scope: The framework studies parameter sharing for cross-domain, cross-application, and cross-lingual transfer rather than restricting transfer to one setting.It presents different sharing architectures for these transfer cases.
- Results: The paper reports significant improvements when target tasks have few labels and are more related to source tasks, including gains over state-of-the-art results with relatively abundant labels.The supplied passages state the result qualitatively and do not provide a complete numerical comparison.
2 RELATED WORK
Prior NLP transfer methods either rely on additional resources or adapt models to exploit task relatedness. This paper instead uses deep recurrent networks to support transfer across domains, applications, and languages within a unified framework.
- Resource-based transfer: Resource-based transfer uses linguistic annotations such as dictionaries, corpora, and word alignments, but is sensitive to the scale and quality of those resources.The passage identifies resource dependence as a limitation of this transfer paradigm.
- Model-based transfer: Model-based transfer exploits source-target similarity by modifying model architectures, training procedures, or feature representations without requiring additional resources.Earlier examples include structural parameter sharing and joint training.
- This work: The proposed method belongs to model-based transfer and focuses on exploiting the generality of deep recurrent neural networks.Its stated application scope includes transfer between domains, applications, and languages.
- Neural sequence tagging: Deep neural sequence taggers provide a unified architecture that has been applied across multiple applications and languages without changing the network architecture.The related work describes this generality as a property of prior end-to-end neural tagging systems.
3 APPROACH
The approach builds hierarchical character- and word-level recurrent representations, then transfers knowledge by sharing selected parameters across source and target tasks. Three architectures cover domain, application, and language transfer under different sharing schemes.
- Base model: The hierarchical model converts character sequences into morphological representations, then combines them with word embeddings to encode word-level context.Character- and word-level layers may use CNNs or RNNs; this paper implements both with GRUs.
- Transfer architectures: T-A, T-B, and T-C extend the base model with different parameter-sharing schemes for transfer learning settings.The architectures are illustrated in Figure 1(b), 1(c), and 1(d), respectively.
- Cross-domain transfer: Cross-domain transfer assumes few target-domain labels and shares parameters differently depending on whether the source and target label sets are mappable.Mappable labels use a label-mapping step, whereas disparate labels use separate CRF layers.
- Cross-application transfer: Cross-application transfer uses T-B, keeping CRF layers disjoint while sharing the remaining representation across applications in the same language.The design reflects shared alphabets but different application label sets.
- Cross-lingual transfer: Cross-lingual transfer targets languages with similar alphabets and shares character embeddings and the character-level layer to exploit morphological similarities.The paper explicitly notes that transfer between languages with disparate alphabets is difficult without additional resources.
- Joint training: Training divides parameters into shared and task-specific sets, samples either source or target batches, and updates both parameter sets using that task’s loss.Shared parameters are jointly optimized, while task-specific parameters are trained separately; AdaGrad computes learning rates.
- Model implementation: The implementation uses bidirectional two-layer GRUs at both character and word levels, with word-level inputs combining character-derived states and word embeddings.The CRF layer defines the tagging objective using a max-margin cost that penalizes high-cost incorrect tag sequences more heavily.
4 EXPERIMENTS
Experiments evaluate transfer across domains, applications, and languages under varied labeling rates. Transfer consistently improves target performance, especially with fewer labels and stronger source-target relatedness, while more parameter sharing yields larger gains.
- Datasets: The evaluation uses PTB POS, CoNLL chunking and NER, Dutch and Spanish NER, Genia, and Twitter datasets.Genia and Twitter use random 80%/10%/10% train/development/test splits because standard splits are unavailable.
- Transfer learning performance: Transfer learning consistently improves over non-transfer results, with larger gains at lower target labeling rates.Figure 2 compares models with and without transfer across multiple labeling rates.
- Cross-domain transfer: 83%+ accuracy is achieved with zero labels and 92% with a 0.001 labeling rate when transferring from PTB POS tagging to Genia.Twitter POS tagging and NER also improve at every labeling rate, with gains above 8% at 0.1 labels.
- Cross-application and cross-lingual transfer: 6% and 3% improvements occur for CoNLL 2000 chunking and CoNLL 2003 NER, respectively, at 0.1 labels when transferring from PTB POS tagging.Cross-lingual transfer also improves performance when few labels are available.
- Transfer architectures: T-A > T-B > T-C: transfer gains decrease as fewer model parameters are shared.Cross-lingual settings are restricted to T-C because source-target similarities are less prominent, making improvements less substantial.
- Transfer relatedness: Transfer gains diminish as transfer becomes indirect across domains, applications, and languages.The combined setting transfers from Spanish NER in a general domain to English POS tagging in a biomedical domain.
- Comparison with state-of-the-art results: New state-of-the-art results are achieved on all considered benchmark datasets except PTB POS tagging.The base model without transfer remains competitive with state-of-the-art systems, providing a strong comparison baseline.
5 CONCLUSION
The paper develops hierarchical recurrent transfer-learning architectures for sequence tagging across domains, applications, and languages. Experiments show significant low-resource improvements and new state-of-the-art results on some benchmarks, with gains shaped by labels, task relatedness, and shared parameters.
- Conclusion: The approach designs three architectures for cross-domain, cross-application, and cross-lingual transfer.The architectures use different parameter-sharing schemes within a unified transfer-learning framework.
- Conclusion: Transfer learning significantly improves sequence-tagging performance under low-resource conditions and achieves new state-of-the-art results on some benchmarks.The conclusion identifies target-label abundance, source-target relatedness, and the number of shared parameters as crucial factors.
- Conclusion: Future work could combine model-based transfer with resource-based transfer for cross-lingual learning.