Source-linked AI summary
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks
Victor Sanh, Thomas Wolf, Sebastian Ruder
TL;DR
Multi-task learning can learn transferable representations, but the settings that make it effective remain unclear. The paper hierarchically supervises four related semantic tasks, placing simpler tasks lower and more complex tasks higher in a shared architecture. It reports state-of-the-art results on NER, EMD, and RE, with hidden states becoming more semantically complex toward the top layers.
Problem
The work addresses limited understanding of how semantic task selection and task interactions affect multi-task learning for rich transferable representations.
Method
A shared model hierarchically supervises NER, EMD, Coreference Resolution, and Relation Extraction, using lower-layer supervision for simpler tasks and deeper-layer supervision for complex tasks.
Results
The model achieves state-of-the-art results on Named Entity Recognition, Entity Mention Detection, and Relation Extraction, while encoder representations become stronger on semantic probing tasks.
Takeaways & Limitations
The learned shared representations encode increasingly complex semantic information from lower to higher layers and support strong performance across several semantic tasks.
Takeaways & Limitations
Coreference Resolution and Relation Extraction are kept at the same hierarchy level because the authors did not find a hierarchical relation between them helpful.
Abstract
from arXiv · showhide
Much effort has been devoted to evaluate whether multi-task learning can be leveraged to learn rich representations that can be used in various Natural Language Processing (NLP) down-stream applications. However, there is still a lack of understanding of the settings in which multi-task learning has a significant effect. In this work, we introduce a hierarchical model trained in a multi-task learning setup on a set of carefully selected semantic tasks. The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. This model achieves state-of-the-art results on a number of tasks, namely Named Entity Recognition, Entity Mention Detection and Relation Extraction without hand-engineered features or external NLP tools like syntactic parsers. The hierarchical training supervision induces a set of shared semantic representations at lower layers of the model. We show that as we move from the bottom to the top layers of the model, the hidden states of the layers tend to represent more complex semantic information.
Introduction
The paper addresses limited understanding of how task selection and task interactions affect multi-task learning by combining four interdependent semantic NLP tasks in a hierarchical architecture.
- Multi-task learning can transfer information across tasks to learn rich distributed representations, but the important settings for effective transfer remain insufficiently understood.
- The model combines Named Entity Recognition, Entity Mention Detection, Coreference Resolution, and Relation Extraction because these tasks share interdependencies.
- Low-level tasks are supervised at lower architecture levels, while higher-level tasks receive supervision at deeper layers to model linguistic hierarchy.
- The end-to-end model avoids external linguistic tools and hand-engineered features while improving results on Relation Extraction and Entity Mention Detection.
- The work reports state-of-the-art results on Named Entity Recognition, Relation Extraction, and Entity Mention Detection and introduces proportional sampling for multi-task learning.
Model
The model hierarchically shares representations across four semantic tasks, supervising simpler tasks lower in the network and deeper semantic tasks higher in the architecture.
- Shortcut connections let higher layers access lower-layer representations while imposing a hierarchy from low-level to higher-level task supervision.
- Words embeddings: Words combine fine-tuned GloVe, contextual ELMo, and character-level CNN embeddings into a shared input representation.
- Named Entity Recognition: NER uses biLSTM encoders and a CRF sequence tagger, providing the first supervised layer of the hierarchy.
- Entity Mention Detection: EMD identifies all mentions associated with real-world entities and builds on lower representations through shortcut-connected biLSTM and CRF layers.
- Coreference Resolution: Coreference Resolution clusters mentions referring to the same entity, using span and mention-pair scoring over representations derived from embeddings and EMD outputs.
- Relation Extraction: Relation Extraction jointly models mention selection and relation classification, using a biLSTM encoder and sigmoid probabilities that allow multiple simultaneous relations.
- Relation Extraction: Coreference Resolution and Relation Extraction share the same hierarchy level because both require deeper semantic modeling to link mentions.
Experiment setting
The experiments train and evaluate the four-task model on OntoNotes 5.0 and ACE05 using established task-specific splits and metrics, while comparing training strategies and prior systems.
- Datasets: NER uses the English OntoNotes 5.0 data split from the CoNLL-2012 coreference task and is evaluated with span-level F1 on the test set.
- Datasets: ACE05 supplies the CR, EMD, and RE data, with annotations for mention boundaries, head spans, and six relation types.
- Datasets and evaluation metrics: Coreference is evaluated on multiple ACE05 splits, including a 482-document training set and 117-document test set, using MUC, B3, and CEAFe.
- Baselines: The model is compared with strong prior graphical-model and neural baselines.
- Training: Training randomly selects a task and its batch after each parameter update, and proportional sampling outperforms uniform sampling in performance and speed.
Overall Performance
The hierarchical multi-task model achieves state-of-the-art results on NER, EMD, and RE, with the strongest gains reported for RE. Ablations show that task interactions and task ordering materially affect performance, while results transfer across datasets.
- Overall results: NER (+0.52), EMD (+3.8) and RE (+6.8) reach state-of-the-art results with the hierarchical multi-task framework.These are the reported gains across the three tasks.
- Ablation study: RE shows the largest difference between single-task and multi-task training, while NER results remain similar across the two settings.EMD falls between these cases, and CR can perform slightly better when trained alone.
- Progressively adding tasks: Adding RE supervision increases NER by approximately 1 point and improves both precision and recall on EMD.The comparison is between setups F and I.
- Progressively adding tasks: Combining RE with another task consistently increases RE F1 by 2–6 points, with most of the improvement coming from precision.The paper attributes this pattern to information flowing through the hierarchical architecture.
- Task hierarchy: Supervising EMD below NER is detrimental to overall performance, supporting a hierarchy that follows the intrinsic difficulty of the tasks.This conclusion comes from comparisons that switch NER and EMD supervision levels.
- Cross-dataset evaluation: Comparable performance on CoNLL-2003 and CoNLL-2012 suggests that the reported improvements are not dataset-dependent.The same hyperparameters were retained for these canonical datasets.
- Embedding ablations: Removing ELMo causes an approximately 4-point F1 drop on each task, while removing character embeddings does not affect EMD.Character embeddings nevertheless strongly affect NER, RE, and CR.
What did the embeddings and encoders learn?
The probing analysis finds that base word embeddings already encode broad linguistic information, while encoder layers specialize differently. Higher-level encoder representations emphasize semantic properties, with the coreference encoder producing the strongest scores.
- Probing setup: The probing classifier evaluates linguistic properties from sentence embeddings derived from word embeddings and module-specific biLSTM hidden states.Sentence embeddings are formed by max-pooling the final-layer hidden states.
- Word embeddings: Base word embeddings exceed 70% accuracy on seven of ten probing tasks.Their strong Bigram Shift performance is reported relative to the BoV-FastText baseline.
- Representation differences: Word embeddings and encoder representations show significant discrepancies, indicating that they encode different linguistic features.Averaged base embeddings surpass encoder embeddings on almost all probing tasks.
- Encoder representations: Encoder representations perform better on semantic probing tasks than on surface and syntactic tasks, except for Sentence Length.NER and EMD encoder performances are generally in the same range.
- Hierarchical representations: The coreference encoder consistently achieves the highest encoder scores, suggesting that coreference is the highest-level task and requires rich, diverse representations.The paper links this pattern to the task’s representation demands.
Multi-task learning accelerating training
The study evaluates multi-task training speed by comparing parameter updates needed for convergence against single-task training. The hierarchical multi-task model generally reaches comparable or higher F1 with fewer updates, except for Relation Extraction.
- Training speed is measured by the number of parameter updates required to reach convergence according to the validation metric.A parameter update corresponds to one back-propagation pass.
- The multi-task framework requires fewer updates for comparable or higher F1 in most tasks, except Relation Extraction.
- The authors interpret the result as evidence that knowledge from one task benefits other tasks in the hierarchical architecture.
Related work
The work differs from prior multi-task and structured prediction studies through its hierarchical semantic-task design, proportional sampling, end-to-end neural training, and avoidance of label embeddings, external tools, and hand-engineered features.
- Multi-task architecture and training: Unlike related hierarchical models, the architecture combines semantic tasks while using proportional sampling instead of the training procedures described in earlier work.
- Multi-task architecture and training: The model shares embeddings and stacks hierarchical encoders without feeding supervised layer outputs through learned label embeddings.
- Multi-task architecture and training: Each task uses its own contextualized multi-layer BiLSTM encoder rather than a shared encoder, which the authors found improved performance.
- Task selection: Prior work found that adding semantic NER to syntactic tasks brought no improvement, supporting the view that task relatedness matters for shared representations.
- Representation analysis: SentEval probing evaluates sentence-embedding richness using logistic regression classification and compares against two baselines from Conneau et al.
- Comparison with structured models: Earlier systems combined entity-focused tasks with graphical models and hand-engineered factors, whereas this work uses an end-to-end neural approach without external NLP tools or hand-engineered features.
- Comparison with structured models: The predictors for entity mention detection and relation extraction do not require dependency trees or hand-engineered heuristics.
Conclusion
The paper concludes that hierarchical multi-task supervision produces strong performance across semantic tasks while enabling shared information flow through common embeddings and encoders.
- The model achieves state-of-the-art results on Named Entity Recognition, Entity Mention Detection, and Relation Extraction, with competitive Coreference Resolution results.
- Common embeddings and encoders allow information to flow from the lowest level to the top of the architecture.
- The authors analyze the learned representations and the effect of each task on overall model performance.