PaperlayerView exact revision on arXiv

Source-linked AI summary

A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, Richard Socher

arXiv:1611.01587v5cs.CLcs.AI

TL;DR

The paper addresses limited multi-task learning across diverse NLP tasks and proposes a hierarchical Joint Many-Task model that grows deeper as task complexity increases. It uses shared representations, lower-level predictions, and successive regularization for end-to-end training. The model achieves state-of-the-art or competitive results across five tasks, while the authors identify unresolved training-strategy and dependency-structure limitations.

  • Problem

    Existing deep NLP models often predict tasks separately or at the same depth, leaving multi-level benefits across diverse linguistic tasks insufficiently explored.

  • Method

    The JMT model successively grows layers for POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment, using shared embeddings, shortcut predictions, and successive regularization.

  • Results

    The single JMT model improves all five tasks over separate single-task models and achieves state-of-the-art or competitive results across the evaluated tasks.

  • Takeaways & Limitations

    Modeling linguistic hierarchies in one end-to-end network can support both higher-level and lower-level NLP tasks.

  • Takeaways & Limitations

    Training currently uses shared epochs and learning-rate schedules despite substantially different dataset sizes and task requirements.

Abstract

from arXiv · show

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.

1 Introduction

The paper argues that NLP tasks can benefit from jointly modeling linguistic levels, and introduces a Joint Many-Task model that assigns increasingly complex tasks to deeper layers. Its single end-to-end model achieves competitive results across five tasks.

  • Existing NLP systems often use pipelines, where lower-level outputs support higher-level tasks without end-to-end training.
  • The JMT model predicts increasingly complex NLP tasks at successively deeper layers to reflect linguistic hierarchies.
  • The model is trained end-to-end for POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment.
  • An adaptive training and regularization strategy grows model depth while avoiding catastrophic interference between tasks.
  • The authors report competitive results across the five tasks, arguing that linguistic hierarchies matter more than handling tasks at the same layer.

2 The Joint Many-Task Model

The JMT model builds a hierarchy of shared representations from word morphology and POS tags through syntax and sentence-level semantics. Higher tasks receive lower-level predictions and representations as inputs.

  • 2.1 Word Representations: Shared word and character n-gram embeddings provide morphological features to all five tasks.
  • 2.2 Word-Level Task: POS Tagging: The first bi-LSTM layer predicts POS tags, while the second layer predicts chunking on top of the POS layer.
  • 2.3 Word-Level Task: Chunking: Predicted POS probabilities are combined with label embeddings, so chunking uses model outputs without requiring gold POS tags.
  • 2.4 Syntactic Task: Dependency Parsing: The third bi-LSTM layer predicts dependency heads and labels using representations and label embeddings from the preceding tasks.
  • 2.4 Syntactic Task: Dependency Parsing: Dependency parsing greedily selects heads and labels, then applies Eisner’s algorithm when the result is not a well-formed tree.
  • 2.5 Semantic Task: Semantic relatedness: The fourth and fifth layers model sentence relatedness and textual entailment, respectively, using pooled sentence representations and semantic task outputs.

3 Training the JMT Model

The model is jointly trained across datasets in hierarchical task order, with successive regularization preserving previously learned information while new layers and tasks are optimized.

  • Training iterates through the full datasets in the order of the corresponding tasks during each epoch.
  • Word and character embeddings are pretrained with Skip-gram and fine-tuned during model training.
  • The POS objective combines task loss, L2 weight regularization, and successive regularization on the embedding parameters.
  • Successive regularization penalizes changes from parameters learned after the previous top-level task, reducing forgetting of other tasks.
  • Each later task optimizes an objective over its inherited parameters and task-specific weights, biases, and label embeddings.

4 Related Work

Prior multi-task NLP models often handled single tasks, closely related tasks, or tasks at a shared depth. This work extends hierarchical multi-task learning to diverse NLP tasks while continuing to train lower-level components.

  • Many deep NLP models are designed for single tasks or apply general-purpose architectures to tasks independently.
  • Earlier multi-task models used different layers effectively for closely related tasks, but involved limited task diversity and did not clearly improve lower-level tasks.
  • Related work jointly modeled combinations such as POS tagging, chunking, language modeling, dependency parsing, entity detection, and relation extraction.
  • Some computer-vision multi-task models assume every sample has annotations for all tasks and do not explicitly model task hierarchies.
  • Unlike progressive models that fix completed task parameters, JMT uses successive regularization to keep training lower-level tasks without significant accuracy drops.

5 Experimental Settings

The model is trained on five NLP tasks in linguistic-hierarchy order, using shared bi-LSTM representations and regularization as depth increases.

  • Experimental Settings: POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment use successively deeper layers of the model.Training follows the same task order at each epoch, from POS tagging through textual entailment.
  • Experimental Settings: 100-dimensional embeddings and bi-LSTM hidden states are used throughout the model.The implementation uses mini-batch stochastic gradient descent and growing gradient-clipping values based on task depth.
  • Experimental Settings: Successive regularization is combined with L2-norm regularization and dropout.The regularization strategy is applied while the model is trained across tasks of increasing depth.

6 Results and Discussion

The Joint Many-Task model improves performance across five tasks and supports a hierarchy in which higher-level semantic tasks can benefit parsing and lower-level tasks. Ablations further show that shortcut connections, label embeddings, task-specific depths, and ordered training are important design choices, while several evaluation and modeling boundaries remain.

  • Overall Results: All five tasks improve over separate single-layer bi-LSTM models in the JMTall setting.The model handles POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment in one model.
  • Overall Results: JMT improves both higher-level and lower-level tasks in closely related task subsets, while semantic tasks improve parsing in JMTCD and JMTCE.These subset results support interaction across the linguistic hierarchy rather than benefits restricted to one task level.
  • Overall Results: 95%+ of greedy JMTABC development parses are well-formed dependency trees, and Eisner conversion changes UAS from 94.52% to 94.53% and LAS from 92.61% to 92.62%.Only 83 of 1,700 development sentences required conversion because of multiple roots, no roots, or cycles.
  • Limitations and Scope: Chunking results are reported only for Single and JMTAB because its evaluation sentences overlap dependency-parsing training data.The model also does not explicitly use learned dependency structures for semantic relatedness, which the authors identify as future work.
  • Published Comparisons: The model reaches state-of-the-art or competitive results across POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment.Chunking, semantic relatedness, and textual entailment are reported as state of the art; POS tagging is close to state of the art, and the greedy parser outperforms an earlier beam-search model.
  • Architecture Analysis: Shortcut connections and lower-layer output label embeddings are important, especially for higher-layer semantic tasks.Removing shortcut connections or label embeddings reduces the effectiveness of the many-task architecture; simply stacking LSTM layers is insufficient.
  • Training Analysis: Successive regularization mainly improves chunking when dataset sizes are imbalanced, while randomized task order lowers semantic-task scores.Sharing word representations and output label embeddings is more effective than merely stacking layers in the evaluated tasks.

7 Conclusion

The paper presents a joint many-task model that handles multiple NLP tasks with growing depth in a single end-to-end model. Across five tasks, it achieves state-of-the-art or competitive results on four named tasks.

  • The joint many-task model handles multiple NLP tasks with growing depth in a single end-to-end model.
  • The model is successively trained according to linguistic hierarchies, feeds word representations into all layers, uses low-level predictions explicitly, and applies successive regularization.
  • The single model achieves state-of-the-art or competitive results on chunking, dependency parsing, semantic relatedness, and textual entailment.

A Training Details

The training setup uses pre-trained word and character n-gram embeddings, fixed 100-dimensional representations, task-ordered optimization, gradient clipping, and regularization.

  • Pre-training embeddings: Word and character n-gram embeddings are pre-trained with Skip-gram and used to initialize the model's representations.Word embeddings use lowercased English Wikipedia, while character n-gram embeddings use case-sensitive Wikipedia text.
  • Optimization: Each epoch trains tasks in the order POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment using mini-batch stochastic gradient descent.Batch sizes are 25 for POS tagging, chunking, and SICK, and 15 for dependency parsing.
  • Optimization: Growing gradient clipping uses the function min(3.0, depth), where depth controls the clipping value for different tasks.
  • Regularization: The model applies LSTM and successive regularization together with dropout to constrain training across its components.The successive regularization coefficient is 10^-2 for classifier parameters and 10^-5 for the other specified term.

B.1 Pre-Training with Skip-Gram Objective

Character n-gram embeddings are pre-trained with a Skip-gram negative-sampling objective and averaged to represent words. Their use improves performance, especially for unknown-word POS tagging and dependency parsing.

  • Pre-training: Character n-gram embeddings are trained with the Skip-gram objective using negative sampling on case-sensitive English Wikipedia.Case sensitivity is retained because it is important for some word types, such as named entities.
  • Word representation: A word's character representation is computed by averaging the embeddings of its K character n-grams.The embedding v_c(w) uses the parameterized embedding of each character n-gram.
  • Skip-gram objective: For each word-context pair, N negative context words are sampled and optimized with a logistic-sigmoid objective.The context-word weight vectors are parameterized for words without character information.
  • Analysis: Joint word and character n-gram embeddings improve unknown-word POS accuracy by about 19%.
  • Analysis: Character-level information is effective for dependency parsing, with a particularly large improvement in LAS.

C Analysis on Dependency Parsing

The dependency parser predicts a head for each word, but development-set inspection reveals root-structure errors, including multiple roots and missing roots. These errors motivate sentence-dependent root representations as future work.

  • The parser predicts a head for each word, so its outputs do not always form correct dependency trees.The analysis examines results from the JMTABC setting on 1,700 development-set sentences.
  • The development results contain 11 sentences with multiple root nodes and 11 with no root nodes.The annotations require exactly one root node per sentence.
  • In one example, the parser assigns both “counsels” and “need” as root children, although only “counsels” is correct.The parser treats two verbs in a sentence containing multiple internal sentences as sentence heads.
  • In another example, no word is connected to the root, while “chairman” should be the root’s child.Even after removing an internal relative clause, the simplified sentence remains incorrectly parsed.
  • The authors attribute these failures partly to using one parameterized root vector and propose sentence-dependent root representations.They suggest that varied root types require more flexible root representations.

D Analysis on Semantic Tasks

Semantic-task analysis exposes an entailment error involving antonymic words and examines how shared embeddings are transformed across task-specific layers. The results indicate that different tasks require different aspects of word similarity.

  • The entailment model labels “The surfer is riding a small wave” as entailed by a premise describing a big wave, although the gold label is contradiction.The model fails to use the distinction between “big” and “small.”
  • Co-occurrence-based word embeddings rarely distinguish antonyms from synonyms, and the entailment data contains only one example teaching this distinction.The authors identify these factors as limiting generalization for the semantic task.
  • Shared embeddings preserve semantic similarity initially, but semantic similarity is almost washed out after transformation through the POS layer.Table 15 uses nearest neighbors of “standing” to illustrate the change.
  • For semantic tasks, projected embeddings capture both syntactic and semantic similarities through simple task-specific linear transformations.The analysis argues that shortcut connections preserve information that lower layers may transform away.
Loading 1611.01587v5…