PaperlayerView exact revision on arXiv

Source-linked AI summary

The Natural Language Decathlon: Multitask Learning as Question Answering

Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, Richard Socher

arXiv:1806.08730v1cs.CLcs.AIcs.LGstat.ML

TL;DR

The paper addresses the difficulty of developing general NLP models when evaluation is centered on individual tasks, datasets, and metrics. It introduces decaNLP and MQAN, which unify ten tasks as question answering and train them jointly; MQAN shows transfer, adaptation, zero-shot, and strong single-task results.

  • Problem

    General NLP models are difficult to develop within paradigms focused on a single task, dataset, and metric.

  • Method

    The paper introduces decaNLP, casts all tasks as question answering, and trains MQAN jointly without task-specific modules or parameters.

  • Results

    MQAN achieves comparable joint-task performance, improves transfer and domain adaptation, supports zero-shot capabilities, and reaches state-of-the-art WikiSQL results.

  • Takeaways & Limitations

    Question-based task specification and multi-pointer generation support a single model’s transfer, adaptation, and zero-shot use across diverse NLP tasks.

Abstract

from arXiv · show

Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, zero-shot relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answering over a context. Furthermore, we present a new Multitask Question Answering Network (MQAN) jointly learns all tasks in decaNLP without any task-specific modules or parameters in the multitask setting. MQAN shows improvements in transfer learning for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification. We demonstrate that the MQAN's multi-pointer-generator decoder is key to this success and performance further improves with an anti-curriculum training strategy. Though designed for decaNLP, MQAN also achieves state of the art results on the WikiSQL semantic parsing task in the single-task setting. We also release code for procuring and processing data, training and evaluating models, and reproducing all experiments for decaNLP.

1 Introduction

decaNLP benchmarks whether one model can generalize across ten diverse NLP tasks, unified by representing each as question answering. MQAN jointly trains all tasks without task-specific modules and shows transfer, adaptation, zero-shot, and single-task gains.

  • Motivation: decaNLP challenges models to simultaneously optimize ten diverse NLP tasks within one benchmark.The tasks include question answering, translation, summarization, parsing, sentiment analysis, inference, role labeling, relation extraction, dialogue, and pronoun resolution.
  • Unified formulation: All decaNLP tasks use natural-language questions to specify the task, with each example represented as context, question, and answer.This formulation replaces explicit task-specific modeling constraints with task descriptions expressed in natural language.
  • MQAN: MQAN combines dual coattention and a multi-pointer-generator decoder to multitask across all decaNLP tasks.The decoder can copy from the question or context, extending pointer mechanisms beyond context-only answering.
  • Findings: Joint MQAN training achieves performance comparable to ten separately trained MQANs and supports transfer learning, domain adaptation, and zero-shot text classification.The reported benefits include transfer to machine translation and named entity recognition, adaptation for sentiment analysis and inference, and zero-shot classification.
  • Findings: MQAN also achieves state-of-the-art performance on decaNLP’s WikiSQL semantic parsing task in the single-task setting.The model was not explicitly designed for one particular task.

2 Tasks and Metrics

decaNLP converts ten publicly available datasets into question-answering triplets and evaluates them with task-specific metrics combined into a decaScore. The benchmark spans language understanding, generation, dialogue, parsing, extraction, and pronoun resolution.

  • Dataset construction: decaNLP contains ten publicly available datasets whose examples are cast as question, context, and answer triplets.Figure 1 illustrates how the original datasets were preprocessed into question-answering problems.
  • Tasks: SQuAD provides question answering, while IWSLT and CNN/DailyMail provide machine translation and summarization tasks.SQuAD answers are copied from context; CNN/DailyMail requires balancing extraction with abstractive generation.
  • Tasks: MNLI, SST, QA-SRL, and QA-ZRE cover natural language inference, sentiment analysis, semantic role labeling, and zero-shot relation extraction.QA-ZRE evaluates performance on relations unseen during testing and accounts for unanswerable questions in its F1 metric.
  • Tasks: Goal-oriented dialogue uses WOZ dialogue-state tracking, while WikiSQL maps natural-language questions to SQL queries and MWSC tests pronoun resolution.WOZ uses turn-based dialogue-state exact match; WikiSQL uses logical-form exact match.
  • Metrics: The decaScore adds ten task-specific metrics, each scaled from 0 to 100, producing a total score between 0 and 1000.The additive combination avoids weighting issues across different metrics, and all metrics are case insensitive.

3 Multitask Question Answering Network (MQAN)

MQAN frames diverse NLP tasks as question answering and jointly trains a shared architecture across them. Its encoder combines recurrent, coattentive, and self-attentive processing, while its decoder attends to prior outputs and copies or generates tokens from multiple sources.

  • Architecture: MQAN represents each example as context, question, and answer sequences and jointly trains across tasks framed as question answering.The framework uses a shared multitask formulation rather than separate task-specific modules in the described architecture.
  • Attention: Dual coattention enriches representations of the input and question, extending coattention with self and recurrent context state.The decoder can therefore focus on encoded information relevant to each time step.
  • Architecture: The encoder applies recurrent, coattentive, and self-attentive layers to produce representations of both context and question sequences.These representations are designed to capture local and global interdependencies.
  • Decoding: Self-attention makes the decoder aware of previous outputs, while attention over the context prepares each next output.A recurrent LSTM state is then computed using the previous answer word and attention.
  • Decoding: The multi-pointer-generator produces output distributions over context tokens, question tokens, and an external vocabulary.Missing entries are set to zero after extending the distributions to their union, and two scalar switches regulate their relative importance.
  • Evaluation: Table 2 compares sequence-to-sequence baselines with self-attention, coattention, and question-pointer additions, including jointly trained models and an anti-curriculum variant.The anti-curriculum phase trains on SQuAD alone before switching to the fully joint strategy.

4 Experiments and Analysis

Experiments show that MQAN’s question pointer and multi-pointer-generator decoder improve performance across tasks, while multitask training supports transfer, adaptation, and zero-shot generalization.

  • Baselines and MQAN: 20 nF1 on SQuAD, 4 nF1 on QA-SRL, and 12 LFEM on WikiSQL result from adding self-attentive encoder and decoder layers.The WikiSQL model nearly matches the prior state-of-the-art validation result of 72.4% without a structured approach.
  • Baselines and MQAN: The question pointer raises SQuAD performance to 75.5 nF1 and boosts MNLI and MWSC above prior baselines.It also makes MQAN the highest-performing SQuAD QA model described here that does not explicitly model span extraction.
  • Baselines and MQAN: 72.4% lfEM and 80.4% database execution accuracy establish a new WikiSQL test state of the art over 71.7% and 78.5%.
  • Multitask Training: QA-ZRE performance increases 11 F1 points over the highest single-task models, supporting better multitask generalization for zero-shot learning.
  • Multi-Pointer-Generator: More than 50 percent performance drops on external-vocabulary tasks persist until the question pointer is added to S2S baselines.The coattended question gives decoder access to question information, supporting generation when copying is less appropriate.
  • Adaptation to New Tasks: MQAN pretraining requires fewer fine-tuning iterations and reaches better final performance than random initialization for English-to-Czech translation and NER.Figure 4 compares decaNLP-pretrained and randomly initialized models on a new language pair and a new task.

5 Related Work

Prior NLP research developed transfer learning, multitask architectures, and methods for mitigating interference, motivating a unified model across diverse tasks. MQAN extends this direction with a single task-agnostic model that shares parameters efficiently across decaNLP.

  • Transfer and multitask learning: Transfer learning and multitask learning exploit relatedness among tasks to learn more generally useful representations.Prior work includes pretrained embeddings, transferable intermediate representations, and unified architectures across several NLP tasks.
  • Optimization: Multitask learning must capitalize on task relatedness while mitigating interference from dissimilar tasks.This optimization challenge connects multitask learning with multi-objective optimization and catastrophic forgetting.
  • MQAN: MQAN is a single model for diverse complex NLP tasks without task-specific modules, intermediate representations, or substantial evidence of catastrophic interference.The model avoids parse trees, chunks, and POS tags while achieving reasonable performance across the decaNLP tasks.
  • MQAN: MQAN performs nearly as well or better jointly than separately on each task with the same parameter cap.Separate per-task MQANs would require far more trainable parameters, suggesting more efficient parameter sharing in the multitask setting.

6 Conclusion

The paper introduces decaNLP as a ten-task benchmark unified through question answering and presents MQAN as a general model for this setting. Joint training with anti-curriculum improves performance, while pretrained MQAN weights support transfer and zero-shot adaptation.

  • Contributions: decaNLP measures NLP performance across ten seemingly disparate tasks unified as question answering.The benchmark provides a common perspective for evaluating general NLP models.
  • Contributions: MQAN jointly handles all decaNLP tasks without task-specific modules by using a multi-pointer-generator decoder.The decoder uses natural-language questions as descriptions of tasks.
  • Results: Anti-curriculum learning further improves jointly trained MQAN performance.The conclusion reports this improvement in the multitask decaNLP setting.
  • Results: Pretrained MQAN weights improve performance on new tasks and provide zero-shot domain adaptation for text classification.These capabilities include transfer learning and adaptation to new domains after decaNLP training.
  • Resources: The benchmark, experiments, and public code are intended to encourage further research into general NLP models.Released resources cover data processing, training, evaluation, and experiment reproduction.

A Further Related Work

The related-work appendix reviews task-specific NLP approaches underlying decaNLP, while highlighting question-answering formulations that reduce specialized constraints and enable broader capabilities.

  • Question Answering: SQuAD-based question answering commonly extracts contiguous answers by predicting document start and end tokens.This assumption does not hold for question answering in general, motivating models that can generate answers.
  • Machine Translation: The IWSLT English-German dataset was chosen to match the approximate training-data scale of the other decaNLP tasks.The authors encourage larger corpora and multiple language pairs but avoid skewing the first challenge toward translation.
  • Natural Language Inference: MNLI succeeds SNLI as a multi-genre inference dataset, with recent work connecting NLI models to question-answering models.Attentional approaches align premise and hypothesis words, while non-attentional sentence representations have nearly closed the gap.
  • Structured Prediction: Question-based formulations for SRL and relation extraction reduce task-specific constraints and support extraction beyond syntax or relations seen during training.QA-ZRE is designed to evaluate zero-shot extraction of unseen relations, while QA-SRL permits structure beyond syntax-based representations.
  • Dialogue and Semantic Parsing: Goal-oriented dialogue commonly uses structured dialogue-state tracking, while WikiSQL systems use sketches, type systems, or dependency graphs for semantic parsing.These task-specific approaches contrast with the shared question-answering formulation used in decaNLP.

B Preprocessing and Training Details

Training preprocesses all tasks with shared tokenization and vocabulary choices, while task-specific batches are dynamically sized under a token-budget constraint.

  • Preprocessing: All data is lowercased and tokenized with reversible RevTok tokenization for evaluation-compatible detokenization.SQuAD contexts longer than 400 tokens were excluded during training.
  • Preprocessing: The generative vocabulary contains the 50000 most frequent words across all decaNLP training sets.This vocabulary is shared across the multitask training data.
  • Model inputs: MQAN uses 300-dimensional Common-Crawl GloVe embeddings and concatenated 100-dimensional character n-gram embeddings.Words without GloVe entries receive zero vectors.
  • Batching: Batches contain examples from one task and are dynamically constructed so context, question, and answer tokens fit within a 10000-token budget.Answer tokens receive five times the weighting in the batching constraint.

C Multitask Question Answering Network (MQAN) Encoder

The MQAN encoder independently projects context and question representations, aligns them with dual coattention, compresses the resulting features, and refines them with self-attention and recurrent encoding before decoding answers.

  • Independent Encoding: The encoder represents context, question, and answer sequences as token-embedding matrices, then projects inputs into a common d-dimensional space.
  • Independent Encoding: A shared bidirectional LSTM produces contextual representations for the projected context and question sequences.
  • Alignment: Dummy embeddings allow context and question tokens to remain unaligned when no token in the other sequence is appropriate.
  • Dual Coattention: Column-wise softmax normalization converts cross-sequence dot-product similarities into alignments used to transfer relevant information between sequences.
  • Compression and Final Encoding: The encoder drops the dummy-embedding column, compresses dual-coattention features with separate BiLSTMs, and applies multi-head attention, feedforward layers, and final BiLSTMs.The resulting matrices are passed to the decoder to generate the answer.

D Curriculum Learning

MQAN multitask training compares round-robin schedules that begin with all tasks, harder tasks, or easier tasks before switching to joint training; anti-curriculum schedules address uneven task difficulty but do not eliminate trade-offs.

  • Fully Joint Training: Fully joint training samples batches round-robin from all tasks throughout training, yet several tasks remain below their single-task performance.The performance gap correlates with the number of iterations those tasks require to converge in single-task training.
  • Anti-Curriculum Training: Anti-curriculum schedules first train a subset of typically more difficult tasks, then switch to fully joint training over all tasks.
  • Anti-Curriculum Training: SQuAD-first training is motivated by the shared question-answering formulation, allowing the model to learn context retrieval before switching among tasks or generating words independently.
  • Anti-Curriculum Training: Adding IWSLT, CNN/DM, and optionally MNLI to the initial phase tests whether large, long-answer, and diverse tasks improve the curriculum.
  • Curriculum Training: Training easy tasks first performs worse both on excluded tasks, especially SQuAD and IWSLT, and on tasks included in the initial curriculum.
  • Trade-offs: Different task orderings improve some tasks while reducing others, and a gap remains between multitask and single-task performance.
Loading 1806.08730v1…