PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Sandeep Subramanian, Adam Trischler, Yoshua Bengio, Christopher J Pal

arXiv:1804.00079v1cs.CL

TL;DR

Learning general-purpose sentence representations remains difficult because task-specific systems learn contextual word representations from scratch and no training objective has clear consensus. The paper combines diverse objectives in a shared recurrent encoder, achieving stronger transfer and low-resource results than prior methods while leaving inductive-bias interpretation for future work.

  • Problem

    General-purpose sentence representations are needed because contextual word representations learned from scratch are not always feasible in low-resource settings, and no training objective has clear consensus.

  • Method

    The paper shares one recurrent sentence encoder across sequence-to-sequence and classification objectives, including multilingual NMT, parsing, skip-thought vectors, and natural language inference.

  • Results

    The shared multi-task representations yield consistent improvements across transfer tasks, including 6% gains on TREC over Infersent and comparable Quora performance using 6% of the available training data.

  • Takeaways & Limitations

    Combining diverse training objectives produces reusable sentence representations that support transfer and low-resource learning across novel tasks.

  • Takeaways & Limitations

    The paper leaves understanding and interpreting the model’s learned inductive biases, including changes from adding tasks, for future work.

Abstract

from arXiv · show

A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

1 INTRODUCTION

General-purpose sentence representations are motivated by the difficulty of learning contextual word representations reliably in low-resource settings and the lack of consensus on suitable training objectives. The paper addresses this by sharing one recurrent encoder across diverse weakly related tasks to capture multiple sentence characteristics for transfer.

  • Low-resource settings make it difficult to learn contextual word representations reliably from scratch, motivating general-purpose sentence representations.
  • No clear consensus exists on which training objective or methodology best suits general-purpose sentence representations.
  • The framework shares a single recurrent sentence encoder across multiple weakly related tasks with different inductive biases.The tasks include skip-thoughts, machine translation, natural language inference, and constituency parsing.
  • The model aims to encode several sentence aspects without assuming that a fixed-length vector captures a sentence’s entire meaning.
  • The combined framework improves performance on novel transfer tasks and supports low-resource learning, reaching comparable performance on Quora duplicate questions with 6% of the available training set.

2 RELATED WORK

Prior work spans compositional, recurrent, autoencoding, discourse, and multi-task approaches to reusable sentence representations, but important trade-offs remain. This paper distinguishes its fixed-length, transfer-oriented framework from related multi-task sequence-to-sequence models and analyzes how tasks encode information.

  • Sentence representation research includes word-composition functions, symbolic-distributed combinations, and neural recurrent or convolutional compositions.
  • Skip-thoughts learn reusable representations from weakly labeled data but can require weeks or months to train, while word averages can match more complex models.
  • Discourse-based objectives can also be used to learn sentence representations.
  • Unlike a related many-to-many multi-task model, this work avoids attention to obtain fixed-length vectors and targets improvements on unseen tasks rather than only training tasks.
  • The paper analyzes how different tasks contribute to encoding distinct information signals in the learned representations.

3 SEQUENCE-TO-SEQUENCE LEARNING

The sequence-to-sequence framework encodes an input sequence into a fixed-length vector and uses a decoder to generate an output sequence. This paper uses a shared bidirectional GRU encoder with task-specific decoders and omits attention to preserve a single sentence representation.

  • Sequence-to-sequence models use an encoder-decoder architecture to model the conditional distribution of output sequences given input sequences.
  • The encoder maps the input sequence to a fixed-length representation hx, which conditions the decoder’s output generation.
  • The hidden representation hx is typically the final hidden state of the encoder RNN.
  • The model omits attention so the sentence representation remains a single fixed-length distributed vector, while conditioning every decoding step on hx reduces information attenuation.
  • A one-to-many architecture shares one bidirectional GRU encoder across tasks and uses task-specific GRU decoders to produce outputs.

4 TRAINING OBJECTIVES & EVALUATION

The framework jointly trains a shared recurrent encoder across diverse sentence-representation objectives, using multiple large datasets and a specified task-sampling procedure. The objectives include skip-thought prediction, multilingual translation, constituency parsing, and natural language inference.

  • Multi-task training setup: Multi-task learning jointly trains a shared recurrent sentence encoder across multiple weakly related tasks.The framework is motivated by the expectation that diverse task inductive biases support generalization to novel tasks.
  • Training objectives: The selected objectives prioritize diversity, large training datasets, and standalone effectiveness for learning sentence representations.
  • Training objectives: Skip-thought training predicts the previous and next sentences from a current sentence using BookCorpus.The encoder and the previous- and next-sentence decoders are parameterized as separate recurrent neural networks.
  • Training objectives: Multilingual neural machine translation uses approximately 4.5 million English-German and 40 million English-French sentence pairs.Multiple target languages are used as part of the training design.
  • Training objectives: Constituency parsing maps sentences to linearized parse trees using 3 million weakly labeled parses plus duplicated gold Penn Treebank parses.Gold parses are duplicated five times and mixed with noisy parses at roughly a 1:5 gold-to-noisy ratio.
  • Training objectives: Natural language inference encodes premise-hypothesis pairs into fixed-length vectors and classifies entailment, contradiction, or neutrality.Training uses about 1 million sentence pairs from SNLI and MultiNLI, with vector combinations fed to an MLP.
  • Multi-task training setup: The training algorithm samples sequence-to-sequence tasks uniformly after each parameter update and inserts an NLI minibatch after every ten such updates.The formal setup uses a shared encoder, task-specific decoders, task datasets, and a shared loss optimized with Adam.

5 EVALUATION STRATEGIES, EXPERIMENTAL RESULTS & DISCUSSION

The evaluation tests fixed sentence representations through transfer learning, low-resource classification, word-embedding benchmarks, probing tasks, and qualitative analyses. Across these evaluations, adding diverse tasks and model capacity improves transfer performance, while parsing and multilingual NMT strengthen syntactic encoding.

  • Evaluation protocol: The evaluation uses fixed sentence representations as features for low-complexity classifiers on novel supervised tasks, including artificially low-resource settings.The representation model parameters remain frozen during transfer evaluation.
  • Transfer results: Adding more tasks improves transfer performance, while larger and deeper sentence encoders provide additional gains.The comparison includes 10 supervised transfer tasks and capacity ablations.
  • Transfer results: 1.1-2.0% gains over Infersent occur on MR, CR, SUBJ, and MPQA, while TREC improves 6% over Infersent and roughly 2% over CNN-LSTM.The TREC result also outperforms a competitive supervised baseline.
  • Transfer results: The model gains 2.3% on MRPC, and adding constituency parsing improves SICK-R and SICK-E performance.The MRPC gains close the gap with supervised approaches trained from scratch.
  • Low-resource results: A fixed-representation MLP improves Quora duplicate-question accuracy by 0.2-0.5% over decomposable attention and performs strongly with roughly 6% of the training data.In the low-resource setting, it also outperforms Siamese, Multi-Perspective CNN, and Deconv LVM models.
  • Representation analysis: Multilingual NMT and parsing improve syntactic-property encoding, while parsing also improves encoding of sentence length and word order.The representations outperform skip-thoughts and match Infersent for image-caption retrieval; cosine similarity correlates reasonably with semantic relatedness.

6 CONCLUSION & FUTURE WORK

The framework combines diverse training signals to learn general-purpose fixed-length sentence representations and achieves competitive or superior results to previous methods. Future work focuses on interpreting learned inductive biases and exploring additional applications.

  • 6 CONCLUSION & FUTURE WORK: The multi-task framework combines sequence-to-sequence and classification objectives to learn general-purpose fixed-length sentence representations.Its tasks include multilingual NMT, constituency parsing, skip-thought vectors, and natural language inference.
  • 6 CONCLUSION & FUTURE WORK: The learned representations yield competitive or superior results to previous general-purpose sentence representation methods.
  • 6 CONCLUSION & FUTURE WORK: Table 5 probes whether sentence characteristics and syntactic properties can be inferred from the learned representations using prediction accuracy.The probes cover length, word content, word order, voice, tense, and top-level syntactic sequence.
  • 6 CONCLUSION & FUTURE WORK: Future work will study how learned inductive biases change when additional tasks are added beyond the current analysis of sentence characteristics and syntax.

7 MODEL TRAINING

The model uses a shared encoder and task-specific decoders, while architectural and hyperparameter choices were not tuned because no clear tuning criterion was identified.

  • 7 MODEL TRAINING: The shared encoder uses a common word-embedding lookup table and GRU, while each task has separate decoder components.Task-specific decoders include word-embedding lookups, conditional GRUs, and fully connected projections to target vocabularies.
  • 7 MODEL TRAINING: The experiments include unidirectional, bidirectional, and two-layer bidirectional GRUs.
  • 7 MODEL TRAINING: Architectural details and hyperparameters were not tuned because the authors could not identify a clear criterion for tuning them.
  • 7 MODEL TRAINING: Gains on a specific task do not often translate into better transfer performance.

8 VOCABULARY EXPANSION & REPRESENTATION POOLING

The method tunes how hidden states are pooled into sentence representations and expands vocabulary by mapping pretrained GloVe embeddings into the model’s embedding space.

  • 8 VOCABULARY EXPANSION & REPRESENTATION POOLING: The representation-generation method is selected between the last hidden state and max-pooling over hidden states using validation performance.
  • 8 VOCABULARY EXPANSION & REPRESENTATION POOLING: Max-pooling performs best on MR, CR, SUBJ, and MPQA sentiment tasks, whereas the last hidden state performs better on all other tasks.
  • 8 VOCABULARY EXPANSION & REPRESENTATION POOLING: Vocabulary expansion trains a linear regression from pretrained GloVe embeddings to the model’s word-embedding space.

9 MULTI-TASK MODEL DETAILS

The multi-task models combine task-specific sentence encoders through concatenated hidden representations, with configurations varying in tasks, depth, and dimensionality. Tables 3 and 5 instead use non-concatenated representations.

  • 9 MULTI-TASK MODEL DETAILS: The ablation configurations use task abbreviations including STN, Fr, De, NLI, STP, and Par.
  • 9 MULTI-TASK MODEL DETAILS: The +STN +Fr +De configuration concatenates final hidden vectors from forward and bidirectional 1500-dimensional GRUs.
  • 9 MULTI-TASK MODEL DETAILS: The +STN +Fr +De +NLI configuration concatenates final hidden vectors from two 1500-dimensional bidirectional GRUs, one trained without NLI.
  • 9 MULTI-TASK MODEL DETAILS: The +STN +Fr +De +NLI +L configuration uses two 2048-dimensional bidirectional GRUs, with one trained without NLI.
  • 9 MULTI-TASK MODEL DETAILS: The +STN +Fr +De +NLI +L +STP configuration uses two 2048-dimensional bidirectional GRUs, with one trained without STP.
  • 9 MULTI-TASK MODEL DETAILS: The +STN +Fr +De +NLI +2L +STP configuration combines a two-layer and a one-layer 2048-dimensional bidirectional GRU, with one trained without STP.
  • 9 MULTI-TASK MODEL DETAILS: The configuration adding parsing uses two 2048-dimensional bidirectional GRUs, with one trained without parsing.
  • 9 MULTI-TASK MODEL DETAILS: Tables 3 and 5 do not concatenate representations from multiple models.

10 DESCRIPTION OF EVALUATION TASKS

The evaluation covers transfer tasks spanning classification, semantic relatedness, retrieval, sentence characteristics, and syntax. It uses task-specific metrics and probes whether representations encode lexical, structural, and syntactic information.

  • Text classification: The benchmark suite includes sentiment, question type, subjectivity, and opinion-polarity classification, evaluated with accuracy.
  • Pairwise text classification: Paraphrase identification on MRPC is evaluated as binary classification using accuracy and F1.
  • Semantic relatedness and entailment: SICK evaluates sentence relatedness with Pearson correlation and entailment with classification accuracy.
  • Semantic textual similarity: STS12–STS16 and STSB compare sentence relatedness through cosine similarity, using Pearson correlation across diverse data sources.
  • Sentence characteristics and syntax: Additional probes test duplicate questions, sentence length, word content, word order, active/passive voice, tense, and top syntactic sequence.The characteristic tasks use an 8-way length classifier, binary content and order classifiers, while TSS is a 20-way classification problem.
Loading 1804.00079v1…