PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Sequence Modeling with Cross-View Training

Kevin Clark, Minh-Thang Luong, Christopher D. Manning, Quoc V. Le

arXiv:1809.08370v1cs.CL

TL;DR

Neural NLP models benefit from unlabeled data, but pre-training does not use task labels during representation learning and noise-based self-training is difficult for text. Cross-View Training addresses this by training restricted-view auxiliary modules to match full-model predictions on unlabeled examples, improving shared representations. Across seven NLP tasks, CVT achieves excellent results, especially with multi-task learning.

  • Problem

    Pre-training learns generally useful representations without task-specific labels, while applying input noise to improve self-training is difficult for discrete text.

  • Method

    CVT trains auxiliary prediction modules on restricted views of unlabeled inputs to match fixed predictions from a full model, using shared representations with supervised learning.

  • Results

    CVT achieves excellent results across seven NLP tasks, with particularly strong performance when combined with multi-task learning.

  • Takeaways & Limitations

    CVT provides a semi-supervised approach that leverages model predictions on unlabeled data to learn representations supporting accurate predictions from restricted input views.

  • Takeaways & Limitations

    For dependency parsing, modules attached to the second Bi-LSTM layer lack restricted views because the first layer propagates information about the whole sentence.

Abstract

from arXiv · show

Unsupervised representation learning algorithms such as word2vec and ELMo improve the accuracy of many supervised NLP models, mainly because they can take advantage of large amounts of unlabeled text. However, the supervised models only learn from task-specific labeled data during the main training phase. We therefore propose Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. On labeled examples, standard supervised learning is used. On unlabeled examples, CVT teaches auxiliary prediction modules that see restricted views of the input (e.g., only part of a sentence) to match the predictions of the full model seeing the whole input. Since the auxiliary modules and the full model share intermediate representations, this in turn improves the full model. Moreover, we show that CVT is particularly effective when combined with multi-task learning. We evaluate CVT on five sequence tagging tasks, machine translation, and dependency parsing, achieving state-of-the-art results.

1 Introduction

The paper introduces Cross-View Training (CVT) to address limitations of unsupervised pre-training and self-training for neural sequence models. CVT uses restricted-input auxiliary modules trained on unlabeled data, improving shared representations and performance across tasks.

  • Motivation: Pre-training learns representations from unlabeled data before supervised training, leaving its initial representation-learning phase untargeted by task-specific labels.This motivates a semi-supervised method that uses labeled and unlabeled examples together.
  • Motivation: Self-training lets a model teach itself from unlabeled examples, but its student receives the same predictions the model already produces, making the process somewhat tautological.Input noise can address this issue in vision, but is difficult to apply to discrete text.
  • Cross-View Training: CVT trains auxiliary prediction modules on restricted views of the input to match predictions from the full model, drawing on multi-view learning.For sequence tagging, an auxiliary module may use only the forward LSTM and therefore lack tokens to the right of the current one.
  • Cross-View Training: The auxiliary modules improve shared contextual representations by learning to predict accurately despite missing input, which also improves the full model.The auxiliary and primary modules are built on shared representations.
  • Evaluation: CVT improves previously published results across dependency parsing, supertagging, named entity recognition, part-of-speech tagging, chunking, and English–Vietnamese translation.The method can also be combined with multi-task learning, which improves results while reducing total training time.

2 Cross-View Training

CVT alternates supervised learning on labeled minibatches with agreement training on unlabeled minibatches. Auxiliary modules use restricted intermediate representations, while multi-task CVT jointly trains task-specific modules over shared data and representations.

  • Training procedure: CVT alternates standard supervised learning on labeled minibatches with agreement-based learning on unlabeled minibatches.The total objective combines supervised loss and CVT loss, L = Lsup + LCVT.
  • Auxiliary modules: Each auxiliary prediction module transforms a restricted intermediate representation into a distribution over labels.The restricted representation is chosen so the module sees only part of the input, depending on the task and architecture.
  • Computational properties: Auxiliary modules are used only during training, so CVT does not change inference time or the number of parameters in the fully trained model.Adding a few modules is generally cheap relative to the representation-building portion of the network.
  • Unlabeled learning: On unlabeled examples, auxiliary modules match the full model’s fixed soft predictions using a distance between probability distributions, implemented with KL divergence.The primary prediction is held fixed so auxiliary modules imitate it rather than updating it through the agreement objective.
  • Multi-task CVT: Multi-task CVT optimizes unlabeled-data agreement jointly across tasks while alternating with supervised updates for randomly selected task minibatches.This creates artificial all-tasks-labeled examples from unlabeled data and improves data efficiency and training time.

3 Cross-View Training Models

CVT extends sequence models with auxiliary predictors that receive restricted input views and trains them across sequence tagging, dependency parsing, and sequence-to-sequence architectures. The designs share encoder or decoder representations while withholding selected context.

  • Cross-View Training Models: CVT uses auxiliary prediction modules with restricted views of intermediate representations to provide cross-view training signals.The modules are attached to shared representations and are trained alongside the primary model.
  • Bi-LSTM Sentence Encoder: The encoder is a two-layer bidirectional LSTM whose second layer receives the concatenated outputs of the first layer.The model first constructs word and character-CNN representations, then applies stacked bidirectional LSTM layers.
  • Sequence Tagging: In sequence tagging, the primary module predicts each token label from its corresponding encoder output, while auxiliary modules use first-layer forward and backward LSTM representations.The forward module omits right context; future omits right context and the current token, with backward and past defined analogously.
  • Dependency Parsing: For dependency parsing, the model classifies typed directed candidate edges using bilinear scores followed by a softmax, with auxiliary modules withholding context for candidate heads and dependents.The parser uses word inputs rather than words plus part-of-speech tags, and second-layer modules would not retain restricted views because whole-sentence information has propagated.
  • Sequence-to-Sequence Learning: For sequence-to-sequence learning, CVT adds two auxiliary decoders: one uses attention dropout, and the other predicts the next target word rather than the current one.Because unlabeled examples lack target sequences for teacher forcing, hard auxiliary targets are produced by beam-search decoding from the primary decoder.

4 Experiments

Experiments evaluate CVT across seven NLP tasks and compare it with supervised, semi-supervised, and multi-task baselines. CVT performs strongly across tasks, is especially effective with limited labeled data, scales better with model size, and combines effectively with multi-task learning.

  • Results: CVT on its own outperforms or matches the best previously published results across all evaluated tasks.The evaluation covers CCG supertagging, chunking, NER, fine-grained NER, POS tagging, dependency parsing, and machine translation.
  • CVT + Multi-Task: Multi-task learning improves results on every task except fine-grained NER, sometimes by large margins, when combined with CVT.The shared encoder excludes machine translation because that task is substantially different and requires more training time.
  • CVT + Multi-Task: Multi-task learning works better with CVT than with ELMo, while the CVT models are on par with ELMo and faster and simpler to train.Increasing CVT+MultiTask to ELMo-sized LSTMs improves results further, significantly exceeding ELMo+Multi-task results.
  • Model Generalization: Both CVT and multi-task learning improve generalization, with CVT continuing to improve dev accuracy near 100% training accuracy.This pattern is reported for CCG, chunking, and NER, suggesting continued learning from unlabeled data after fitting the training set.
  • Results: Using 25% of labeled data, CVT performs as well as or better than a fully supervised model trained on 100%.The advantage over purely supervised learning grows as the amount of labeled data decreases.
  • Training Larger Models: CVT accuracy scales better with model size than supervised approaches, whose performance improves little when LSTM size increases.The authors suggest semi-supervised learning may enable larger models for NLP tasks with limited labeled data.

5 Related Work

The paper situates Cross-View Training among unsupervised representation learning, self-training, consistency regularization, multi-view learning, self-supervision, and multi-task learning.

  • Unsupervised Representation Learning: Pre-training learns representations from unlabeled data before supervised training, while self-training continually learns from labeled and unlabeled examples.Self-training has been effective for several NLP tasks but is less commonly used with neural models.
  • Consistency Regularization: Consistency regularization trains models to produce stable predictions under noisy or stochastically transformed inputs, especially in computer vision.The approach encourages distributional smoothness around nearby data points.
  • Multi-View Learning: Multi-view methods train models using distinct feature subsets, whereas CVT uses auxiliary modules within one unified model.Related approaches include co-training and co-regularization, where separate models teach each other on unlabeled data.
  • Self Supervision: Self-supervised methods train auxiliary modules on label-free tasks, while CVT bases its auxiliary losses on the model’s predictions.The distinction separates CVT from auxiliary objectives such as language modeling or image colorization.
  • Multi-Task Learning: Prior NLP multi-task learning often uses a small number of related tasks, while many-task systems are less commonly developed.The paper places its multi-task combination against this broader multi-task learning literature.

6 Conclusion

The conclusion presents Cross-View Training as a semi-supervised method that leverages model predictions on unlabeled data to learn representations supporting accurate restricted-view predictions.

  • 6 Conclusion: CVT achieves results across seven NLP tasks and is especially effective when combined with multi-task learning.The method is described as learning representations that yield accurate predictions even when part of the input is unavailable.

A Detailed Results

The detailed-results appendix adds precision, variability estimates, and additional prior-work comparisons to the paper’s test-set results.

  • A Detailed Results: Table 5 reports test-set results with two additional decimals, standard deviations across five runs, and more prior work.These additions provide finer-grained performance and variability reporting.

B Model Details

The appendix specifies architectures, task-specific preprocessing and evaluation choices, training details, multi-task data handling, and baseline configurations.

  • B Model Details: The models use two-layer CNN-BiLSTM encoders with task-specific prediction modules.This architecture is used for the appendix’s sequence-modeling implementations.
  • B Model Details: Sequence tagging uses BIOES labels for chunking and named entity recognition, with label smoothing at rate 0.1 on labeled data.Dependency parsing omits punctuation from evaluation and represents ROOT with a fixed vector hROOT.
  • B Model Details: Machine translation applies decoder dropout, uses a beam size of 10, and replaces target words occurring 5 or fewer times with UNK during training.The implementation is based heavily on the Google NMT Tutorial and uses several stated modeling improvements.
  • B Model Details: For multi-task learning, datasets are treated as separate tasks and overlapping training examples are removed to prevent evaluation leakage across tasks.The Penn Treebank tasks do not all share the same train, development, and test splits.
  • B Model Details: Training uses dropout, an exponential-moving-average of model weights, SGD with momentum, and fine-tuned GloVe initialization.The EMA slightly improves accuracy and significantly reduces variance across random initializations.
  • B Model Details: Baselines use the same architecture and hyperparameters as CVT, with specified word-dropout and virtual-adversarial-training settings.The multi-task ELMo baseline uses task-specific softmax-normalized weights for task prediction modules and shared weights for the encoder inputs.

C CVT for Image Recognition

CVT is adapted to image recognition by using CNN feature vectors and auxiliary prediction layers with restricted image views. On semi-supervised CIFAR-10, CVT improves models without CVT, while CVT combined with VAT remains competitive with current state-of-the-art approaches.

  • Approach: CVT uses CNN feature representations H(x_i) indexed by spatial coordinates, with deeper features potentially reflecting broader image regions.The experiments use CNNs whose feature vectors correspond to regions of the input image.
  • Approach: The primary CNN prediction layer averages H across spatial dimensions before feeding the resulting vector to a softmax layer.This produces a d-dimensional representation for the main prediction.
  • Approach: CVT adds n2 auxiliary prediction layers, each receiving a single feature vector rather than the spatially averaged representation.These auxiliary layers provide restricted views corresponding to local image regions.
  • Data: The CIFAR-10 experiments use only labels for a subset of training examples and treat the remaining examples as unlabeled.This creates the semi-supervised setting used to evaluate the image-recognition models.
  • Model: The CNN architecture contains 9 convolutional layers and 2 max pooling layers.The models adapt the convolutional network from Miyato et al. (2017b).
  • Model: 36 auxiliary softmax layers are attached to the CNN’s 6 × 6 feature-vector collection, with image patches ranging from 21 × 21 to 29 × 29 pixels.Some experiments also perturb student inputs when computing the CVT loss.
  • Results: CVT improves models without CVT on semi-supervised CIFAR-10, while CVT + VAT is competitive with current state-of-the-art approaches.Continuous input noise works better for images than language, producing stronger VAT results on CIFAR-10 than on NLP tasks.

D Negative Results

The paper reports several approaches that were ineffective or added little benefit in initial experiments, including hard targets, confidence thresholding, Mean Teacher, and purely supervised CVT.

  • Scope of findings: These negative findings come from early one-off experiments that were not pursued with further adjustment or tuning.The paper notes that the approaches might work with proper changes and tuning.
  • Hard vs soft targets: Initial experiments found hard self-training targets decreased performance compared with soft targets.The result is consistent with findings from knowledge distillation favoring soft targets.
  • Confidence thresholding: Neither hard nor soft confidence thresholding appeared to improve performance in the tested models.The hard version ignored low-confidence examples, while the soft version weighted examples by teacher confidence.
  • Mean Teacher: Mean Teacher provided little to no benefit in the experiments, although using EMA model weights at test time improved results slightly.Mean Teacher generates student targets from an exponential moving average of model weights.
  • Purely supervised CVT: Purely supervised CVT provided little to no benefit, suggesting the gains came from semi-supervised learning rather than auxiliary prediction layers alone.The supervised variant was tested as a possible regularizer without unlabeled-data training.
Loading 1809.08370v1…