PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Self-Training for Neural Sequence Generation

Junxian He, Jiatao Gu, Jiajun Shen, Marc'Aurelio Ranzato

arXiv:1909.13788v3cs.LGcs.CLstat.ML

TL;DR

Sequence-generation self-training remains insufficiently understood because pseudo-targets can be far from ground truth. The paper analyzes self-training, identifies noise as critical, and proposes noisy self-training, which improves supervised baselines on translation and summarization benchmarks.

  • Problem

    Self-training is well studied for classification but its effectiveness and important ingredients remain unclear for complex neural sequence-generation tasks.

  • Method

    The paper evaluates self-training, uses ablations and synthetic experiments to study its mechanisms, and injects input noise to create noisy self-training.

  • Results

    Noisy self-training improves supervised baselines across machine translation and text summarization, exceeding the supervised baseline by over 6 BLEU points on WMT100K and consistently improving Gigaword results.

  • Takeaways & Limitations

    Noise injected during self-training plays a critical role through smoothing, and noisy self-training can improve generalization when labeled data is scarce as well as in higher-resource settings.

  • Takeaways & Limitations

    Performance is sensitive to the synthetic-noise level, with intermediate values working best, and the paper does not identify an a priori way to select that level.

Abstract

from arXiv · show

Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model's prediction (i.e. the pseudo-parallel data). While self-training has been extensively studied on classification problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space. In this work, we first empirically show that self-training is able to decently improve the supervised baseline on neural sequence generation tasks. Through careful examination of the performance gains, we find that the perturbation on the hidden states (i.e. dropout) is critical for self-training to benefit from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs. Such effect helps the model correct some incorrect predictions on unlabeled data. To further encourage this mechanism, we propose to inject noise to the input space, resulting in a "noisy" version of self-training. Empirical study on standard machine translation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.

1 INTRODUCTION

The paper revisits self-training for neural sequence generation, where its effectiveness and key ingredients remain unclear. It evaluates self-training and analyzes why it can improve supervised models using unlabeled data.

  • 1 INTRODUCTION: Self-training augments a small labeled dataset with pseudo-labeled unlabeled examples produced by a teacher model, then trains a student model on the combined data.The paper describes this as a simple semi-supervised method originally designed for classification.
  • 1 INTRODUCTION: Self-training has not been studied extensively for natural-language generation because predicted targets can differ substantially from ground truth, particularly in low-resource settings.Prior machine-translation studies reported only limited gains, leaving the source of self-training's effectiveness unclear.
  • 1 INTRODUCTION: The paper asks whether inaccurate pseudo targets are catastrophic and which ingredients make self-training effective for sequence generation.These questions cover both performance on translation and summarization and the mechanisms behind any gains.
  • 1 INTRODUCTION: Experiments first find significant gains over a supervised baseline, then use ablations to identify decoding and hidden-state perturbation as contributors to self-training success.Dropout is reported as the crucial ingredient responsible for most gains because it helps prevent convergence to the base model's local optimum.
  • 1 INTRODUCTION: The paper motivates input perturbation as a noisy self-training variant after toy experiments suggest noise can propagate labels to nearby inputs and sometimes correct errors.Noisy self-training is subsequently evaluated on machine translation and text summarization.

2 SELF-TRAINING

Classic self-training iteratively generates predictions for unlabeled inputs and uses selected pseudo-parallel data with labeled data to update the model. Its formulation highlights choices about pseudo-data selection and data combination, while the paper connects its unlabeled loss to entropy regularization.

  • 2 SELF-TRAINING: Classic self-training trains a base model on labeled parallel data, predicts labels for unlabeled instances, selects pseudo-parallel examples, and retrains on their union.The process repeats until convergence or a maximum number of iterations.
  • 2 SELF-TRAINING: The method has two key design choices: how to select the pseudo-data subset and how to combine real and pseudo-parallel data.Selection may use confidence scores or the whole pseudo-data set, while joint training can require weighting the two sources.
  • 2 SELF-TRAINING: The unlabeled loss averages the model's log probability of pseudo-targets generated from the previous iteration's parameters.The previous-iteration parameters remain fixed during the current iteration, initially matching the supervised baseline.
  • 2 SELF-TRAINING: Figure 1 tracks BLEU on WMT100K for the supervised baseline and self-training variants across three iterations, distinguishing ST from noisy ST.The caption defines ST as self-training and NST as noisy self-training.
  • 2 SELF-TRAINING: The paper relates the self-training objective to entropy regularization, which favors low-density separation between classes in classification settings.This connection provides a regularization perspective on the objective, although the section concerns conditional sequence generation.

3 A CASE STUDY ON MACHINE TRANSLATION

The machine-translation case study evaluates self-training on WMT100K and examines why pseudo-training improves a supervised baseline despite using model-generated targets. Ablations indicate that dropout accounts for most of the gains, while beam search contributes only partially.

  • Setup: The study evaluates self-training and its contributing factors on a machine-translation task using WMT100K experiments.The analysis includes performance evaluation and ablation experiments.
  • Observations: Self-training improves test BLEU across the first three iterations, converging to 3 BLEU points above the initial baseline.Pseudo-training improves BLEU even when trained only on the model’s predictions, and fine-tuning increases performance further.
  • Observations: The improvement is unexpected because pseudo-targets come from the base model, so translation errors might persist or become magnified.This differs from back-translation, which can introduce knowledge through another translation model and real monolingual targets.
  • Ablation analysis: Continuing training from the baseline improves performance by 1.9 BLEU points, comparable to initializing from a random model.This result rules out a training-from-scratch trajectory as the sole explanation for the gain.
  • Ablation analysis: Beam-search decoding contributes partially, while dropout is the crucial perturbation enabling most of self-training’s performance gains.Sampling reduces performance by 0.5 BLEU, sampling with dropout still gains 1.4 BLEU, and removing dropout substantially reduces pseudo-training improvements.

4 NOISE IN SELF-TRAINING

Noise helps self-training by smoothing predictions over unlabeled inputs, while input perturbations further encourage this effect. Experiments show that noisy self-training improves translation performance, with dropout and input noise providing complementary benefits.

  • The role of noise: Local smoothness maps similar inputs to similar targets, regularizing the larger monolingual data space beyond the real parallel data.This smoothing can support later fine-tuning, although excessive smoothness may collapse predictions toward a constant value.
  • The role of noise: ST pseudo-training improves smoothness, which reduces test errors during subsequent fine-tuning on the toy summation task.Table 3 reports smoothness and symmetry after pseudo-training, but test errors after fine-tuning at the first iteration.
  • Noisy self-training: Input perturbation defines noisy self-training; shuffling the two integers improves smoothness and symmetry while reducing test errors on the toy task.The shuffling perturbation also encourages the commutative behavior expected for integer addition.
  • Noisy self-training: Smoothing can self-correct isolated large errors when neighboring predictions are more accurate, but it can also worsen errors surrounded by poor predictions.Thus smoothing alone need not improve pseudo-training performance, whereas fine-tuning benefits consistently in the reported datasets.
  • Observations on machine translation: On WMT100, noisy self-training outperforms the supervised baseline by over 6 BLEU points and normal self-training by 3 BLEU points.Synthetic noise and paraphrasing perform similarly; synthetic noise is used subsequently because it is simpler and more general.
  • Observations on machine translation: Noisy self-training without dropout improves the baseline by 2.3 BLEU points, while adding dropout yields another 1.4 BLEU improvement.The result indicates that input perturbation and hidden-state perturbation are complementary.

5 EXPERIMENTS

Experiments evaluate noisy self-training across machine translation and summarization, resource settings, data scales, and noise levels. Noisy self-training generally improves supervised baselines, with gains depending on monolingual data, domain, and noise magnitude.

  • Machine Translation: Noisy self-training improves translation baselines by 1–5 BLEU in almost all tested cases, including a very weak supervised system.The evaluation covers WMT14 English-German and low-resource FloRes English-Nepali.
  • Machine Translation: Noisy self-training beats back-translation on WMT100K and FloRes English-origin tests, whereas back-translation performs better on Nepali-origin tests.The comparison is attributed to the monolingual data language matching the test-set origin.
  • Text Summarization: Noisy self-training consistently improves Gigaword baselines, with especially large gains using 100K or 640K labeled examples.It beats back-translation with 100K parallel data but underperforms it with 640K parallel data.
  • Analysis: Increasing parallel-data size always improves the baseline, while noisy self-training gains are larger at intermediate parallel-data sizes.The analysis fixes monolingual data at 20M News Crawl sentences and repeats noisy self-training for three iterations.
  • Analysis: Performance improves as monolingual-data size increases, but the gains show diminishing returns.The analysis evaluates 100K, 500K, 1.5M, 3.8M, and 20M monolingual sentences.
  • Analysis: Performance is sensitive to synthetic-noise level, with intermediate word-blanking probabilities performing best.The tested probabilities are 0.2, 0.4, 0.6, and 0.8, alongside noise-free self-training.
  • Noise Process on Parallel Data Only: Applying noise to parallel data with fake targets yields only +0.4 BLEU, compared with +1.0 BLEU from 100K monolingual data and +3.7 BLEU from 3.8M.Using real targets with noisy sources performs much worse than the baseline.

6 RELATED WORK

The paper situates self-training within pseudo-label semi-supervised learning and connects noisy self-training to input- or feature-space perturbation methods. It also reports results using a single iteration for the parallel-data noise comparison.

  • Pseudo-Label Methods: Self-training assigns model predictions as pseudo labels to unlabeled data, while co-training uses two independent feature sets and confidence scores.Co-training is presented as a related pseudo-label approach that can reduce modeling bias.
  • Experimental Comparison: The parallel-data noise comparison reports all results after one iteration.The table distinguishes parallel data with real or fake targets from the normal monolingual-data noisy self-training process.
  • Perturbation Methods: Input- or feature-space perturbation methods provide the broader context for self-training with dropout and noisy self-training.The cited classification methods use perturbations as data augmentation techniques.

7 CONCLUSION

The paper revisits self-training for neural sequence generation and identifies noise-induced smoothing as central to its success. Experiments show noisy self-training improves generalization across machine translation and summarization in low- and high-resource settings.

  • Conclusion: Noise injected during self-training plays a critical role by producing a smoothing effect that improves generalization, particularly when labeled data is scarce.The conclusion is based on comprehensive ablation analysis and synthetic experiments.
  • Conclusion: Noisy self-training explicitly perturbs the input and improves performance across machine translation and text summarization in both low- and high-resource settings.This variant is proposed to encourage the behavior identified in the analysis.

A EXPERIMENTS DETAILS

The experiments use standard fairseq-based optimization and validation procedures, with dataset-specific preprocessing and model architectures. Most experiments run on eight GPUs with a 33K-token effective batch size.

  • Training Setup: All experiments use Adam with β1 = 0.9, β2 = 0.98, and ϵ = 1e-8, alongside fairseq learning-rate schedules and label smoothing.Except for the toy sum dataset, experiments run on eight GPUs with an effective batch size of 33K tokens.
  • Model Architectures: The toy sum model and WMT10K baseline use single-layer LSTMs with dropout rate 0.3.The toy model uses 32-dimensional embeddings and hidden states, while WMT10K uses 256-dimensional embeddings and hidden states.

A.2 JUSTIFICATION OF THE WMT100K BASELINE

The WMT100K baseline is trained with specified optimization and model-selection settings, with validation loss used to select the best checkpoint.

  • The baseline uses Adam with learning rate 0.0005 and trains for 30K update steps without early stopping.The best model is selected according to validation loss.
  • Self-training uses 300K pseudo-training update steps followed by 100K fine-tuning update steps.
  • Figure 4 plots validation loss against update steps for the baseline model on WMT100K.

B COMPARISON REGARDING SEPARATE TRAINING, JOINT TRAINING, AND FILTERING

On WMT100K, separate training with all monolingual data is retained because it outperforms the compared joint-training and filtering variants.

  • Separate training with all monolingual data produces the best performance among the compared variants.The authors therefore use this version in the paper.
  • Filtering improves joint training but still underperforms separate-training methods by over 1.5 BLEU points.
  • Within separate training, filtering produces results comparable to using all monolingual data.
  • The filtering experiments use the most confident 2.5M, 3M, and 3.8M monolingual examples across three iterations.

C ADDITIONAL RESULTS ON THE TOY SUM DATASET

Additional toy-sum experiments visualize how noisy self-training changes errors across the full data space over the first two iterations.

  • The experiments show error heat maps for the entire toy-sum data space over the first two iterations.Deeper colors represent larger errors.
  • At each iteration, pseudo-training smooths the data space, while fine-tuning benefits from this smoothing and greatly reduces errors.
  • The pseudo-training model is initialized from the previous iteration to examine decoding changes caused by injected noise.
Loading 1909.13788v3…