Source-linked AI summary
Professor Forcing: A New Algorithm for Training Recurrent Networks
Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, Yoshua Bengio
TL;DR
Recurrent networks can generate poorly when free-running predictions drift from the teacher-forced contexts used in training. Professor Forcing uses adversarial domain adaptation to align hidden-state and output dynamics across these modes, and reports improved likelihood, sample quality, and long-horizon behavior across several sequential tasks.
Problem
Free-running prediction can compound small errors because its conditioning context diverges from sequences seen during teacher-forced training, limiting robust long-term generation.
Method
Professor Forcing trains a discriminator to distinguish teacher-forced and free-running behavior, encouraging the generative RNN to make their hidden-state and output dynamics similar.
Results
Professor Forcing acts as a regularizer, improving test likelihood and sample quality across sequential tasks, while T-SNEs show more similar training and sampling dynamics.
Takeaways & Limitations
The method is intended to support robust generation beyond training-sequence lengths and improve modeling of long-term dependencies.
Takeaways & Limitations
The framework allows generated output sequences to have lengths different from their conditioning sequences, depending on the task.
Abstract
from arXiv · showhide
The Teacher Forcing algorithm trains recurrent networks by supplying observed sequence values as inputs during training and using the network's own one-step-ahead predictions to do multi-step sampling. We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of the recurrent network to be the same when training the network and when sampling from the network over multiple time steps. We apply Professor Forcing to language modeling, vocal synthesis on raw waveforms, handwriting generation, and image generation. Empirically we find that Professor Forcing acts as a regularizer, improving test likelihood on character level Penn Treebank and sequential MNIST. We also find that the model qualitatively improves samples, especially when sampling for a large number of time steps. This is supported by human evaluation of sample quality. Trade-offs between Professor Forcing and Scheduled Sampling are discussed. We produce T-SNEs showing that Professor Forcing successfully makes the dynamics of the network during training and sampling more similar.
1 Introduction
Professor Forcing addresses the mismatch between teacher-forced training and free-running generation in recurrent networks. The paper proposes matching these behaviors to improve long-term sampling and reports benefits across sequential tasks.
- Motivation: RNNs model sequential data with ordered conditional distributions, and teacher forcing feeds ground-truth values back to predict later outputs.Maximum-likelihood training is the dominant strategy described for recurrent networks.
- Motivation: Small prediction errors can compound during free-running generation, causing the conditioning context to diverge from sequences seen during training.At test time, ground-truth values are unavailable, so each output is sampled from the previous generated context.
- Proposed idea: Professor Forcing explicitly matches generative and teacher-forced behavior to support robust generation beyond the training-sequence length and better model long-term dependencies.Its objective is not focused solely on one-step-ahead prediction.
- Contributions: Professor Forcing is introduced as a method for improving long-term sequence sampling, with sample quality evaluated by human assessors.The method is presented as a novel training approach for generative RNNs.
- Results: Professor Forcing acts as a regularizer, improving test likelihood on character-level Penn Treebank, Sequential MNIST Generation, and speech synthesis.The authors also report improved training performance and conjecture that longer-term dependencies may be more easily captured.
- Results: Hidden-state dynamics diverge between sampling and teacher forcing, and T-SNE analysis shows Professor Forcing can mitigate this divergence.The paper also considers settings where training sequences are shorter than the sequences generated at test time.
2 Proposed Approach: Professor Forcing
Professor Forcing trains a generative RNN and a discriminator so that teacher-forced and free-running behaviors become indistinguishable. The generator combines data likelihood with adversarial objectives, while the discriminator distinguishes behavior sequences from the two modes.
- Core approach: Professor Forcing uses adversarial training to make outputs and hidden-state dynamics similar in teacher-forcing and free-running modes.The comparison concerns distributions of behavior sequences rather than isolated outputs.
- Discriminator: A discriminator processes variable-length behavior sequences and uses evidence from both past and future time steps through a bidirectional RNN in the experiments.The behavior sequence is derived from the generative RNN’s hidden states and outputs.
- Discriminator: The discriminator is trained to classify behavior sequences produced under teacher forcing versus free-running generation.Training minibatches combine N sequences from each mode, with free-running outputs sampled from the generator.
- Generator objective: The generator is trained both to maximize data likelihood and to fool the discriminator.The likelihood term is the usual teacher-forced training criterion, while adversarial terms adjust behavior across modes.
- Generator objective: One adversarial variant changes only free-running behavior toward teacher-forced behavior, while an optional variant also makes teacher-forced behavior indistinguishable from free-running behavior.These alternatives correspond to different adversarial objectives for the generator.
- Optimization: Experiments update the generator with NLL + Cf or NLL + Cf + Ct, while discriminator parameters are updated using Cd.The discriminator’s optimization problem changes as the generator changes, as in other GAN setups.
3 Related Work
The paper relates Professor Forcing to GANs, adversarial domain adaptation, scheduled sampling, actor-critic methods, and behavior matching in Boltzmann machines. Its distinguishing focus is matching recurrent hidden-state behavior across training and sampling modes.
- Adversarial methods: Professor Forcing resembles GANs because a discriminator supplies gradients to train a generative model, but its classifier compares hidden states from sampling and teacher-forcing modes.GAN discriminators instead distinguish real samples from generated samples.
- Adversarial methods: Adversarial domain adaptation also discriminates hidden states from source and target domains, but had not been applied to improving long-term generation from recurrent networks.Professor Forcing transfers this comparison into generative recurrent modeling.
- Non-adversarial methods: Scheduled sampling mixes ground-truth and self-generated inputs, increasing the probability of self-generated inputs during training.Its relation to Professor Forcing is through exposure to sampling-mode inputs rather than adversarial behavior matching.
- Non-adversarial methods: Actor-critic methods are another approach explored for modeling long-term dependencies in generative recurrent neural networks.The passage identifies this as related work without detailing its mechanism here.
- Behavior matching: Matching free-running and observed-data behavior is also connected to Boltzmann-machine training, where sufficient statistics are matched across modes.The paper places Professor Forcing within a broader family of behavior-matching ideas.
4 Experiments
Experiments evaluate Professor Forcing across recurrent-network architectures and sequence-generation tasks, including language modeling, sequential MNIST, handwriting, and raw-waveform synthesis. Results include faster training-cost reduction, improved hidden-state alignment, long-horizon handwriting generation, and human-rated sample-quality gains, while some short-sequence and word-level settings show no difference.
- Networks Architecture and Professor Forcing Setup: The generative RNN uses a GRU, while Professor Forcing compares teacher-forced and free-running dynamics through a discriminator operating on hidden states and outputs.The discriminator is bidirectional and recurrent; generator and discriminator training use Adam.
- Character-Level Language Modeling: Professor Forcing training cost decreases faster than Teacher Forcing on character-level Penn Treebank, although training takes 3 times more time.The model uses a one-hidden-layer GRU with 1024 hidden units and reports character-level evaluation using BPC.
- Sequential MNIST: At t = 500, teacher-forced and free-running hidden states occupy distinct regions, whereas Professor Forcing makes the regions largely overlap.Across 30 T-SNEs, mean centroid distance falls from 3000.0 to 1800.0, a 40% relative reduction.
- Handwriting Generation: Handwriting models trained on 50-step sequences are used for conditional generation over 1000 steps, with human evaluation comparing Professor Forcing and Teacher Forcing samples.The evaluation used randomly paired samples, equal training time, and volunteers unaware of model identity.
- Music Synthesis on Raw Waveforms: Vocal-synthesis evaluation considers negative log-likelihood and sample quality, including human assessment, while visual sample inspection can be flawed because models may memorize training examples.The authors argue this concern is less likely here because validation likelihood also improved.
- Negative Results on Shorter Sequences: No difference appears between Teacher Forcing and Professor Forcing on word-level Penn Treebank or speech-synthesis training with sequences shorter than 100.The authors suggest character-level language modeling may depend more on long-term dependencies.
5 Conclusion
Professor Forcing matches recurrent-network behavior during teacher-forced training and free-running generation, targeting robust generation beyond the training sequence length. The approach also improves one-step prediction despite adding a potentially conflicting objective term.
- Professor Forcing uses a discriminator to match recurrent-network behavior under data-constrained and free-running modes.The discriminator compares behavioral statistics rather than only single-step predictions.
- The method targets generalization when generated sequences are much longer than the training sequences.This setting arises when training sequences are shorter because of memory constraints.
- Professor Forcing also improves one-step prediction, suggesting a regularizing effect despite its potentially conflicting training objective.