Source-linked AI summary
GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
Matt J. Kusner, José Miguel Hernández-Lobato
TL;DR
GANs struggle with discrete sequences because multinomial samples are not differentiable with respect to distribution parameters. The paper addresses this by using recurrent GANs with Gumbel-softmax output distributions and evaluates them on discrete sequence generation, finding proof-of-concept promise in a context-free grammar task.
Problem
GAN training relies on gradients through generated samples, but sampling discrete sequences from multinomial distributions is not differentiable.
Method
The paper trains LSTM-based GANs for discrete sequences using Gumbel-softmax outputs, with temperature annealing during training.
Results
The GAN models learn to generate alternating x sequences resembling the context-free grammar training data, with samples close to the training data.
Takeaways & Limitations
The proof-of-concept results show strong promise for training GANs to generate discrete sequence data.
Abstract
from arXiv · showhide
Generative Adversarial Networks (GAN) have limitations when the goal is to generate sequences of discrete elements. The reason for this is that samples from a distribution on discrete objects such as the multinomial are not differentiable with respect to the distribution parameters. This problem can be avoided by using the Gumbel-softmax distribution, which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function. In this work, we evaluate the performance of GANs based on recurrent neural networks with Gumbel-softmax output distributions in the task of generating sequences of discrete elements.
1 Introduction
GANs generate synthetic data by training a discriminator to distinguish real from synthetic samples while training a generator to confuse it. This framework is difficult to apply to discrete sequences because multinomial sampling is nondifferentiable; the paper evaluates Gumbel-softmax as a differentiable approximation for this setting.
- GANs jointly train a discriminator to distinguish synthetic from real data and a generator to produce samples that confuse the discriminator.
- Backpropagation works for continuous generated data but yields zero gradients through samples of discrete items such as text or SMILES sequences.
- Gumbel-softmax provides a differentiable approximation to multinomial sampling for one-hot encoded discrete data.
- The paper proposes using Gumbel-softmax to train GANs on sequences of discrete tokens and evaluates performance in that setting.
- A related approach treats discrete-sequence generation as a stochastic policy in reinforcement learning and updates the policy gradient directly.
2 Gumbel-softmax distribution
The Gumbel-softmax distribution replaces nondifferentiable multinomial sampling with a smooth, differentiable approximation controlled by an inverse temperature. As temperature decreases toward zero, its samples recover the multinomial distribution, supporting annealed GAN training.
- The softmax maps a continuous vector h to probabilities p for a multinomial distribution over one-hot encoded d-dimensional vectors y.The probabilities satisfy p_i = p(y_i = 1) for i = 1, . . . , d.
- Multinomial sampling can be represented by adding independent zero-location, unit-scale Gumbel variables to h and applying one-hot arg max.
- The one-hot arg max sample has zero gradient with respect to h, so the paper replaces it with a differentiable softmax transformation.
- For finite positive τ, Gumbel-softmax samples are smooth and differentiable with respect to h; as τ → 0 they match the distribution from multinomial sampling.As τ → ∞, samples become the uniform probability vector.
- A GAN can use Gumbel-softmax outputs by starting with a relatively large τ and annealing it toward zero during training.
3 A recurrent neural network for discrete sequences
The paper constructs a recurrent GAN for discrete sequences using LSTM generator and discriminator networks. The generator samples sequences recurrently from initial noise, while Gumbel-softmax makes the generator pathway differentiable for adversarial optimization.
- An example: The proposed generator is an LSTM that produces a hidden vector at each character step and uses softmax probabilities over the character vocabulary.The example vocabulary contains x, +, −, /, and ∗.
- An example: Generation begins from a sample pair replacing the initial cell and hidden states, then feeds each prediction into the next LSTM unit.
- An example: The training objective is to design a generator that produces real-looking discrete sequences rather than train only by maximum likelihood estimation.
- Generative adversarial modeling: Adversarial training alternates discriminator and generator updates using real data and noise-driven generated samples.
- Using the Gumbel-softmax distribution: The generator and discriminator are both LSTMs, and Gumbel-softmax enables gradient-based optimization of their parameters despite discrete sequence sampling.
- Using the Gumbel-softmax distribution: The adversarial procedure generates a full sequence, sends it to an LSTM discriminator alongside real sequences, and updates weights to improve discrimination and generator fooling.
4 Experiments
The experiments evaluate GANs for discrete sequences on a context-free grammar, varying sample size, temperature handling, and hidden-state noise. Generated samples learn alternating x sequences and can closely resemble training data, supporting the approach as a proof of concept.
- The experiment models a context-free grammar using 5,000 training samples of at most 12 characters, padding shorter sequences with spaces.
- Figure 3 tracks generator and discriminator losses for the default and alternative training settings.The intended pattern is increasing discriminator loss and decreasing generator loss as generated data better mimics real data.
- The study compares default Gumbel-softmax temperature annealing with larger generated samples, input-temperature variation, and hidden-state noise.The four settings correspond to plots (a)–(d) in Figure 3.
- Figure 4 shows GAN models learning alternating x sequences, with some samples closely matching the training data across the tested settings.In plot (a), the 4th, 10th, and 17th rows are specifically identified as very close to training data.
- The results are presented as a proof of concept showing strong promise for training GANs to generate discrete sequence data.The authors suggest that variational divergence minimization and density-ratio estimation could improve future results.