Source-linked AI summary
Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models
Gabriel Lima Guimaraes, Benjamin Sanchez-Lengeling, Carlos Outeiral, Pedro Luis Cunha Farias, Alán Aspuru-Guzik
TL;DR
Sequence generators often need guidance toward desirable metrics without losing similarity to the training distribution. ORGAN combines reinforcement learning with GAN-based rewards to control sequence properties, and experiments report tuned quality and structure across molecular and music generation. The paper concludes that desired metrics and diversity can improve while adversarial training retains non-repetitious samples.
Problem
Sequence generation needs to target desirable characteristics while keeping generated samples similar to the initial data distribution.
Method
ORGAN combines SeqGAN-style adversarial training with reinforcement learning, weighting domain-specific objective rewards and discriminator rewards.
Results
ORGAN tunes the quality and structure of generated samples and improves desired metrics over recurrent networks trained with MLE or SeqGAN.
Takeaways & Limitations
Domain-specific reward functions can guide sequence generation while adversarial training maintains non-repetitious samples, often requiring only a simple objective hint.
Takeaways & Limitations
Property optimization can encounter dataset- or property-dependent ceilings, with some generated druglikeness values remaining below the maximum of 1.
Abstract
from arXiv · showhide
In unsupervised data generation tasks, besides the generation of a sample based on previous observations, one would often like to give hints to the model in order to bias the generation towards desirable metrics. We propose a method that combines Generative Adversarial Networks (GANs) and reinforcement learning (RL) in order to accomplish exactly that. While RL biases the data generation process towards arbitrary metrics, the GAN component of the reward function ensures that the model still remembers information learned from data. We build upon previous results that incorporated GANs and RL in order to generate sequence data and test this model in several settings for the generation of molecules encoded as text sequences (SMILES) and in the context of music generation, showing for each case that we can effectively bias the generation process towards desired metrics.
1 Introduction
ORGAN combines reinforcement learning with adversarial training to steer sequence generation toward domain-specific objectives while preserving resemblance to the data distribution. It is evaluated on molecular and music generation, where results show tunable sample quality and structure.
- 1 Introduction: Sequence generation with RNNs and maximum likelihood works well but can suffer from exposure bias and miss multiscale structures or salient features.
- 1 Introduction: ORGAN weights domain-specific objective rewards with discriminator rewards to guide generation while retaining characteristics of the initial data distribution.The objective selects desired traits, while the discriminator keeps samples within the data distribution's boundaries.
- 1 Introduction: The method extends SeqGAN by adding domain-specific rewards and tests Wasserstein-GANs to improve adversarial-training stability.
- 1 Introduction: ORGAN is tested on molecular and music generation while optimizing several domain-specific metrics.
- 1 Introduction: The results show that ORGAN tunes the quality and structure of generated samples compared with MLE, SeqGAN, and a reinforcement-learning approach.
2 Related work
Related work combines recurrent, variational, adversarial, and reinforcement-learning approaches for sequence and molecular generation. Prior methods often use task-specific mechanisms to optimize desired properties or address GAN training challenges.
- 2 Related work: Prior reinforcement-learning approaches for sequence generation introduced penalties to discourage unrealistic sequences and avoid local maxima far from the global reward maximum.
- 2 Related work: Molecular-generation research includes recurrent neural networks, variational autoencoders with surrogate optimization, GANs, and earlier applications of this approach to molecular design.
- 2 Related work: Related adversarial-training work studies actor-critic connections, Wasserstein distance, diversity promotion, mode-collapse avoidance, and GAN convergence.
3 Background
SeqGAN frames discrete sequence generation as an adversarial reinforcement-learning problem: a generator produces tokens, a discriminator classifies sequences, and policy learning uses sequence rewards. Monte Carlo rollouts estimate rewards for incomplete sequences.
- 3 Background: SeqGAN uses a generator to produce sequences and a discriminator to distinguish generated sequences from real data in an alternating minimax game.
- 3 Background: Because discrete sampling is nondifferentiable, the generator is trained as a reinforcement-learning agent using REINFORCE.
- 3 Background: At each incomplete sequence state, the generator selects the next token as an action, following a stochastic policy whose expected long-term reward is maximized.
- 3 Background: The action-value function Q estimates expected reward after taking an action and completing the sequence with the current policy.For a complete sequence, Q equals the sequence reward; for partial sequences, Monte Carlo completion estimates future reward.
- 3 Background: SeqGAN's reward function is supplied by the discriminator.
4 ORGAN
ORGAN combines adversarial training with reinforcement learning to optimize domain-specific sequence objectives while preserving data-distribution likeness. Its design mixes discriminator and objective rewards, penalizes repeated sequences, and optionally uses Wasserstein training for stability.
- Implementation: The framework uses Monte Carlo sampling to pass rewards back to the policy and trains the discriminator on mixed real and generated data.The implementation uses Adam optimization, dropout, L2 regularization, and RDKit or MIDI-based metrics across experiments.
- Reward design: ORGAN extends adversarial sequence generation by rewarding the generator with both discriminator feedback and domain-specific objectives.The reward is a weighted combination of Dφ and Oi, controlled by λ.
- Reward design: λ = 0 yields naive RL, whereas λ = 1 yields SeqGAN.The objective can also vary across adversarial-training iterations, alternating rewards among objectives and the discriminator.
- Diversity: Repeated sequences receive diminished rewards by dividing their reward by the number of copies, helping prevent mode collapse.Domain-specific similarity metrics are presented as an alternative penalty.
- Wasserstein variant: Wasserstein-1 training is implemented to improve learning stability and avoid GAN convergence problems such as a perfect discriminator.Under W, D becomes a K-Lipschitz critic used to compute the Wasserstein distance rather than classify samples.
- Implementation: The generator is an LSTM-based RNN, while the discriminator is a CNN designed for text classification.Molecule experiments use drug-like and nondrug-like ZINC molecules; music experiments distinguish folk from videogame tunes.
5 Experimental results
ORGAN is evaluated on molecule and music sequence generation, with experiments testing domain-specific objectives, diversity, validity, and training behavior. The results show that objective reinforcement can bias generated samples while GAN-based training preserves diversity and exposes capacity and optimization trade-offs.
- Experimental setup: ORGAN is tested on molecule and musical-melody generation, optimizing domain-specific metrics while promoting diversity.Objective values are mapped to [0, 1], and models use MLE pre-training before further training with ORGAN or baseline methods.
- 5.1 Experiment: Molecules: Molecules are represented as fixed-length integer sequences derived from character vocabularies, with SMILES validity and duplicate generation explicitly evaluated.SMILES grammar permits invalid expressions, so invalid and duplicate molecules are penalized during evaluation.
- 5.1 Experiment: Molecules: ORGAN outperforms SeqGAN and MLE across the three optimized molecular metrics, while its Wasserstein variant provides better diversity properties.The molecular evaluation reports mean values over valid generated molecules and compares optimization scenarios against the MLE baseline.
- 5.2 Experiment: Musical melodies: Naive RL achieves a higher Ratio of Steps score than ORGAN but underperforms in diversity, producing simpler and less interesting musical sequences.The paper attributes the difference to GAN/WGAN training enforcing greater diversity than naive RL.
- 5.2 Experiment: Musical melodies: The Ratio of Steps and Tonality objectives are inversely related, while lower Wasserstein-metric values are explained by slower training.Consecutive notes counted as steps do not have the perfect-fifth frequency ratio associated with increased tonality.
6 Conclusions and future work
ORGAN combines adversarial training with reinforcement learning to optimize sequence properties while preserving diversity and data-likeness. Experiments show improved desired metrics, while future work targets heuristic choice, dataset boundaries, and non-sequential data.
- Conclusions: ORGAN extends SeqGAN with reinforcement learning to optimize arbitrary objectives in sequence generation.The framework controls generated-sample properties through domain-specific rewards alongside adversarial training.
- Conclusions: ORGAN outperforms recurrent networks trained with MLE or SeqGAN on desired metrics while maintaining non-repetitious samples.The paper frames this as combining domain-specific reward functions with adversarial generation.
- Conclusions: The RL component is identified as a major driver of property optimization and diversity, with higher values when RL is present.This conclusion is based on the reported experiments.
- Future work: The choice of heuristic remains an open performance factor, and alternative GAN formulations for discrete sequences could also receive RL extensions.These are proposed directions for future investigation.
- Future work: Future work should push dataset property boundaries because outliers may be more valuable in drug and materials discovery.The paper identifies this as an area for improvement, especially in domains where unusual properties matter.
- Future work: Extending ORGAN to images or audio requires casting GAN training as reinforcement learning to accommodate arbitrary, potentially nondifferentiable objectives.The authors describe this extension as promising because real-valued GANs are better understood than sequence GANs.