PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Experience Replay

Cong Lu, Philip J. Ball, Yee Whye Teh, Jack Parker-Holder

arXiv:2303.06614v4cs.LGcs.AIstat.ML

TL;DR

RL agents often have limited experience to replay, constraining the data available for training. SYNTHER uses diffusion models to generate additional experience for offline and online RL. The paper reports benefits across offline and online settings, including parity with real data on standard offline datasets and higher online update-to-data ratios.

  • Problem

    RL agents must collect their own experience, which is often limited, and suitable prior data may not exist for particular behaviors.

    Source: §1 · p. 2

  • Method

    SYNTHER uses diffusion models to generate synthetic experience that can replace offline data or augment online replay.

    Source: §1 · p. 2

  • Results

    SYNTHER reports parity with real data on standard D4RL and V-D4RL offline datasets and higher online update-to-data ratios with increased sample efficiency.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Synthetic experience can support larger policy and value networks offline and higher update-to-data ratios online without algorithmic changes.

    Source: §1 · p. 2, §6 · p. 10

    Pixel-based data generation is demonstrated in latent space; generating in original image space and extending the approach to online settings are left for future work.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay, whereby a dataset of past experiences is used to train a policy or value function. However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SynthER), a diffusion-based approach to flexibly upsample an agent's collected experience. We show that SynthER is an effective method for training RL agents across offline and online settings, in both proprioceptive and pixel-based environments. In offline settings, we observe drastic improvements when upsampling small offline datasets and see that additional synthetic data also allows us to effectively train larger networks. Furthermore, SynthER enables online agents to train with a much higher update-to-data ratio than before, leading to a significant increase in sample efficiency, without any algorithmic changes. We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data. Finally, we open-source our code at https://github.com/conglu1997/SynthER.

1 Introduction

RL replay is constrained by agents’ limited collected experience, while suitable prior data may not exist for particular behaviors. SYNTHER uses diffusion models to upsample experience, with reported improvements across offline and online settings.

1 Introduction

  • Replay lets RL agents train on collected experience, but that experience is often limited unless agents use many costly workers and fast simulation.Offline data can also be mismatched with the online environment, and prior successes often rely on simple, low-dimensional proprioceptive domains.Source: §1 · p. 2
  • Because suitable prior data may not exist for particular behaviors, SYNTHER uses diffusion generative models to create synthetic experience rather than only reusing existing data.Earlier work considered upsampling online RL data with VAEs or GANs.Source: §1 · p. 2
  • SYNTHER reports improvements in offline and online RL, including parity with real data on standard D4RL and V-D4RL datasets and higher online update-to-data ratios without algorithmic changes.The paper also reports benefits from upsampling small offline datasets and training larger networks with synthetic data.Source: §1 · p. 2
  • SYNTHER is presented as a way to arbitrarily upsample a limited dataset for replay-based agents, with evaluations spanning proprioceptive and pixel-based environments.Figure 3 visualizes generated walker2d transitions and a t-SNE projection of 100,000 samples, described as cohesive, plausible, and diverse.Source: §1 · p. 2, Figure 2 · p. 2, Figure 3 · p. 3

2 Background

The background defines the RL setting and offline datasets, then introduces diffusion models as a way to learn and reverse a noising process.

2.1 Reinforcement Learning

  • RL is framed as optimizing a policy to maximize expected discounted return in a Markov Decision Process defined by states, actions, transitions, rewards, initial-state distribution, and discount factor.The MDP is represented as M = (S, A, P, R, ρ0, γ).Source: §2.1 · p. 3

2.2 Offline Reinforcement Learning

  • Offline RL uses a static dataset of transitions collected by one or more behavioral policies, whose distribution is called the behavioral distribution.Transitions can include a terminal flag when episodes end early or have finite horizons.Source: §2.2 · p. 3

2.3 Diffusion Models

  • Diffusion models learn to reverse a sequence of Gaussian noising steps, starting from noise when the maximum noise level greatly exceeds the data standard deviation.The paper describes a probability-flow ODE whose score term points toward data at a given noise level; its diffusion models approximate online or offline behavioral distributions.Source: §2.3 · p. 3

3 Synthetic Experience Replay

SYNTHER uses diffusion to generate synthetic replay data, with offline modeling of the behavioral distribution and an online variant that continually trains on new experience. Its offline synthetic data reproduces original-data performance across proprioceptive D4RL datasets and outperforms prior generators on downstream performance in the reported comparison.

3.1 Offline SYNTHER

  • Offline SYNTHER models the offline behavioral distribution, using a residual MLP denoiser and a Karras sampler with 128 diffusion steps.The same model can generate pixel-environment data in a low-dimensional latent space.Source: §3.1 · p. 4
  • On halfcheetah medium-replay, SYNTHER better captures dataset statistics and achieves higher downstream performance than prior generative models.The comparison uses 100K samples per model for statistics and 5M samples per model for offline RL performance.Source: Table 1 · p. 4, §3.1 · p. 4
  • Across proprioceptive D4RL datasets, SYNTHER-generated data faithfully reproduces original performance, allowing training without the original data.The table reports mean and standard deviation over 8 seeds; highlighted results indicate at least parity within each algorithm and environment group.Source: Table 2 · p. 5

3.2 Online SYNTHER

  • For online replay-based agents, SYNTHER periodically trains the diffusion model on real transitions and adds generated transitions to a synthetic buffer for mixed sampling.The reported setup uses a real/synthetic sampling ratio of 0.5; the synthetic buffer can have finite capacity to limit stale data.Source: §3.2 · pp. 4–5

4 Empirical Evaluation

Across offline, online, and pixel-based evaluations, SYNTHER improves or matches established approaches while extending training with synthetic experience. Results include gains on reduced offline datasets, larger networks, and online agents trained at higher update-to-data ratios.

4.1 Offline Evaluation

  • Offline, SYNTHER achieves at least parity across environment and algorithm groups, with significant maze2d improvements and dataset compression up to 12.9×.The authors attribute locomotion parity partly to D4RL datasets already being at least 1M samples.Source: §4.1 · p. 5

4.1.1 Upsampling for Small Datasets

  • SYNTHER significantly improves walker2d performance over explicit augmentation, reaching near-original medium-expert performance from 3% of the data.The comparison uses Gaussian noise augmentation; reduced datasets contain approximately 20K–200K transitions and are upsampled to 5M.Source: §4.1.1 · p. 6, Figure 4 · p. 6
  • On the reduced 15% walker2d medium-replay dataset, SYNTHER generates more distant samples with lower dynamics error than explicit augmentation.The analysis compares distance from training data with dynamics accuracy; samples far from the data still have lower dynamics MSE.Source: §4.1.1 · pp. 6–7, Figure 5 · p. 6

4.1.2 Scaling Network Size

  • 11.7% overall improvement on locomotion datasets comes from pairing SYNTHER with larger TD3+BC networks; larger networks with original data do not improve performance.For IQL and EDAC, increasing network size did not improve performance.Source: §4.1.2 · p. 7, Table 3 · p. 7

4.2 Online Evaluation

  • SAC (SYNTHER) matches or outperforms REDQ on most online environments while training at UTD=20, compared with regular SAC’s UTD=1.The comparison spans three DeepMind Control Suite and three OpenAI Gym environments; the caption reports mean and standard deviation over six seeds.Source: §4.2 · pp. 7–8, Figure 6 · p. 8

4.3 Scaling to Pixel-Based Observations

  • +9.5% on DrQ+BC and +6.8% on BC are reported for pixel-based offline tasks using SYNTHER-generated latent transitions.The method uses a frozen CNN encoder pretrained on the same dataset; generating in original image space and extending to online pixel settings are left for future work.Source: §4.3 · p. 8

5 Related Work

Prior generative-data approaches in RL include VAEs and GANs, while diffusion work often models trajectories or actions. SYNTHER instead synthesizes transitions for use with existing RL algorithms, with a distinct relationship to forward-dynamics model-based methods.

Generative Training Data

  • Earlier VAE and GAN approaches for online RL did not match original performance on simple environments such as CartPole.Other GAN-generated samples were used to pre-train policies, with a reported modest sample-efficiency improvement on CartPole.Source: §5 · pp. 8–9

Generative Modeling in RL

  • Diffuser and Decision Diffuser generate sequences or episodes with reward or goal guidance, whereas SYNTHER operates at the transition level.SYNTHER is intended to remain compatible with existing reinforcement-learning algorithms.Source: §5 · p. 9

Model-Based Reinforcement Learning

  • Unlike forward-dynamics rollouts, SYNTHER generates experiences without starting from a real state and avoids compounding errors from model inaccuracy.The authors describe it as orthogonal to forward-dynamics methods and suggest combining them by generating initial states with diffusion.Source: §5 · p. 9

6 Conclusion

SYNTHER applies diffusion-based experience upsampling across offline and online RL, proprioceptive and pixel-based environments, and six algorithms without algorithmic modification. The reported benefits include training from very small offline datasets and improved online sample efficiency.

6 Conclusion

  • SYNTHER was integrated into six distinct algorithms across proprioceptive and pixel-based settings without algorithmic modification.Each algorithm was fine-tuned for its own use case.Source: §6 · p. 10
  • In offline RL, SYNTHER supports training from extremely small datasets, larger policy and value networks, and high levels of data compression.The conclusion presents these as capabilities enabled by synthetic data.Source: §6 · p. 10
  • In online RL, additional synthetic data allows higher update-to-data ratios, leading to increased sample efficiency.Source: §6 · p. 10
  • The authors identify n-step replay, image-space transition generation, pretrained foundation models, and guided diffusion as directions for extending SYNTHER.These are proposed future possibilities, not evaluated results in the conclusion passage.Source: §6 · p. 10

Supplementary Material

SYNTHER models full environment transitions and normalizes continuous dimensions for diffusion modeling. Its generated samples match high-level dataset statistics, while binary terminals require separate handling.

A Data Modeling

  • The diffusion model generates full environment transitions, concatenating states, actions, rewards, next states, and terminals when present.Each continuous non-terminal dimension is normalized to zero mean and unit standard deviation.Source: A Data Modeling · p. 17
  • Because terminals are binary and can be rare, the method leaves them unnormalized and thresholds generated values at 0.5.Walker2d-expert has approximately 0.0001% terminals; categorical diffusion is noted as a potentially cleaner treatment.Source: A Data Modeling · p. 17
  • SYNTHER's samples reproduce the high-level marginal statistics of the HalfCheetah medium-replay dataset.Figure 8 compares synthetic samples with original data across state, action, and reward dimensions.Source: Figure 8 · p. 17, A Data Modeling · p. 17

A.1 Data Compression

The supplementary material describes SYNTHER's compression trade-off, baseline hyperparameter searches, denoiser configuration, sampling cost, and ablation plan. It reports compact model-based storage at the cost of sampling speed.

A.1 Data Compression

  • SYNTHER amortizes dataset storage in the denoising network's parameters, achieving high compression at the cost of sampling speed.Generating 100K transitions takes around 90 seconds.Source: §A.1 · p. 18, §B.3 · p. 19

B.1 TVAE and CTGAN

  • The TVAE and CTGAN baselines were evaluated with hyperparameter searches, using 30 trials and the default settings.Tables 6 and 7 identify the respective search spaces and highlight the default choices.Source: §B.1 · p. 18, Table 6 · p. 18, Table 7 · p. 18

B.2 Denoising Network

  • The paper uses the Elucidated Diffusion Model (EDM, Karras et al. [38]); its denoising network is an MLP with previous-layer skip connections, depth 6, and approximately 6M parameters.For halfcheetah medium-expert, hopper medium, and hopper medium-expert, increasing width up to 2048 yields more performant samples and raises parameters to ≈25M.Source: §B.2 · pp. 18–19

B.3 Elucidated Diffusion Model

  • The diffusion model takes approximately 17 minutes to train for 100K steps and 90 seconds to generate 100K samples.Sampling uses 128 diffusion timesteps.Source: §B.3 · p. 19, Table 9 · p. 19

C SYNTHER Ablations

C.1 Size of Upsampled Dataset

Offline upsampling gains eventually saturate around 5M samples, which the authors use as a heuristic across datasets. The network ablation reports benefits from greater depth and residual connections.

C.1 Size of Upsampled Dataset

  • Performance gains with synthetic data eventually saturate around 5M samples, a heuristic used to upsample offline datasets originally sized 100K–2M.The walker medium-replay ablation tested 10 log-uniform sample levels from 50K to 5M.Source: §C.1 · p. 19, Figure 9 · p. 20

C.2 Network Ablations

  • On hopper medium-expert, greater denoiser depth and residual connections improve downstream offline RL performance.The table reports final performance averaged over four seeds, with mean and standard deviation.Source: Table 10 · p. 20, §C.2 · p. 20

D RL Implementation

The appendix records the offline and online implementations, defines the explicit augmentation baselines, and reports the runtime breakdown for online SynthER. It also notes that further offline experiments are included.

D RL Implementation

  • The offline evaluation uses CORL implementations, while online SAC uses the REDQ implementation and the dmcgym wrapper for DeepMind Control Suite.The cited codebases are released under Apache and MIT licenses, respectively.Source: D RL Implementation · p. 20

D.1 Data Augmentation Hyperparameters

  • The explicit baselines perturb states with additive Gaussian noise, multiplicative uniform noise, or noise applied to the next-state delta.The stated distributions are N(0, 0.1), Unif([0.8, 1.2]), and Unif([0.5, 1.5]), respectively.Source: §D.1 · pp. 20–21

D.2 Online Running Times

  • SYNTHER's online SAC run took ≈21.1 hours versus ≈22.7 hours for REDQ on a V100 GPU, with RL training the largest component.Diffusion training took 4.3 hours, sampling 5 hours, and RL training 11.8 hours at UTD=20; default SAC at UTD=1 took ≈2 hours.Source: §D.2 · p. 21

E Further Offline Results

E.1 AntMaze Data Generation

The appendix tests SynthER on AntMaze, mixes synthetic and real offline data, and describes latent-space generation for pixel-based V-D4RL tasks. These experiments report parity or improvement in the stated settings and detail how pixel transitions are compressed and used for retraining.

E.1 AntMaze Data Generation

  • SYNTHER improves TD3+BC on AntMaze umaze and achieves at least parity otherwise.The evaluation uses AntMaze-v2 and reports final performance averaged over six seeds.Source: Table 11 · p. 21, §E.1 · p. 21

E.2 Offline Data Mixtures

  • The 50-50 real-and-synthetic offline data mix achieves parity with original data, as does synthetic data alone.The mix adds as many synthetic samples as real samples; the authors do not expect gains because most D4RL datasets are at least 1M transitions.Source: Table 12 · p. 21, §E.2 · p. 21

F Latent Data Generation with V-D4RL

  • For pixel-based V-D4RL, SYNTHER generates low-dimensional latent transitions using separate supervised actor and critic features.The cheetah-run latent has dimension 207; training first uses original data, then retrains only the fully connected actor and critic portions with synthetic data.Source: F Latent Data Generation with V-D4RL · p. 22

F.1 Ablations On Representation

The ablation compares separate supervised actor and critic features with using actor-only or critic-only features for both networks. Separate features perform best in the cheetah expert setting.

F.1 Ablations On Representation

  • The compared alternatives use actor-only or critic-only features for both the actor and critic networks.Source: §F.1 · p. 22
  • The authors suggest that non-specific image compression, such as auto-encoders, could be less suitable for this task.Source: §F.1 · p. 22
  • Separate supervised actor and critic features are essential for downstream performance, with a particularly large drop when both networks use critic features alone.The ablation uses the V-D4RL cheetah expert dataset and averages final performance over four seeds.Source: Table 13 · p. 22, §F.1 · p. 22
Loading 2303.06614v4…