Source-linked AI summary
Repetition as Reinforcement: Enhancing Sample Efficiency via Instant Episode Repetition in Reinforcement Learning
Hoda Yamani, Yuning Xing, Koen van Rijnsoever, Bruce A. MacDonald, Henry Williams
TL;DR
Reinforcement learning often requires many environment interactions, while existing replay-based methods reuse experience passively during updates rather than repeating successful behaviors during interaction. The paper introduces Instant Episode Repetition, which immediately re-executes high-reward action sequences, and reports consistent improvements in data efficiency and training stability across simulated and real-world robotic tasks.
Problem
Reinforcement learning remains limited by poor sample efficiency, while replay-based methods do not directly repeat successful behaviors during environment interaction.
Method
Instant Episode Repetition immediately re-executes high-reward action sequences at episode boundaries, integrating with off-policy reinforcement learning without architectural modifications.
Results
IER consistently improves data efficiency and training stability across simulated benchmarks and real-world robotic tasks, outperforming respective baselines across most evaluated tasks.
Takeaways & Limitations
Immediate repetition of successful episodes is a simple extension that can improve learning efficiency across off-policy reinforcement-learning settings.
Takeaways & Limitations
Baseline performance can vary with simulator stochasticity and random seeds, although relative differences between methods remain consistent across seeds.
Abstract
from arXiv · showhide
Repetition is a fundamental mechanism in human learning, where revisiting successful experiences strengthens memory, consolidates skills, and improves future performance. Motivated by this biological principle, we introduce Instant Episode Repetition (IER), a simple and novel mechanism that improves sample efficiency by immediately repeating action sequences from successful episodes during environment interaction. Unlike conventional approaches such as Experience Replay and Self-Imitation Learning (SIL), which passively reuse past experience during training updates, IER directly influences the data collection process. Upon identifying a high-reward episode, the agent repeats its action sequence for a fixed number of subsequent episodes, reinforcing valuable behaviors through renewed interaction with the environment. We integrate IER into state-of-the-art SAC and TD3 algorithms and evaluate its effectiveness on continuous-control benchmarks, including MuJoCo, the DeepMind Control Suite, and a real-world dynamic object translation task with a robotic manipulator. Experimental results demonstrate that this simple mechanism improves learning performance over standard and self-imitation-based baselines.
1 Introduction
The introduction identifies sample efficiency as a central limitation of reinforcement learning and motivates Instant Episode Repetition (IER) as a mechanism that immediately repeats newly discovered high-reward action sequences during data collection. IER is presented as an extension to off-policy methods and is evaluated across simulated and real-world continuous-control tasks.
- Motivation: Sample efficiency remains a key challenge limiting reinforcement learning’s broader applicability despite successes in robotics, autonomous driving, and game playing.
- Motivation: Reward-driven repetition motivates reinforcing successful action–outcome sequences to support efficient learning, procedural memory, and motor skills.
- Related work: Experience replay, PER, and SIL reuse valuable experiences passively during updates rather than directly altering the agent’s behavior during environment interaction.
- Method: IER improves sample efficiency by immediately repeating action sequences from newly discovered high-reward episodes, thereby modifying the data-collection process.
- Evaluation: IER is integrated into standard off-policy frameworks and evaluated on continuous-control benchmarks spanning simulated and real-world robotic environments, where it outperforms baseline algorithms.
2 Background and Related Work
Biological learning links reinforcement and repetition to the consolidation and efficient execution of successful behaviors while preserving adaptability. In reinforcement learning, IER translates these principles into interaction-level repetition, addressing the passivity of prior self-imitation methods by re-executing successful trajectories.
- Biological Foundations: Reinforcement increases repetition of rewarding actions, while repeated activation stabilizes and refines those behaviors through practice.These mechanisms are associated with reward prediction signaling and structured neural-circuit activation.
- Biological Foundations: Repeated activation strengthens synaptic connections and supports memory consolidation, transforming transient successes into stable, more efficiently executed behaviors.Hebbian learning and long-term potentiation provide the cited neural mechanisms.
- Biological Foundations: Effective repetition varies execution conditions, promoting robustness and generalization while increasing processing efficiency for familiar patterns.This combines “repetition without repetition” with repetition suppression.
- Self-Imitation Learning: SIL improves policy learning from high-return experiences but remains passive because the current policy still samples actions and successful trajectories do not shape new experience directly.IER is motivated as a deliberate re-execution mechanism for high-reward episodes during interaction.
- Instant Episode Repetition: IER re-executes entire successful trajectories during interaction, reinforcing temporally coherent action sequences and enabling on-policy evaluation under dynamically evolving conditions.This moves beyond purely policy-driven sampling and is presented as a novel direction in self-imitation.
3 Methodology
IER modifies episode-level data collection in off-policy actor–critic RL by repeating high-reward action sequences while preserving the reward, learning, and optimization procedures. Repetition uses perturbed episode starts and subsequent state evolution to concentrate sampling near successful behaviors without duplicating identical transitions.
- IER integration: IER operates at episode boundaries, modifying interaction dynamics while preserving the underlying learning objective and optimization procedure.It changes how episodes are generated rather than the reward function, loss formulation, network architecture, or optimization procedure.
- Episode selection: IER uses the sum of episode rewards, Rep(τ), to determine whether an episode should be repeated.At episode termination, the agent computes Rep and updates the best observed value.
- Repetition and exploitation: When an episode achieves a new maximum reward, IER stores its full action sequence and executes it sequentially for RN consecutive episodes.RN controls repetition strength; after repetition ends, or when no new maximum is detected, the agent samples actions from its current policy.
- Replay and updates: Transitions collected during repetition and exploitation enter replay buffer M for standard off-policy updates, so IER changes only the visitation distribution of newly collected episodes.The underlying learning dynamics remain unchanged.
- Sampling behavior: Re-executing a stored action sequence under reset-state variation generates locally related episodes rather than identical trajectories.Variations in starting states and subsequent state evolution increase sampling density near empirically high-reward regions without duplicating identical transitions.
4 Experiments
Experiments evaluate IER with TD3 and SAC against standard and SIL-based baselines across eight simulated continuous-control tasks and a stochastic real-world dexterous manipulation task. The evaluation examines episodic repetition in diverse, high-dimensional environments spanning simulation and robotic control.
- Algorithms and baselines: IER is integrated into TD3 and SAC as IER-TD3 and IER-SAC, enabling direct comparison with the corresponding standard algorithms.The standard TD3 and SAC implementations use no episode repetition.
- Algorithms and baselines: SIL-TD3 and SIL-SAC provide self-imitation baselines using the standard SIL formulation to avoid confounding mechanisms such as demonstrations or intrinsic rewards.This comparison distinguishes passive experience reuse in SIL from active episodic repetition in IER.
- Simulated tasks: Eight continuous-control tasks from the DeepMind Control Suite and MuJoCo assess IER across diverse dynamics, task complexities, and high-dimensional state-action spaces.The two simulation platforms are used to provide a comprehensive evaluation of IER’s robustness.
- Real-world task: A stochastic dexterous in-hand manipulation task requires moving a cube sideways while maintaining a stable grasp under autonomous episode resets.The gripper must coordinate precise movements to prevent slipping or falling without external support.
- Real-world task: The real-world setup uses uniform LED illumination, ArUco-based camera tracking, and rewards for stable grasping and increased lateral displacement.The task incorporates contact dynamics, gravity, sensor noise, and constraints on data collection.
5 Experimental Results
The experiments evaluate IER-SAC and IER-TD3 across eight simulated continuous-control tasks and one real-world robotic task. They compare these methods with standard SAC and TD3 baselines and their respective self-imitation methods.
- IER-SAC and IER-TD3 are evaluated as the proposed repetition-based methods.
- The evaluation covers eight simulated continuous-control tasks and a real-world robotic task.
- The proposed variants are compared against standard SAC and TD3 baselines and their respective self-imitation methods.
SAC
IER-SAC improves learning over SAC and SIL-SAC across simulated and real-world tasks, with particularly strong gains in dynamic locomotion. Its effectiveness depends on moderate repetition, while excessive repetition can reduce state diversity and diminish returns.
- Simulated SAC: IER-SAC improves over SAC in six of eight tasks and performs comparably in the remaining two, with strongest gains in Ant-v4, HalfCheetah-v4, Walker-Walk, and Cheetah-Run.In these dynamic locomotion tasks, repetition reinforces stable behavioral sequences, producing faster convergence and improved long-term rewards.
- Simulated SAC: SIL-SAC is competitive in HalfCheetah-v4 and Walker-Walk but inconsistent in Humanoid-v4 and Cartpole-Swingup, sometimes falling below SAC by reinforcing suboptimal trajectories.This risk arises when exploration remains incomplete.
- Real-world evaluation: IER-SAC achieves faster convergence and higher final performance than SAC and SIL-SAC in the real-world gripper manipulation task.The results indicate that repetition remains effective under sensor noise, friction and contact inconsistencies, actuation delays, and other real-world stochasticity.
- Repetition sensitivity: 10.66% is the mean ∆AUC% peak at RN3 across all eight tasks jointly averaged over SAC and TD3, followed by RN2 and RN4 before declining at larger repetition levels.The unimodal pattern indicates that moderate repetition accelerates learning most effectively, whereas excessive repetition can reduce state diversity.
- Repetition sensitivity: Moderate repetition at RN2–RN4 often achieves competitive and near-peak performance, but the optimal RN varies across environments and should be treated as a tunable hyperparameter.Higher-instability or higher-reward-variance environments, including Finger-Turn-Hard and Walker-Walk, can benefit substantially from moderate or higher RN values.
6 Conclusion
The paper introduces IER, which directly reinforces action sequences from high-reward episodes through immediate re-execution during data collection. As a simple, general, biologically inspired extension requiring no architectural changes, IER improves data efficiency and training stability across simulated and real-world tasks.
- 6 Conclusion: IER reinforces high-reward behaviors by immediately re-executing their action sequences during data collection.This distinguishes IER from approaches that only replay past transitions during training.
- 6 Conclusion: IER is a simple, general, biologically inspired extension to off-policy reinforcement learning that requires no architectural modifications.
- 6 Conclusion: Empirical results across simulated benchmarks and real-world robotic tasks show consistent improvements in data efficiency and training stability.
Supplementary Materials The following content was not necessarily subject to peer review. · Appendix A: IER Algorithm
The supplementary materials provide methodological details and experimental results, including a pseudo-code specification of Instant Episode Repetition (IER). Appendix A describes how IER initializes training, collects transitions, identifies the best episode, and repeats its action sequence.
- Supplementary Materials The following content was not necessarily subject to peer review.: The supplementary materials organize additional content into the IER pseudo-code, repetition-number analysis, environment descriptions, and hyperparameter configurations.The environments include MuJoCo tasks, the DeepMind Control Suite, and a real-world Dynamic Object Translation with Gripper Manipulation setup.
- Appendix A: IER Algorithm: Appendix A presents the pseudo-code for the Instant Episode Repetition (IER) algorithm as Algorithm 1.The algorithm is titled “Policy Training with Instant Episode Repetition (IER).”
- Appendix A: IER Algorithm: IER initializes the policy, environment, replay buffer, exploration steps Texplore, training steps Ttrain, repetition length RN, and episode-tracking variables.Initialization sets Rmax ← 0, r ← 0, a∗ ← [], resets the environment, and sets Re and step to zero.
- Appendix A: IER Algorithm: During exploration, IER samples actions uniformly from the action space; afterward, it uses policy actions with added exploration noise unless an action sequence is being repeated.This exploration phase promotes state diversity before policy-driven action selection begins.
- Appendix A: IER Algorithm: At each interaction step, IER executes the selected action, observes the transition, stores it in replay buffer M, accumulates episode reward Re, and updates the policy every K steps.The loop continues until Ttrain training steps are completed.
- Appendix A: IER Algorithm: When an episode’s reward Re exceeds Rmax, IER stores that episode’s action sequence as a∗ and repeats it for the next RN episodes.The counter r tracks the number of remaining repeated episodes, while the best-sequence criterion is evaluated when an episode terminates or is truncated.
- Appendix A: IER Algorithm: After episode termination, IER either updates the best action sequence when Re exceeds Rmax or decrements the repetition counter before continuing training.This decision process combines best-episode tracking with fixed-length repetition during environment interaction.
Appendix B: Extended Analysis of the Repetition Number (RN)
The appendix analyzes how the repetition number (RN) affects IER performance across tasks and algorithms. Results favor moderate repetition, while no repetition underuses successful trajectories and excessive repetition can reduce returns.
- Across nearly all tasks, RN=0 is among the weakest settings, while RN=1 improves performance by revisiting high-reward episodes.
- RN=6 remains strong on many tasks, but performance can decline at RN=7, suggesting excessive repetition may cause overexploitation or diminishing returns.
- Domain-Level Analysis: In MuJoCo locomotion, moderate repetition yields the greatest improvements, particularly under TD3, whereas DeepMind Control effects are smoother and less sensitive to RN.
- Agreement Between Baseline Algorithms: Agreement is assessed using overlapping near-optimal RN sets for IER-SAC and IER-TD3, or combined normalized-AUC ranks when no intersection exists.
- Agreement Between Baseline Algorithms: RN2, RN3, and RN5 occur most frequently as agreement choices, while RN2–RN4 provides a stable and effective operating range across environments.
Appendix C: Environment Descriptions
The evaluation uses continuous-control environments from MuJoCo and the DeepMind Control Suite, with low-dimensional vector observations rather than visual input. The tasks span dynamic locomotion, balance, precision control, and varied reward structures.
- Experiments cover continuous-control environments from MuJoCo and the DeepMind Control Suite, two widely adopted platforms for physically realistic motor-control simulations.
- Observations are low-dimensional vectors containing proprioceptive signals such as joint positions, velocities, and orientations, excluding high-dimensional visual input.
- MuJoCo Environments: MuJoCo tasks emphasize high-speed, dynamic locomotion across Ant-v4, HalfCheetah-v4, Humanoid-v4, and Hopper-v4, with challenges involving coordination, stability, balance, and exploration.
- MuJoCo Environments: HalfCheetah-v4 provides dense, unbounded rewards, with cumulative rewards often exceeding 12,000, supporting evaluation of long-horizon learning and experience prioritization.
- DeepMind Control Suite Environments: DeepMind Control Suite tasks use structured state vectors and typically bounded rewards in [0, 1000], emphasizing precision rather than reward accumulation.
- DeepMind Control Suite Environments: DMC includes locomotion, swing-up, and object-rotation tasks, including Walker-Walk, Cheetah-Run, Cartpole-Swingup, and Finger-TurnHard, with differing control sensitivities and success requirements.
Real-World Robotic Task · Appendix D: Configuration and Hyperparameter Settings
The real-world task requires dexterous in-hand cube manipulation with sustained force closure while maximizing displacement, using vision-based state tracking and servo-position actions. Appendix D standardizes PyTorch implementations and shared training configurations across SAC, TD3, and SIL experiments to support fair comparisons.
- Real-World Robotic Task: The dexterous task requires securely grasping a cube and moving it to a desired image-space position while maintaining sustained force closure.Unlike extrinsic-dexterity or caging approaches, the gripper must preserve grasp stability throughout manipulation.
- Real-World Robotic Task: A 90-degree x-axis rotation orients the gripper downward, removes the cube’s supporting base, and increases gravitational demands on grasp stability.The setup builds on a prior two-finger push manipulation task and requires strategies that prevent dropping the object.
- Real-World Robotic Task: Success requires continuously manipulating the cube to maximize displacement while maintaining a stable grasp in a physical environment with limited data and hardware constraints.The task therefore extends beyond simulation through intrinsic gripper dexterity and real-world operating limitations.
- Real-World Robotic Task: The setup uses a Logitech webcam, ArUco markers for tracking cube positions, aluminum extrusions, 3D-printed joints, and a single PC for real-time control.The camera view includes a blue operational-area rectangle and green/red markers for current and previous cube positions.
- Real-World Robotic Task: Rewards combine successful force-closure grasping with lateral cube displacement between consecutive time steps.This structure encourages stable gripping while moving the object steadily over longer distances.
- Real-World Robotic Task: The state contains servo positions, image-space ArUco-marker coordinates, and cube coordinates, while actions specify desired angular positions for each Dynamixel servo joint.These vectors directly control the gripper’s orientation and configuration for precise object manipulation.
- Appendix D: Configuration and Hyperparameter Settings: All agents use PyTorch implementations and consistent architectures, optimization settings, and task configurations, with five random seeds for main experiments and ten for the RN ablation.Table 5 separates shared SAC/TD3 parameters from algorithm-specific SAC, TD3, and SIL settings to isolate algorithmic modifications.