PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning

Hoda Yamani, Henry Williams, Bruce A. MacDonald

arXiv:2608.17373v1cs.LGcs.AI

TL;DR

Image-based reinforcement learning needs more efficient ways to select informative experiences while encouraging exploration. This paper introduces NSPER and NSPER+R, which combine novelty and surprise for replay prioritization and intrinsic rewards, improving performance and training efficiency across DeepMind Control Suite tasks.

  • Problem

    Image-based reinforcement learning remains sample-inefficient because high-dimensional visual data creates redundancy, memory demands, and computational overhead, while integrating replay prioritization with exploration signals remains underexplored.

  • Method

    NSPER uses novelty and surprise to prioritize replay experiences, while NSPER+R also incorporates both signals as intrinsic rewards to support exploration and policy learning.

  • Results

    NSPER outperforms baseline algorithms on four of five tasks and matches the best methods on the fifth, while NSPER+R achieves the best overall results.

  • Takeaways & Limitations

    Novelty and surprise provide complementary signals for replay prioritization and exploration, supporting faster, more efficient learning in image-based reinforcement learning.

Abstract

from arXiv · show

Sample efficiency is a central challenge in reinforcement learning (RL), particularly in image-based domains where agents must learn from high-dimensional visual inputs. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration. Prioritized Experience Replay (PER) addresses part of this challenge by reusing high-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states. However, their integration has not been extensively studied. This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding of the environment. We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER+R improve training efficiency and convergence speed compared to existing methods in image-based RL.

I. INTRODUCTION · II. RELATED WORKS

The introduction frames image-based reinforcement learning as sample-inefficient because high-dimensional visual inputs create redundancy, memory demands, and computational overhead. It presents NSPER and NSPER+R, which combine novelty and surprise for experience prioritization and intrinsic-reward-driven exploration, and evaluates them in PixelTD3 on DeepMind Control Suite tasks.

  • I. INTRODUCTION: Image-based RL enables learning directly from visual input across gaming, robotics, and autonomous driving, but high-dimensional images increase redundancy, memory demands, and computational overhead.These challenges make policy learning more complex and resource-intensive.
  • I. INTRODUCTION: Intrinsic signals can improve sample efficiency by encouraging diverse experiences as rewards and prioritizing informative transitions during replay.The two uses target exploration and experience selection as complementary mechanisms.
  • I. INTRODUCTION: Novelty directs exploration toward unfamiliar or underrepresented states, whereas surprise highlights outcomes that deviate from predictions and expose gaps in environmental understanding.The paper identifies both signals as especially effective intrinsic rewards.
  • I. INTRODUCTION: TD-PER prioritizes transitions by temporal-difference error, but its effectiveness declines in continuous-control environments.TD error measures the discrepancy between predicted and actual outcomes after an action.
  • I. INTRODUCTION: NSPER combines novelty and surprise as prioritization signals within PER to select diverse, informative transitions with high learning potential.The method is proposed to improve the quality of replayed experiences.
  • I. INTRODUCTION: NSPER+R extends NSPER by using novelty and surprise both for replay prioritization and as intrinsic rewards, targeting exploration and policy learning together.The paper describes this extension as further enhancing exploration and policy learning.
  • I. INTRODUCTION: The methods are integrated into PixelTD3 and evaluated on challenging DeepMind Control Suite tasks, with reported improvements in complex, high-dimensional visual environments.The introduction states that the approach leads to faster convergence and more generalizable policies, while the implementation is publicly available.
  • II. RELATED WORKS: Related work establishes experience prioritization and intrinsic motivation as complementary strategies for identifying informative transitions and guiding agents toward novel or surprising experiences.This foundation motivates combining the two ideas for learning in complex environments.

A. Experience Prioritization Methods · B. Intrinsic Motivation Strategies

Experience prioritization methods improve sample efficiency but TD error can be noisy and unstable in continuous high-dimensional control, motivating alternative signals. Intrinsic motivation promotes exploration through novelty and surprise, although integrating these signals with replay prioritization remains challenging.

  • A. Experience Prioritization Methods: PER improves sample efficiency by assigning higher sampling probability to transitions with greater learning potential.
  • A. Experience Prioritization Methods: TD-PER ranks transitions by TD error, assuming larger errors indicate less familiar and more informative experiences.
  • A. Experience Prioritization Methods: In continuous high-dimensional spaces, small action or observation perturbations can cause large value-estimate fluctuations, making TD error noisy and unstable.
  • A. Experience Prioritization Methods: LA3P decouples learning signals by assigning low-TD-error transitions to the actor and high-TD-error transitions to the critic.
  • A. Experience Prioritization Methods: Prioritizing recent low-TD-error transitions emphasizes up-to-date experiences but risks catastrophic forgetting by discarding older valuable data.
  • A. Experience Prioritization Methods: RPE prioritization measures discrepancies between expected and actual rewards, providing feedback about prediction accuracy for policy adjustment.
  • B. Intrinsic Motivation Strategies: Intrinsic motivation encourages agents to explore beyond extrinsic rewards by seeking novel states and behaviors.
  • B. Intrinsic Motivation Strategies: Novelty targets rarely encountered states, while surprise uses prediction errors such as next-state errors and state-transition divergences.

C. Integrating Intrinsic Signals in Experience Replay · III. METHODOLOGY · A. PER

The paper integrates novelty and surprise into experience prioritization and intrinsic rewards to support exploration and policy optimization, building on PER and related curiosity-based methods. Its methodology reviews PER and introduces NSPER and NSPER+R for image-based reinforcement learning.

  • C. Integrating Intrinsic Signals in Experience Replay: CDP prioritizes trajectories leading to rare goal states, but its reliance on hindsight experience replay limits applicability to goal-oriented tasks.
  • C. Integrating Intrinsic Signals in Experience Replay: ACDER likewise integrates curiosity signals into hindsight experience replay, retaining the same task-oriented limitation.
  • C. Integrating Intrinsic Signals in Experience Replay: CCLF selects surprising augmented inputs for sample efficiency in image-based RL but emphasizes representation learning over policy-focused experience selection.
  • C. Integrating Intrinsic Signals in Experience Replay: The proposed method combines novelty and surprise in prioritization so selected transitions support both exploration and policy optimization.
  • C. Integrating Intrinsic Signals in Experience Replay: NSPER integrates intrinsic rewards with structured prioritization to address limitations of PER and enable more effective learning in challenging environments.
  • III. METHODOLOGY: The methodology reviews PER, PixelTD3, and novelty and surprise computation before presenting NSPER and its NSPER+R extension.
  • A. PER: PER improves sample efficiency by replaying transitions with higher learning potential, sampling according to priorities that can be defined using TD error or alternative signals.In TD-PER, δ_i is the TD error and ϵ > 0 ensures non-zero priority; α controls prioritization sharpness and β compensates for sampling bias.

B. PixelTD3 · C. Novelty And Surprise Computation

PixelTD3 provides NSPER with a shared, jointly trained latent representation for image-based policy learning. NSPER derives novelty from reconstruction differences and surprise from latent dynamics-prediction errors, combining both into the Novelty-Surprise Signal for targeted exploration.

  • B. PixelTD3: PixelTD3 extends TD3 with a convolutional autoencoder that maps raw observations into a compact latent representation shared by actor and critic networks.The encoder–decoder is jointly trained with reconstruction loss to preserve structural image content.
  • B. PixelTD3: Unlike SACAE, PixelTD3 uses one shared autoencoder updated end-to-end at every step, forming NSPER’s backbone.The update includes both convolutional and linear layers, while architectural specifications are provided in Table III.
  • C. Novelty And Surprise Computation: Novelty measures how different an observation is from previously encoded observations using the complement of SSIM between the input and its autoencoder reconstruction.The reconstruction is defined as ŝ_t = Dec(Enc(s_t)).
  • C. Novelty And Surprise Computation: Higher novelty indicates poorer reconstruction, so the autoencoder reconstruction error identifies observations that are more novel to the agent.Figure 1 illustrates novelty computation from the input image’s reconstruction error.
  • C. Novelty And Surprise Computation: Surprise measures predicted-versus-observed dynamics mismatch in latent space using an ensemble of models that predict the next latent state from the current state and action.It is defined as mean squared error between predicted and observed latent states; averaging M models reduces variance and improves stability under non-stationary encoder updates.
  • C. Novelty And Surprise Computation: Combining novelty and surprise produces the Novelty-Surprise Signal (NSS), an intrinsic reward that encourages targeted exploration of informative environmental regions.NSS(s, a) captures the joint contribution of the two signals and is derived from equations (5) and (7).

D. NSPER: Novelty and Surprise Prioritization

NSPER prioritizes replay using a Novelty–Surprise Signal rather than TD error, then applies these priorities to transition sampling and importance weighting. Its α parameter balances emphasis on informative experiences against over-concentration on extreme transitions.

  • D. NSPER: Novelty and Surprise Prioritization: NSPER replaces PER’s TD-error term δ_i with the Novelty–Surprise Signal NSS(s, a) to assign transition priorities.The strategy uses novelty and surprise as intrinsic signals to guide experience replay.
  • D. NSPER: Novelty and Surprise Prioritization: ϵ > 0 keeps all transitions eligible for replay.This condition ensures no transition is excluded entirely from replay.
  • D. NSPER: Novelty and Surprise Prioritization: NSPER uses novelty- and surprise-based priorities in sampling probabilities and importance weights to favor diverse, informative experiences.The method is intended to better support critic training.
  • D. NSPER: Novelty and Surprise Prioritization: The parameter α regulates prioritization strength, emphasizing informative transitions without concentrating sampling on a few extreme cases.This provides a balance in replay prioritization.

E. Enhancing Exploration with Intrinsic Rewards

NSPER+R enhances exploration by defining intrinsic rewards from novelty and surprise, while using the same Novelty–Surprise Signal (NSS) to prioritize replay experiences. The combined framework prioritizes transitions supporting task success and deeper environmental understanding.

  • Intrinsic reward design: Intrinsic rewards use novelty and surprise to promote exploration when extrinsic rewards are sparse or delayed.The intrinsic reward is defined from the two signals.
  • Intrinsic reward design: NSS is used both to prioritize experiences during replay and to define the intrinsic reward signal.The total reward combines extrinsic and intrinsic components.
  • NSPER+R: NSPER+R integrates intrinsic rewards with prioritized experience replay to prioritize transitions contributing to task success and deeper environmental understanding.The paper presents NSPER+R as the full version of the framework.
  • NSPER+R: The NSS combines novelty and surprise, and NSPER+R sets the intrinsic reward to NSS while NSPER uses the extrinsic reward.New replay items receive priority |NSS| + ε, and refreshed priorities use the same expression.

IV. EXPERIMENTS

The experiments define the evaluation setup, simulation tasks, algorithmic settings, and baseline methods, then analyze learning performance across NSPER, NSPER+R, and competing approaches.

  • Experimental setup: The evaluation specifies the experimental setup and simulation tasks.
  • Experimental setup: It reports the algorithmic settings used in the experiments.
  • Learning performance: The analysis compares learning performance for NSPER, NSPER+R, and competing approaches.

A. Experimental Setup · B. Experimental Approach

The evaluation uses five diverse image-based continuous-control tasks from the DeepMind Control Suite under a unified configuration. NSPER isolates novelty and surprise for replay prioritization, while NSPER+R also uses them as intrinsic rewards and is compared with established PixelTD3 replay strategies.

  • A. Experimental Setup: The study evaluates algorithms on five DeepMind Control Suite tasks: Cartpole-Balance, Finger-Spin, Ball-in-Cup, Walker-Walk, and Cheetah-Run.The tasks cover balance, manipulation, and locomotion in image-based continuous control.
  • A. Experimental Setup: All algorithms use identical training, optimization, evaluation, and replay configurations so differences are attributed to prioritization mechanisms rather than hyperparameters or implementation bias.NSPER and NSPER+R use a shared latent-space architecture for action-value estimation.
  • A. Experimental Setup: Training runs for 1 × 106 environment steps with prioritization exponent α 0.7, importance-sampling exponent β 0.4, evaluation every 1 × 104 steps, 10 episodes per checkpoint, and 5 random seeds.The exponents control prioritization sharpness and sampling-bias correction, respectively.
  • B. Experimental Approach: NSPER uses novelty and surprise solely for replay prioritization to isolate their effect on experience selection.This variant does not add novelty or surprise as intrinsic rewards.
  • B. Experimental Approach: NSPER+R extends NSPER by combining novelty- and surprise-based intrinsic rewards with extrinsic rewards to assess their joint impact on prioritization and exploration.The variant therefore evaluates both replay selection and exploration effects.
  • B. Experimental Approach: Comparisons use established replay strategies within PixelTD3, including Uniform, Uniform+R, TD-PER, TD-PER+R, RPE-PER, RPE-PER+R, and CCLF.All implementations share one codebase; CCLF is adapted from its original pixel-based SAC backbone to PixelTD3.

C. Learning Efficiency Analysis

Figure 3 evaluates learning efficiency across five challenging DeepMind continuous control environments using average evaluation reward over training steps. It compares NSPER, NSPER+R, and baseline methods with and without intrinsic rewards, while noting that relative method rankings remained consistent across multiple runs despite baseline variability.

  • Learning curves: Figure 3 compares NSPER, NSPER+R, and baseline methods across five challenging DeepMind continuous control environments using average evaluation reward over training steps.The learning curves provide insight into convergence speed and performance differences.
  • Learning curves: Dashed-line plots show PixelTD3 with different replay-buffer sampling methods without intrinsic rewards.
  • Learning curves: Solid lines of the same color show the corresponding sampling methods with intrinsic rewards, enabling direct comparisons of sampling strategies under both conditions.
  • Robustness and limitations: Baseline performance deviations from prior work can arise from environmental stochasticity, initialization randomness, or implementation and runtime differences.Despite these deviations, the relative ranking of methods remained consistent across multiple runs.

V. RESULTS AND DISCUSSION · A. Task-Specific Analysis

Across DeepMind Control Suite tasks, NSPER improves PixelTD3 on four of five tasks and remains comparable on the fifth, while NSPER+R achieves the strongest overall results. Task-specific outcomes show that prioritization effectiveness depends on the environment, with novelty and surprise often supporting efficient learning.

  • V. RESULTS AND DISCUSSION: NSPER improves PixelTD3 on four of five tasks and matches the best image-based RL methods on the fifth.Uniform sampling performs suboptimally across all tasks, while novelty and surprise signals improve learning efficiency.
  • V. RESULTS AND DISCUSSION: NSPER+R achieves the best overall results, supporting the combination of prioritization strategies and intrinsic rewards across nearly all algorithms.CCLF consistently outperforms Uniform but remains unremarkable overall.
  • V. RESULTS AND DISCUSSION: RPE-PER performs comparably to NSPER across tasks, but NSPER consistently does better, indicating stronger novelty-and-surprise heuristics for informative experience selection.The findings also suggest that complex prioritization scales can benefit sample-efficient image-based RL.
  • A. Task-Specific Analysis: Across individual environments, the analysis compares methods’ strengths and limitations across five image-based reinforcement-learning tasks.The tasks are Cartpole-Balance, Finger-Spin, Ball-in-Cup, Walker-Walk, and Cheetah-Run.
  • A. Task-Specific Analysis: In Cartpole-Balance, all methods using prioritized buffers or intrinsic rewards perform well, suggesting PixelTD3 benefits from either approach in simpler environments.Cartpole-Balance requires balancing a pole with a cart.
  • A. Task-Specific Analysis: In Finger-Spin, NSPER variants outperform other methods, while NSPER+R further enhances performance by combining novelty, surprise, PER, and intrinsic rewards.The task is difficult because of complex contact dynamics and sparse rewards.
  • A. Task-Specific Analysis: In Ball-in-Cup, NSPER+R outperforms other approaches, while NSPER+R and RPE-PER+R consistently exceed Uniform+R, linking aligned prioritization and intrinsic rewards to stronger performance.Ball-in-Cup provides sparse rewards only when the ball is successfully caught.
  • A. Task-Specific Analysis: In Walker-Walk, NSPER+R and RPE-PER+R improve performance, whereas TD-error-based PER remains suboptimal with intrinsic rewards; in Cheetah-Run, TD-PER performs best and NSPER closely follows.TD-PER’s contrasting results across Walker-Walk and Cheetah-Run indicate that prioritization should be tailored to each environment, while NSPER remains stable.

B. Ablation Study: The Role of Novelty and Surprise in Prioritization · VI. CONCLUSION

The ablation study finds that jointly prioritizing novelty and surprise outperforms either signal alone, while their effects differ across tasks and intrinsic-reward settings. The conclusion reports that NSPER and NSPER+R improve replay quality, exploration, training efficiency, convergence, and policy generalization, while motivating adaptive signal weighting.

  • B. Ablation Study: The Role of Novelty and Surprise in Prioritization: NSPER jointly uses novelty and surprise for experience prioritization, whereas NoveltyPER and SurprisePER use only one signal.Variants with intrinsic rewards are shown as solid curves, and variants without them as dashed curves.
  • B. Ablation Study: The Role of Novelty and Surprise in Prioritization: Combining novelty and surprise outperforms prioritization based on either signal alone.The comparison includes methods with and without intrinsic rewards.
  • B. Ablation Study: The Role of Novelty and Surprise in Prioritization: In Cheetah-Run, prioritization alone is sufficient, while adding surprise as an intrinsic reward reduces performance.This task-specific result shows that intrinsic rewards do not uniformly improve prioritization.
  • B. Ablation Study: The Role of Novelty and Surprise in Prioritization: NoveltyPER outperforms SurprisePER in most tasks, indicating greater benefit from encouraging diverse states than from focusing on unexpected transitions.SurprisePER and SurprisePER+R show similar performance in most tasks.
  • VI. CONCLUSION: NSPER and NSPER+R address sample inefficiency in image-based RL by using novelty and surprise for replay prioritization, with NSPER+R also using them as intrinsic rewards.The dual mechanism is described as improving replay quality and exploration while producing faster convergence and more generalizable policies.
  • B. Ablation Study: The Role of Novelty and Surprise in Prioritization: The ablation compares novelty, surprise, and combined prioritization with and without intrinsic rewards using five runs and 95% confidence intervals.Shaded regions in Fig. 4 represent the confidence intervals.
  • VI. CONCLUSION: Experiments with the PixelTD3 backbone on the DeepMind Control Suite show that NSPER and NSPER+R outperform existing baselines and improve training efficiency in complex image-based tasks.The conclusion also reports complementary contributions of novelty and surprise to prioritization and exploration.
  • VI. CONCLUSION: Future work should investigate additional intrinsic motivators and adaptively weight novelty and surprise because the signals affect learning efficiency differently.Dynamic balancing is proposed to refine the agent’s exploration strategy.
Loading 2608.17373v1…