Source-linked AI summary
Deep Epistemic Value Functions for Optimistic Exploration
Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter, Andreas Krause
TL;DR
Deep RL needs purposeful exploration, but deep epistemic value methods remain brittle in representing, propagating, and optimizing uncertainty. The paper studies these failure modes and combines targeted remedies in DEVOTE, a model-free algorithm. DEVOTE explores more reliably and achieves stronger task performance than competitive model-free and model-based baselines across the reported settings.
Problem
Method
The paper studies uncertainty representation, propagation, and optimization, then combines targeted remedies in the model-free DEVOTE algorithm.
Source: §1 · p. 2Results
Takeaways & Limitations
The findings provide evidence that deep epistemic value functions are a promising path toward scalable, principled exploration.
Source: §7 · p. 9The study is primarily empirical and focused on state-based continuous control; it establishes no theoretical guarantees, and its smoothness-based priors may be limiting in higher-dimensional or representation-learning settings.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Principled exploration in reinforcement learning requires an agent to quantify its epistemic uncertainty and act to resolve it. Uncertainty over the value function provides a natural signal for exploration, yet existing deep approximations remain brittle and perform inconsistently. The central challenge is therefore to scale these ideas robustly. We conduct a systematic empirical study of how epistemic uncertainty is represented, propagated, and optimized in deep epistemic value functions, and uncover distinct failure modes along each of these axes. These findings motivate DEVOTE, a model-free reinforcement learning algorithm that controls how uncertainty generalizes beyond observed data, stabilizes its temporal propagation, and preserves adaptation to the resulting non-stationary exploration objective. Across reward-free exploration and challenging continuous-control tasks, DEVOTE reaches novel states more effectively and achieves higher task return than strong model-free and model-based exploration baselines. These results provide evidence that deep epistemic value functions are a promising path toward scalable, principled exploration.
1 INTRODUCTION
Deep RL needs purposeful exploration to find informative states, but reliable uncertainty estimates with deep networks make principled exploration difficult. The paper studies how to make deep epistemic value functions robust by addressing uncertainty representation, propagation, and optimization.
1 INTRODUCTION
- Local, temporally independent action perturbations cannot sustain coherent exploration over long horizons and may take time exponential in the horizon to find reward.Purposeful exploration instead seeks informative parts of the state space to reduce uncertainty about optimal actions.Source: §1 · p. 1
- Deep epistemic value functions quantify uncertainty about long-term consequences of a policy’s actions and offer a value-directed route to sample-efficient exploration.The paper notes that value-directed exploration admits strong theoretical guarantees.Source: §1 · p. 2
- The study identifies three practical failure modes: uncertainty collapses beyond observed data, TD learning destabilizes its propagation, and a moving exploration objective erodes plasticity.DEVOTE combines controls on uncertainty generalization, separate epistemic propagation from reward learning, and plasticity preservation.Source: §1 · p. 2
- DEVOTE is a model-free algorithm that outperforms strong model-free and model-based exploration baselines across challenging benchmarks.The paper also uses ablations to isolate the contributions of its components.Source: §1 · p. 2
2 THE EXPLORATION PROBLEM
The paper frames exploration as gathering information that helps determine how to act optimally. Because the optimal Q-function determines optimal actions, uncertainty about it is relevant to decision-making.
2 THE EXPLORATION PROBLEM
- An MDP specifies state and action spaces, a transition kernel, an initial-state distribution, a reward function, and a discount factor.The setting is infinite-horizon discounted reinforcement learning.Source: §2 · p. 2
- Uncertainty about the optimal Q-function is directly relevant to decisions because it determines the optimal action in each state.Deep epistemic value functions estimate a value function together with its epistemic uncertainty.Source: §2 · p. 2
- In episodic learning, purposeful exploration collects information that can resolve uncertainty about how to act optimally.The agent observes transitions and rewards after rolling out its policy in an unknown MDP.Source: §2 · p. 2
- Figure 2 compares predictive uncertainty from an ensemble and two ENN priors; the RFN function-space prior yields better-calibrated uncertainty.The figure shows predictive means and uncertainty bands alongside the true function and training observations.Source: Figure 2 · p. 3
3 ADDRESSING LIMITATIONS OF DEEP EPISTEMIC VALUE FUNCTIONS
The paper examines failures in estimating, propagating, and optimizing epistemic value uncertainty with deep networks, then evaluates targeted remedies for each. Its remedies address uncertainty calibration, stable value learning, and continued adaptation during exploration.
3 ADDRESSING LIMITATIONS OF DEEP EPISTEMIC VALUE FUNCTIONS
- Deep epistemic value implementations are brittle because uncertainty estimation, propagation, and optimization each pose challenges for deep neural networks.The study organizes its remedies around these three failure axes.Source: §3 · p. 3
3.1 ESTIMATION: CALIBRATING EPISTEMIC NEURAL NETWORKS
- RFN function-space priors retain uncertainty that grows with distance from training data, unlike the collapsing uncertainty of ensembles and MLP-prior ENNs.The prior’s kernel specifies how uncertainty changes with input similarity.Source: §3.1 · pp. 3–4
3.2 PROPAGATION: STABILIZING EPISTEMIC VALUE ESTIMATION
- TUD improves uncertainty calibration and value fit across prior complexities by bootstrapping reward and prior residuals separately.Naive TD learning mixes the fixed random prior difference into reward, injecting high-variance noise into the Bellman target.Source: §3.2 · pp. 4–5
- TUD-OPT propagates absolute residuals, producing a more stable exploration signal that improves exploration performance.The resulting residual bootstrap is a direct optimistic estimate of reachable novelty.Source: §3.2 · p. 5
3.3 OPTIMIZATION: MAINTAINING PLASTICITY IN OPTIMISTIC EXPLORATION
- Stronger soft resets improve PointMaze state coverage and suppress dormant-neuron growth, helping exploration continue as its objective shifts.Resets periodically interpolate the residual bootstrap network and exploration actor toward new random initializations, while leaving the base critic unchanged.Source: §3.3 · pp. 5–6, Figure 4 · p. 5
4 DEVOTE
DEVOTE combines a local novelty signal with a residual bootstrap critic that estimates reachable novelty, alongside separate exploitation and exploration actors. Different bootstrap policies and soft resets help preserve the intended value estimates and adaptation.
4 DEVOTE
- The residual signal marks poorly covered state-action pairs, and the residual bootstrap critic propagates it into a temporally extended estimate of reachable novelty.This turns local novelty into an exploration signal extending over future actions.Source: §4 · p. 6
- Separate actors maximize the base mean value and optimistic value, while the base and residual critics bootstrap under the exploitation and exploration actors, respectively.The exploration actor is used for environment interaction and converges to the exploitation actor as the optimal policy is identified.Source: §4 · p. 6
- Robust-percentile normalization makes the base value scale comparable to the epistemic term; DEVOTE fixes β = 1 and applies soft resets to support adaptation.The base critic bootstraps under the exploitation actor because using the exploration actor would distort its mean value estimate.Source: §4 · p. 6
5 RESULTS
DEVOTE achieves strong results in both reward-free exploration and high-dimensional control, while component ablations indicate that its remedies contribute differently across environments.
Insight 1
- In pure exploration, Figure 5 reports state coverage over training, with DEVOTE outperforming all baselines on PointMaze and Cheetah and matching the strongest on DeepSea.SAC has poor coverage, while RND, TS, and SOMBRL perform inconsistently across tasks.Source: Figure 5 · p. 7, §5 · pp. 7–8
Experimental Settings
- The evaluation spans three reward-free coverage tasks and two high-dimensional control tasks, using SAC, RND, Thompson Sampling, and SOMBRL as baselines.The control tasks use shaped rewards and sparse goals; results report means across five seeds with standard errors.Source: §5 · p. 7
Insight 2
- Ablations show complementary roles: RFN provides local uncertainty, TUD learning learns it reliably, TUD-OPT stabilizes propagation, and soft resets support adaptation.Soft resets matter most on PointMaze, while the RFN prior and TUD(-OPT) learning matter more on Cheetah; relative component importance depends on environment structure.Source: §5 · p. 8, Figure 6 · p. 8
Insight 1 and Insight 3
- DEVOTE achieves the highest return in AntMaze and PickAndPlace, and outperforms baselines on PointMaze and Cheetah while matching the strongest on DeepSea.In AntMaze and PickAndPlace, it also reaches the sparse goal earlier than the baselines.Source: §5 · pp. 7–8
6 RELATED WORK
Prior work motivates directed exploration through uncertainty over models and value functions, while deep RL also uses several other directed-exploration signals. DEVOTE focuses on scaling value-based uncertainty methods.
Sequential Decision Making
- Sequential decision-making research studies optimism, posterior sampling, and information-directed exploration; value-based methods offer another route with theoretical guarantees.Related work also recursively propagates model-based uncertainty through Bellman-style equations.Source: §6 · pp. 8–9
Deep Reinforcement Learning
- Figure 7 reports normalized return over training in high-dimensional control, where DEVOTE achieves the strongest task performance in both environments.The passage presents this as evidence that its exploration signal can be combined with extrinsic reward.Source: Figure 7 · p. 9
- DEVOTE follows the value-based perspective, addressing how epistemic uncertainty is represented, propagated, and optimized in deep value functions.This focus distinguishes it from approaches that derive exploration from uncertainty in a learned dynamics model.Source: §6 · p. 9
- Deep RL approaches directed exploration through pseudo-counts, prediction error, model disagreement, and uncertainty-aware value functions, alongside largely undirected action noise.Robust optimization remains difficult at scale amid bootstrapping, non-stationarity, and loss of plasticity.Source: §6 · p. 9
7 CONCLUSION
The paper concludes that DEVOTE combines methods for representing, propagating, and adapting epistemic uncertainty, and reports stronger exploration and task performance than competitive baselines. The study is empirical, focuses on state-based continuous control, and does not establish theoretical guarantees on calibration or exploration efficiency.
7 CONCLUSION
- DEVOTE combines structured function-space priors, Temporal Uncertainty Difference learning, and plasticity regularization, and outperforms competitive model-free and model-based baselines across reward-free exploration and continuous-control tasks.The study is empirical and focuses on state-based continuous control; it establishes no theoretical guarantees on uncertainty calibration or exploration efficiency.Source: §7 · p. 9
A.1 MODEL-FREE REINFORCEMENT LEARNING
- Model-free learning estimates policy values from experience, reuses replay-buffer transitions off-policy, and improves continuous-action policies through actor updates that maximize critic estimates.DEVOTE separates task-value learning from its epistemic exploration objective using distinct critics and actors.Source: §A.1 · p. 15
A.2 UNCERTAINTY QUANTIFICATION
- Gaussian processes offer function-space uncertainty, but fixed kernels limit rich task-dependent representations; finite random Fourier features can underestimate uncertainty away from observed data.Random Fourier features approximate stationary kernels with a finite feature map, reducing inference to Bayesian linear regression.Source: §A.2 · p. 16
A.3 RANDOM NETWORK DISTILLATION
- DEVOTE shares RND’s prior–corrector residual and value-based propagation, but uses a distribution of RFN priors and plasticity regularization for reliable exploration.The paper reports that both choices are empirically important across environments.Source: §A.3 · p. 16
B.1 TUD LEARNING
- TUD separates instantaneous corrector residuals from their temporally propagated value, recovering an uncertainty Bellman structure without an environment model.The z-conditioned channel values the pseudo-reward formed by the corrector and prior; its minimizers also minimize the TD loss under the stated expressive-function-class argument.Source: §B.1 · pp. 17–18
C EXTENDED RELATED WORK
- Because learning the prior changes both residual-bootstrap targets and exploration actions, DEVOTE soft-resets those components while retaining the extrinsic base critic.The paper situates this design among methods addressing loss of plasticity in value learning.Source: C EXTENDED RELATED WORK · p. 19
D.1 BASE ALGORITHM AND HYPERPARAMETERS
- DEVOTE uses a SAC-style off-policy actor-critic, rescales extrinsic value by a running magnitude estimate, and fixes the exploration weight β at 1 across environments.Its extrinsic categorical critic uses two-hot likelihood training, while the residual bootstrap critic uses scalar mean-squared-error regression.Source: §D.1 · pp. 20–21
D.2 ENN CALIBRATION
- The calibration study evaluates prior–corrector uncertainty on eight UCI datasets and a synthetic Gaussian-process task, using gap splits and prediction-error-based metrics.UCI correlations use a full-data reference MLP as a denoised proxy, not noiseless ground truth; regression results average ten splits.Source: §D.2 · pp. 21–22
D.3 TD-LEARNING WITH ENNS
The study freezes intermediate SAC policies and trains critics to compare joint TD with disentangled TUD targets. It evaluates uncertainty across states spanning common and low-visitation regions.
D.3 TD-LEARNING WITH ENNS
- Critics are trained for two million steps with the actor and temperature fixed, comparing joint TD and TUD targets across five RFN length scales and three seeds.The frozen policies come from intermediate SAC checkpoints for DMC cartpole swingup or cheetah run.Source: §D.3 · p. 22
- At six anchor configurations per task, evaluation sweeps pole angular velocity over [−12, 12] or cheetah torso forward velocity over [−5, 12], with Monte Carlo values as references.Reference values average 16 frozen-policy rollouts after the anchor action, using discount 0.99 and up to 2,500 agent steps.Source: §D.3 · p. 22, Table 3 · p. 22, Table 4 · p. 22
- The final-checkpoint comparison reports OOD AUROC and correlation between predicted uncertainty and absolute Monte Carlo value error, defining ID/OOD by visitation-density terciles.The middle density tercile is excluded from the ID/OOD comparison.Source: §D.3 · p. 22
D.4 NON-STATIONARITY IN EPISTEMIC OBJECTIVES
The appendix describes a reward-free PointMaze experiment varying soft-reset strength, then outlines pure-exploration benchmarks, baselines, and reporting procedures. These settings assess coverage of novel states across discrete, navigation, and continuous-control environments.
D.4 NON-STATIONARITY IN EPISTEMIC OBJECTIVES
- In a reward-free U-shaped PointMaze, the experiment varies soft-reset strength across five settings and five seeds, resetting the exploration actor and residual bootstrap critic every 50,000 steps.The 4M-step task requires discovering and traversing both maze arms; coverage is the fraction of visited discretized cells.Source: §D.4 · p. 22
D.5 EXPLORATION BASELINES AND REPORTING
- Exploration baselines share a SAC backbone, and the pure-exploration and high-dimensional-control comparisons report seed means and standard errors with smoothed, equally weighted environment curves.The comparisons use five seeds; their seeds are separate from the regression and policy-evaluation experiments.Source: §D.5 · p. 23
D.6 PURE EXPLORATION
- Pure exploration removes task rewards and evaluates state coverage across DeepSea, spiral PointMaze, and DMC Cheetah, with environment-specific step budgets.DeepSea uses 500,000 steps; PointMaze and Cheetah each use 5 million steps. Ablations vary RFN length scale and remove the prior, soft resets, or TUD learning.Source: Figure 8 · p. 23, §D.6 · p. 24
D.7 HIGH-DIMENSIONAL CONTROL
The control tasks pair shaped rewards for basic skills with sparse goals beyond shaping-induced local optima. AntMaze requires a detour around an obstruction, while PickAndPlace tests lifting and placement from an already-grasped starting state.
D.7 HIGH-DIMENSIONAL CONTROL
- AntMaze places a Brax ant in a 9×9 maze with a three-block wall obstructing the direct route, making a detour necessary despite a reward gradient toward the wall.The task provides a one-time reward of 2,500 within one metre of the goal.Source: §D.7 · p. 24
- PickAndPlace starts with the cube grasped, isolating lifting and placement; its lifting reward saturates at 0.15 m and gives no directional signal toward the displaced goal.A one-time reward of 1,250 is granted when the cube is within 0.05 m of the goal.Source: §D.7 · p. 24
- Both control tasks run for 40 million steps, and coverage counts visited position or velocity bins, including cube position while grasped for PickAndPlace.AntMaze coverage divides open maze cells into four bins and excludes walls.Source: §D.7 · p. 24
E.1 CALIBRATING EPISTEMIC NEURAL NETWORKS
The prior determines whether epistemic uncertainty distinguishes familiar from unfamiliar inputs. RFN priors retain uncertainty between observed clusters and achieve the strongest mean OOD detection among evaluated settings, though OOD detection and error–uncertainty correlation favor different methods.
E.1 CALIBRATING EPISTEMIC NEURAL NETWORKS
- Between training clusters, RFN priors preserve uncertainty, whereas bootstrapped ensembles and ENNs with ReLU MLP priors become confident despite inaccurate predictions.The sampled MLP priors are approximately linear over the illustrated domain; RFN variability is controlled by its length scale.Source: §E.1 · p. 25, Figure 9 · p. 25
- Across eight UCI datasets, the RFN ensemble with ℓ=0.25 achieves mean OOD-detection AUROC 0.844, versus 0.749 for the MLP-prior ensemble and 0.734 for the bootstrapped ensemble.This is the strongest mean OOD-detection result among evaluated settings.Source: §E.1 · p. 25
- Error–uncertainty correlation and OOD detection favor different settings: bootstrapping leads mean UCI correlation, while RFN ℓ=0.25 yields better mean OOD detection than RFN ℓ=0.5.RFN ℓ=0.5 has stronger mean correlation than RFN ℓ=0.25, and RFNs improve correlation on the synthetic GP task.Source: §E.1 · p. 25
E.2 STABILIZING EPISTEMIC VALUE FUNCTIONS
TUD separates reward fitting, prior cancellation, and residual bootstrapping to make epistemic value estimates reflect where data constrain them. In the PointMaze diagnostics, resets help maintain adaptation as novelty shifts, while results remain sensitive to prior scale and value estimates can still be poor in low-density regions.
E.2 STABILIZING EPISTEMIC VALUE FUNCTIONS
- TUD separates reward fitting, prior cancellation, and residual bootstrapping because an informative prior alone does not ensure reliable value-function uncertainty under standard TD learning.The intended behavior is to reduce uncertainty in frequently visited states while retaining informative uncertainty elsewhere.Source: §E.2 · p. 25
- Both methods predict values poorly in parts of low-density regions, and TUD improves uncertainty’s relationship to data availability without eliminating extrapolation error.The plotted bands are one predicted standard deviation, not calibrated confidence intervals.Source: §E.2 · p. 26
- In cartpole, TUD reduces uncertainty in frequently visited states while retaining it in unfamiliar regions; standard TD produces comparatively uniform uncertainty across the slices.TUD closely matches Monte Carlo returns, whereas standard TD slightly overestimates reference values.Source: §E.2 · p. 26, Figure 10 · p. 27
- For RFN length scales ℓ≥2.5, TUD improves both OOD detection and error–uncertainty correlation in cheetah, while standard TD shows pronounced overestimation at small scales.Improvements are not uniform across length scales, so separating the learning objectives does not remove sensitivity to prior complexity.Source: §E.2 · p. 26, Figure 11 · p. 27
E.3 MAINTAINING PLASTICITY IN OPTIMISTIC EXPLORATION
- Stronger resets improve aggregate coverage and suppress dormant-neuron growth, but moderate resets can nearly eliminate dormancy while the agent still underexplores.The coverage and dormant-neuron curves aggregate five seeds, whereas the spatial snapshots show individual runs.Source: §E.3 · p. 28
E.4 ADDITIONAL EXPLORATION AND CONTROL RESULTS
The exploration results show that the most effective RFN prior scale depends on the environment, while DEVOTE expands coverage quickly in both complex-control tasks. Coverage complements return by indicating how broadly agents reach beyond behavior already encouraged by rewards.
E.4 ADDITIONAL EXPLORATION AND CONTROL RESULTS
- PointMaze favors RFN length scale ℓ=1, while Cheetah favors ℓ=7.5; the sweep suggests prior complexity must balance variation in unfamiliar regions with learnability in visited ones.In DeepSea, coverage is largely unchanged across ℓ∈{1, 2.5, 5, 7.5}, consistent with one-hot state encoding.Source: §E.4 · p. 28, Figure 13 · p. 30
- DEVOTE expands coverage fastest in AntMaze and PickAndPlace; RND reaches full AntMaze coverage but covers less cube-position space than SAC in PickAndPlace.Figure 14 measures visited-bin coverage in ant x–y position and grasped-cube x–y–z position, respectively.Source: §E.4 · p. 28, Figure 14 · p. 30
- In AntMaze and PickAndPlace, higher coverage indicates reaching broader positions, while return indicates whether behavior advances the task; DEVOTE gains in both quantities.These tasks require exploration beyond behavior already encouraged by reward shaping.Source: §E.4 · p. 28