Source-linked AI summary
Conservative Offline Distributional Reinforcement Learning
Yecheng Jason Ma, Dinesh Jayaraman, Osbert Bastani
TL;DR
Offline RL must learn safe policies from observational data, but expected-return estimates alone do not quantify action risk. CODAC penalizes return quantiles for out-of-distribution actions; it obtains conservative quantile bounds and performs well on both risk-sensitive and risk-neutral tasks.
Problem
Offline RL learns from observational data, while expected-return-focused approaches do not quantify risk or ensure that learned policies act safely.
Source: §1 · p. 1Method
CODAC adapts distributional RL to offline learning by penalizing return-quantile estimates for out-of-distribution actions, especially less-frequent state-action pairs.
Source: §1 · pp. 1–2, §3 · p. 5Results
CODAC outperforms prior approaches on stochastic risk-sensitive offline RL benchmarks and traditional risk-neutral benchmarks, and its quantiles uniformly lower-bound true quantiles in finite MDPs.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The conservative quantile bounds also lower-bound objectives formed by integrating quantiles, including expected return and CVaR.
Source: §1 · p. 2CODAC requires tuning hyperparameters, particularly penalty magnitude α, using online environment rollouts.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Many reinforcement learning (RL) problems in practice are offline, learning purely from observational data. A key challenge is how to ensure the learned policy is safe, which requires quantifying the risk associated with different actions. In the online setting, distributional RL algorithms do so by learning the distribution over returns (i.e., cumulative rewards) instead of the expected return; beyond quantifying risk, they have also been shown to learn better representations for planning. We propose Conservative Offline Distributional Actor Critic (CODAC), an offline RL algorithm suitable for both risk-neutral and risk-averse domains. CODAC adapts distributional RL to the offline setting by penalizing the predicted quantiles of the return for out-of-distribution actions. We prove that CODAC learns a conservative return distribution -- in particular, for finite MDPs, CODAC converges to an uniform lower bound on the quantiles of the return distribution; our proof relies on a novel analysis of the distributional Bellman operator. In our experiments, on two challenging robot navigation tasks, CODAC successfully learns risk-averse policies using offline data collected purely from risk-neutral agents. Furthermore, CODAC is state-of-the-art on the D4RL MuJoCo benchmark in terms of both expected and risk-sensitive performance.
1 Introduction
Offline RL must quantify risk without gathering new, potentially unsafe data, while existing approaches often optimize only expected return. CODAC applies conservative, quantile-based distributional learning offline, with theoretical guarantees and strong results on risk-sensitive and risk-neutral tasks.
1 Introduction
- Offline RL learns from historical observational data rather than risky active environment interactions, but expected-return objectives alone cannot quantify risk or ensure safe behavior.Distributional RL learns full return distributions, which can support risk-aware planning and richer learning signals.Source: §1 · p. 1
- CODAC adapts distributional RL to offline learning by penalizing predicted return quantiles for OOD actions, selectively penalizing less-frequent state-action pairs more heavily.It integrates conservative return-distribution estimation into a quantile-based actor-critic framework.Source: §1 · pp. 1–2
- For finite MDPs, CODAC converges to return quantiles that uniformly lower-bound the true quantiles, with tightness up to finite-data quantile-estimation error.The bounds extend to integrations of quantiles, including expected return and CVaR.Source: §1 · p. 2
- CODAC’s analysis uses novel techniques for the distributional Bellman operator, which acts on an infinite-dimensional space of return distributions rather than a finite-dimensional vector space.The techniques can also bound error in the empirical distributional Bellman operator’s fixed point.Source: §1 · p. 2
- CODAC learns risk-averse policies on two robot-navigation tasks from offline data collected purely by risk-neutral agents, while all baselines fail to solve them.It also achieves state-of-the-art results on risk-neutral and modified risk-sensitive D4RL MuJoCo benchmarks.Source: §1 · p. 2
- CODAC extends conservative offline Q-learning from expected values to all return quantiles, while prior offline distribution-matching methods can perform poorly on data from suboptimal policies.Quantile-based objectives encompass expected return and risk-sensitive criteria such as CVaR, Wang, and CPW.Source: §1 · pp. 2–3
2 Background
The background defines offline RL and distributional evaluation, then motivates conservative estimation: fitted distributional evaluation can overestimate returns for rarely observed state-action pairs. Quantile representations support both expected-return and risk-sensitive objectives.
Offline RL
- Offline RL learns a policy from a fixed dataset of transitions without interacting with the environment, and actions with low empirical behavior-policy probability are treated as OOD.The dataset need not come from a single behavior policy.Source: §2 · p. 3
Distributional RL
- Distributional RL estimates the return distribution rather than only its expectation, and the distributional Bellman operator has the true policy return distribution as its fixed point.The operator can be iteratively applied to obtain that distribution.Source: §2 · pp. 3–4
Fitted distributional evaluation
- Fitted distributional evaluation approximates the distributional Bellman operator using offline data and iteratively fits a return distribution under a Wasserstein-based loss.Its estimation can overestimate returns for rarely observed state-action pairs, which policy optimization may exploit.Source: §2 · p. 4
Quantile-based distributional RL
- Quantile-based distributional RL represents returns with a quantile function and supports objectives including expected return, CVaR, CPW, and Wang measures.CVaR considers only the bottom ξ-percentile of returns.Source: §2 · p. 4
Fitted distributional evaluation
- The policy-learning objective is to avoid overestimating uncertain, rarely observed actions, because optimizing those estimates can favor actions with high uncertainty.The paper proposes a penalty to address this issue.Source: §2 · p. 4
3 Conservative offline distributional policy evaluation
CDE adds a penalty to distributional policy evaluation, producing conservative return-quantile estimates in finite MDPs. Its analysis establishes a convergent lower bound and shows greater conservatism for actions that are rare in the dataset.
3 Conservative offline distributional policy evaluation
- CDE penalizes quantile values for out-of-distribution actions relative to in-distribution actions, with α controlling penalty magnitude.The penalty coefficient c0 should be larger for OOD actions.Source: §3 · p. 5
- The CDE operator is a γ-contraction in maximum Wasserstein distance, so its iterates converge to a unique fixed point.The contraction follows from composing the distributional Bellman operator with a non-expansion shift operator.Source: §3 · pp. 5–6
- CDE’s fixed point lower-bounds the true return quantiles uniformly, and the bound is tight up to finite-data quantile approximation error.The result applies under the stated condition on α.Source: §3 · p. 6
- The quantile lower bound also yields conservative estimates for distorted-expectation objectives, including expected return and CVaR.The result extends to objectives such as Wang and CPW risk measures.Source: §3 · p. 6
- CDE expands the quantile gap between in-distribution and out-of-distribution actions, making its estimates more conservative for rarer actions.The resulting lower bounds can be less conservative for in-distribution actions.Source: §3 · p. 6
- The theorem’s condition on α can allow a small penalty when dataset coverage is wide enough to keep ∆(s, a) small.The passage relates wide coverage to large counts n(s, a).Source: §3 · p. 6
4 Conservative offline distributional actor critic
CODAC incorporates conservative distributional evaluation into an actor-critic framework through a min-max objective. It represents return distributions with quantile functions and optimizes them using distributional temporal-difference learning.
4 Conservative offline distributional actor critic
- A min-max objective has an inner policy-maximization step and an outer minimization of the conservative distributional evaluation objective.The framework interprets µ as the actor policy and its first term as the actor objective.Source: §4 · p. 7
- CODAC represents the quantile function with a neural network and averages its objective over quantiles sampled uniformly from [0, 1].The paper calls this strategy conservative offline distributional actor critic (CODAC).Source: §4 · p. 7
- Distributional temporal-difference learning estimates the Wasserstein-distance objective using sampled transitions, actions, and quantiles.The TD error compares the reward plus discounted target quantile with the current quantile; the loss uses τ-Huber quantile regression.Source: §4 · p. 7
- The resulting objective can be optimized with an off-policy actor-critic method; the paper uses distributional soft actor-critic.Its critic replaces SAC’s Q-network with a quantile distributional critic.Source: §4 · p. 7
5 Experiments
CODAC achieves strong results across risk-sensitive and risk-neutral offline RL evaluations, including risky robot navigation and D4RL. Experiments also examine conservative value estimates and quantile-gap expansion.
5.1 Risky robot navigation
- CODAC-C outperforms other approaches on CVaR0.1 return and violations, and is the only method reported to learn risk-averse behavior in risky navigation.On Risky PointMass, it achieves zero violations despite that behavior being absent from the dataset; it also outperforms ORAAC, which is encouraged to imitate suboptimal, risk-neutral trajectories.Source: Table 1 · p. 8, §5.1 · p. 8
5 Experiments
- CODAC achieves the best overall performance in both risk-sensitive and risk-neutral D4RL variants and state-of-the-art results across the reported offline RL tasks.The experiments include risky robot navigation, risk-sensitive D4RL, and the risk-neutral D4RL benchmark.Source: §5 · p. 7, Table 2 · p. 9
5.2 Risk-sensitive D4RL
- On risk-sensitive D4RL, CODAC-N and CODAC-C outperform CQL and ORAAC on most datasets, though direct CVaR optimization can be unstable.CODAC-C exceeds CODAC-N on CVaR0.1 on about half the datasets; both CODAC-C and ORAAC perform worse than risk-neutral algorithms on Walker2d-Medium.Source: §5.2 · p. 9
5.3 Risk-neutral D4RL
- On risk-neutral D4RL, CODAC achieves state-of-the-art results on five of 12 datasets, and the results suggest its conservative penalty is important to its strong performance.The five datasets are walker2d-random, hopper-medium, hopper-mixed, halfcheetah-medium-expert, and walker2d-medium-expert.Source: §5.3 · pp. 9–10
5.4 Analysis of Theoretical Insights
- CODAC’s predicted expected and CVaR values lower-bound Monte Carlo estimates, while its quantile gap exceeds the no-penalty variant’s on more than 90% of sampled tuples.CODAC also obtains significantly higher returns than the no-penalty variant in the gap-expansion analysis.Source: Table 3 · p. 10, §5.4 · p. 10, Table 4 · p. 10
6 Conclusion
CODAC provides conservative return-quantile estimates whose implications include lower bounds on Q and CVaR values. The appendix proof passage rewrites the objective using the p-Wasserstein distance and derives a condition for its minimizing distribution.
6 Conclusion
- CODAC’s conservative return-quantile estimates translate into lower bounds on Q and CVaR values.The conclusion also reports outperformance on stochastic risk-sensitive and traditional risk-neutral offline RL benchmarks.Source: §6 · p. 10
A.1 Proof of Lemma 3.4
- Using the p-Wasserstein distance as the Lp metric between quantile functions, the CODAC objective is rewritten; arbitrary smooth, compactly supported perturbations imply Z(s,a) must minimize it, or a perturbation decreases its value.Source: §A.1 · p. 14
A.2 Proof of Theorem 3.6
The proof bounds quantile-estimation error and uses those bounds in the convergence argument. The supplied passages show concentration and fixed-point steps, but omit the theorem’s complete statement.
A.2 Proof of Theorem 3.6
- A concentration bound on quantile values is used to bound the estimation error of the empirical distributional Bellman operator against the true operator.The proof invokes a concentration result for the empirical CDF and monotonicity to bound quantile functions.Source: §A.2 · pp. 14–15
- The proof applies the concentration bound to the true return distribution, whose distribution is a fixed point of the true distributional Bellman operator.The argument identifies Zπ = TπZπ and then rearranges the resulting bound.Source: §A.2 · p. 15
- A final proof step uses a bound for the ≥ case and requires the last term to be positive.The passage relates this positivity condition to c0(s, a) > 0.Source: §A.2 · p. 15
A.3 Proof of Theorem 3.8
The proof uses inequalities involving policy and data-distribution expectations to establish a result for sufficiently large penalty magnitude. The supplied passages provide intermediate steps rather than the theorem’s full statement.
A.3 Proof of Theorem 3.8
- The proof compares expectations under µ and ˆπβ, defining ¯c(s) and ¯∆(s) from their respective c0 and ∆ terms.The expressions are given as ¯c(s) = Eµ(a|s)c0(s, a) − Eˆπβ(a|s)c0(s, a) and ¯∆(s) = Eµ(a|s)∆(s, a) + Eˆπβ(a|s)∆(s, a).Source: §A.3 · p. 16
- With Z0 = ˜Z0 and Zk = (T π)kZ0, ˜Zk = (˜T π)k˜Z0, applying Lemma A.3 with Z = ˜Zk and ¯∆ = 4Vmax gives the next bound.Source: §A.3 · p. 16
A.4 Proof of Lemma A.1
The appendix links empirical-CDF concentration to quantile error, then describes how distributional Bellman bounds propagate. It also states that these techniques yield finite-sample error bounds for the empirical operator’s fixed point.
A.4 Proof of Lemma A.1
- If a CDF F is ζ-strongly monotone and within ε in sup norm of CDF G, their inverse CDFs are within ε/ζ in sup norm.The proof uses this result to translate empirical-CDF concentration into quantile-function error bounds.Source: §A.4 · pp. 17–18
A.5 Proof of Lemma A.2
- The A.5 argument propagates a distributional bound by shifting the CDF argument, with the constant offset contracted by γ under the Bellman operator.The proof identifies this as the operator’s additive behavior on a constant β and uses its γ-contraction.Source: §A.5 · p. 19
A.6 Bound on error of the fixed-point of the empirical distributional bellman operator
- A.7 states that the techniques give finite-sample bounds on value-iteration error using the empirical Bellman operator relative to the true operator.The supplied theorem statement is truncated after the opening of its norm bound.Source: §A.6 · p. 19
B Algorithm and implementation details
CODAC uses a learnable penalty formulation and distributional actor-critic updates, with neural networks representing the policy and quantile function.
B.1 Actor-Critic objective
- A learnable penalty thresholds and scales the quantile-value difference between the policy and the dataset policy, with α learned by dual gradient descent.Source: §B.1 · p. 20
B.2 Neural network architecture
- The policy uses a two-layer, 256-unit ReLU network, while the quantile function combines state-action features with quantile embeddings.The quantile architecture uses one-layer 256-unit state-action features, a one-layer 64-unit quantile embedding, and a 256-unit output network; layer normalization follows each activation.Source: §B.2 · p. 20
B.3 Actor-Critic updates
- Each update samples quantiles, computes a distributional TD loss, applies the CODAC penalty, and updates the critic, penalty, and policy.Source: §B.2 · p. 21
- To reduce computation, CODAC penalizes one randomly selected quantile per update and computes the risk objective by averaging quantiles below the risk threshold.The authors report that subtracting average quantile values is more stable than subtracting only the corresponding quantile value.Source: §B.3 · p. 21
C.1 Risky robot navigation
CODAC is evaluated on risky PointMass and Ant navigation tasks using offline data from a risk-neutral agent. The reported results show risk-averse behavior and compare alternative risk-sensitive objectives.
C.1 Risky robot navigation
- CODAC-C achieves the best performance on most risky-navigation metrics and is the only method reported to learn risk-averse behavior.In the Ant trajectories, it avoids the risky region while still reaching the goal.Source: Table 6 · p. 22, §C.1 · p. 22, Figure 5 · p. 23
- The PointMass and Ant tasks penalize entering a risky region with a low-probability large negative reward, while rewarding progress toward a goal.Offline datasets come from a distributional SAC agent trained online, and evaluations use 100 episodes with the same initial states.Source: §C.1 · p. 22
- CODAC-Wang performs similarly to CODAC-CVaR, trading slightly better average performance for less safety, while CODAC-CPW is on par with CODAC-Neutral.The authors relate these patterns to Wang assigning vanishingly small weight above the risk cutoff and CPW modeling human gameplay behavior.Source: §C.1 · p. 22, Table 5 · p. 22
C.2 Stochastic D4RL Mujoco suite
The appendix reports additional stochastic and risk-neutral D4RL results, behavioral statistics, hyperparameter choices, and compute details. In the stochastic tasks, CODAC-C has the lowest HalfCheetah violation rate, while safety comparisons on Walker2d depend on whether agents learned to walk.
C.2 Stochastic D4RL Mujoco suite
- Stochastic reward modifications add penalties for high forward velocity in Half-Cheetah and excessive pitch in Hopper and Walker2d.The appendix reports evaluation on the modified environments and includes full results for Expert, Medium, and Mixed datasets.Source: §C.2 · p. 23
- CODAC-C has the lowest violation percentage on stochastic HalfCheetah; on Walker2d, among methods that learned to walk, it has the lowest average angular velocity while maximizing return.CQL appears safest on Walker2d, but the authors attribute that result to its failure to learn desirable walking behavior.Source: Table 8 · p. 24, §C.2 · p. 24
C.3 D4RL Mujoco suite
- The risk-neutral D4RL results include additional baselines BEAR and BRAC, with normalized-return results reported in Table 9.Algorithms were trained for 1M gradient steps and evaluated for 10 episodes on the original environments.Source: §C.3 · p. 24, Table 9 · p. 24
C.4 Hyperparameters
- CODAC uses a learnable α in most cases and searches different penalty scales for risk-neutral and risk-sensitive D4RL; the best risk-sensitive penalty scale is smaller across most datasets.For risk-neutral tasks, ω = 1 is effective for Mixed and Random datasets, and ω = 10 for Medium and Medium-Expert datasets.Source: §C.4 · pp. 24–25
C.5 Compute resources
- Experiments ran on one Nvidia 2080-Ti with 32 cores, and each CODAC run took about 10 hours.Source: §C.5 · p. 25