Reward-Conditioned Policies

• Suboptimal trajectories can provide optimal supervision for policies conditioned on the reward or advantage they achieve.This lets actions from mediocre-return trajectories supervise policies targeting those corresponding return or advantage values.Source: §1 · pp. 1–2
• Reward-conditioned policies represent behaviors for different reward values in one model and can generalize to larger returns.Source: §1 · pp. 1–2
• The proposed practical algorithm uses standard supervised learning as an inner-loop component of reinforcement learning.The method is derived from a policy-improvement objective and supports multiple implementation variants.Source: §1 · p. 2
• Some variants perform well on standard benchmark and fully off-policy tasks, but a significant gap remains relative to state-of-the-art reinforcement learning algorithms.Source: §1 · p. 2

• Policy-gradient and value-based reinforcement learning methods are difficult to use effectively because of hyperparameter sensitivity, high sample complexity, and delicate implementation choices.Source: §2 · p. 2
• Supervised learning is comparatively well understood, but it cannot be applied directly when expert demonstrations are unavailable.Source: §2 · p. 2
• Prior work has used supervised learning inside reinforcement learning updates and has combined imitation learning with reinforcement learning.Source: §2 · p. 2
• Reward-conditioned policies generalize the idea that suboptimal trajectories can supervise other tasks by conditioning on reward rather than a goal state.Source: §2 · p. 2

• Reinforcement learning seeks a policy that maximizes expected long-term return in a Markov decision process.The return is the cumulative discounted reward J(θ).Source: §3 · p. 2
• The paper aims to avoid complex policy-gradient estimators and temporal-difference learning.Source: §3 · p. 2
• The generic RCP algorithm samples a target value, rolls out the conditioned policy, labels state-action pairs with observed values, and stores them in a dataset.Source: §3 · p. 3
• Each iteration fits the policy by maximizing log-likelihood on stored state-action-value tuples, then updates the target-value distribution.Source: §3 · p. 3

4 Reward-Conditioned Policies

• RCP alternates supervised training on all collected data with rollouts from the latest policy conditioned on sampled target values.Each rollout is relabeled with observed return-to-go or estimated advantage values, and the resulting (s_t, a_t, Z_t) tuples enter a FIFO dataset.Source: §4 · p. 3, §4.1 · p. 3

4.1 Reward-Conditioned Policy Training

• RCP-R conditions on discounted reward-to-go, whereas RCP-A conditions on the advantage A(s, a) = Q(s, a) − V(s).RCP-A estimates Q with Monte Carlo returns and V with a separately fitted value function, using TD(λ) for the value estimator.Source: §4.1 · p. 3

4.2 Implementation and Architecture Details

• The target-value distribution is represented as a Gaussian whose mean and variance are updated from a soft maximum of observed dataset values.Targets are sampled once per trajectory for RCP-R and at every timestep for RCP-A; evaluation uses the deterministic target μ_Z + σ_Z.Source: §4.1 · p. 3, §4.2 · p. 4
• The policy is a three-layer fully connected network using multiplicative interactions between an embedding of Z and intermediate layers.This architecture is chosen instead of simply concatenating Z with the state, following prior observations about conditioning variables.Source: §4.2 · p. 4, Figure 1 · p. 4

4.3 Theoretical Motivation for Reward-Conditioned Policies

• RCP variants arise by solving a constrained policy-search problem that maximizes return while keeping the trajectory-return distribution close to that of the sampling policy.The optimal nonparametric solution is projected into the parametric policy class, yielding maximum-likelihood training objectives for conditioned trajectories and target distributions.Source: §4.3.1 · pp. 4–5

4.3.2 Advantage-Conditioned Variant (RCP-A)

• The advantage-conditioned derivation replaces the intractable policy state distribution with the sampling-policy distribution when the two policies are close.This approximation motivates a trainable objective that updates target advantages toward higher values.Source: §4.3.2 · p. 5

5.1 Performance and Comparisons on Standard Benchmarks

• RCP-A outperforms TRPO and performs comparably or better than PPO across benchmark tasks, although RCPs remain behind the best current reinforcement-learning algorithms.RCP-R performs at par with RWR, while RCP-A learns successful policies on every benchmark task; results are averaged across five random seeds.Source: §5.1 · p. 7, Figure 2 · p. 7
• With exponential reweighting, both RCP variants improve substantially, and RCP-A performs similarly to AWR because both use the same weighted regression step.RCP-A additionally conditions its policy on advantage values, unlike AWR.Source: §5.1 · p. 7, Figure 3 · p. 7
• RCP-A reaches 238 ± 1.3 on LunarLander-v2, exceeding UDRL’s reported final result of around 150.The comparison suggests that reward conditioning can work effectively, but simple design decisions remain important for performance.Source: §5.1 · p. 7

5.2 Performance in Fully Offline Settings

• In fully offline learning, RCP-A achieves better-than-dataset-average performance on both tested environments, whereas RCP-R performs similarly to behavioral cloning.The method trains directly on static datasets collected from a mediocre partially trained policy.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.3 Ablation Experiments

• RCP-A performs better with 100k than 50k transitions, but performance degrades with larger buffers; AWR degrades significantly as buffer size increases.The experiments vary buffers from 20k to 500k on HalfCheetah-v2, and the authors speculate that larger buffers may help more complex tasks.Source: §5.3 · p. 8, Figure 5 · p. 8
• Multiplicative conditioning outperforms concatenating the target value with the state for both RCP variants on HalfCheetah-v2 and Hopper-v2.Figure 6 evaluates both architectures with 100k replay buffers, including weighted versions.Source: §5.3 · p. 8, Figure 6 · p. 8
• After training, both RCP variants achieve returns or advantages similar in magnitude to their specified target values.Figure 7 visualizes co-occurrence frequencies between target values and rollout outcomes using two-dimensional heatmaps.Source: §5.3 · pp. 8–9, Figure 7 · p. 9

6 Discussion and Future Work

• Reward-conditioned policies treat sub-optimal trajectories as optimal supervision for matching trajectory rewards rather than maximizing reward directly.Conditioning on reward lets one model represent policies for multiple reward values and generalize to larger returns.Source: §6 · p. 9
• The approach attains good results across reinforcement learning benchmarks, but its sample efficiency and final performance lag behind soft actor-critic and AWR.The cited comparison identifies these methods as stronger or more efficient alternatives.Source: §6 · p. 9
• Reward-conditioned policies rely on generalization and random chance to discover trajectories that improve over those already present in the dataset, making exploration a major challenge.Further theoretical and empirical analysis of this behavior could support more performant methods and improved sampling strategies.Source: §6 · p. 9