Source-linked AI summary
Soft Policy Optimization: Online Off-Policy RL for Sequence Models
Taco Cohen, David W. Zhang, Kunhao Zheng, Yunhao Tang, Remi Munos, Gabriel Synnaeve
TL;DR
Unlike pretraining, reinforcement learning still repeats exploration in each training run, much like members of a species unable to learn from others’ discoveries and mistakes. SPO is an online off-policy soft RL method that uses online and offline data and parameterizes token-level soft Q-values through policy and reference-model log-probabilities. On code contests, SPO leverages diverse, highly off-policy data and, unlike PPO, preserves policy diversity, resulting in improved pass@10; speedups of about 85% compared to PPO are reported.
Problem
Unlike pretraining, reinforcement learning still repeats exploration in each training run, much like members of a species unable to learn from others’ discoveries and mistakes.
Source: §1 · p. 1Method
SPO is an online off-policy soft RL method that uses online and offline data and parameterizes token-level soft Q-values through policy and reference-model log-probabilities.
Source: §1 · p. 1Results
Takeaways & Limitations
Learning a high-entropy policy is more aligned with generative-modeling objectives than classical hard RL that converges to a deterministic policy.
Source: §2.1 · p. 2The current implementation does not update the reference model; doing so would require online estimation of Q0, and preliminary experiments suggest it may improve pass@1 and pass@10.
Source: §6 · p. 9
Abstract
from arXiv · showhide
RL-based post-training of language models is almost exclusively done using on-policy methods such as PPO. These methods cannot learn from arbitrary sequences such as those produced earlier in training, in earlier runs, by human experts or other policies, or by decoding and exploration methods. This results in severe sample inefficiency and exploration difficulties, as well as a potential loss of diversity in the policy responses. Moreover, asynchronous PPO implementations require frequent and costly model transfers, and typically use value models which require a large amount of memory. In this paper we introduce Soft Policy Optimization (SPO), a simple, scalable and principled Soft RL method for sequence model policies that can learn from arbitrary online and offline trajectories and does not require a separate value model. In experiments on code contests, we shows that SPO outperforms PPO on pass@10, is significantly faster and more memory efficient, is able to benefit from off-policy data, enjoys improved stability, and learns more diverse (i.e. soft) policies.
1. Introduction
On-policy RL repeatedly rediscovers good behavior and discards rollouts, limiting reuse of hard-won solutions and exploration data. SPO is an asynchronous off-policy soft RL method that uses online and offline experience, with experiments reporting better pass@10 than PPO and faster training.
1. Introduction
- On hard prompts, finding a correct response can take thousands of rollouts, which are then discarded after a small clipped update.Standard sampling is required for unbiased gradients and convergence guarantees, while alternative decoding or exploration produces off-policy trajectories.Source: §1 · p. 1
- SPO is an asynchronous online off-policy soft RL method that leverages both online and offline data to learn high-entropy, high-return policies.Its Cumulative Q-Parameterization expresses token-level soft action values using policy and reference-model log-probabilities.Source: §1 · p. 1
- Cumulative Q-parameterization unifies policy and value functions, saving memory and satisfying soft Bellman-consistency and path-consistency by construction except at reward-bearing tokens.This makes Q-learning and path-consistency losses unnecessary at non-terminal tokens; the paper derives off-policy losses equivalent to terminal Q-value regression or full-sequence path-consistency learning.Source: §1 · p. 1
- If either the policy or value function is optimal, the other is optimal too, so their losses can be combined on online and offline trajectories with a shared optimum.The paper proves this optimality relationship.Source: §1 · p. 1
- SPO consistently outperforms PPO on pass@10, and off-policy learning yields speedups of about 85% compared to PPO.The experiments also report that offline data speeds learning and improves results; an ablation improved results by regressing Q-values toward Monte-Carlo targets for selected Pivotal Token Search tokens.Source: §1 · p. 2
2. Theoretical Results
The theory formulates KL-regularized Soft RL and derives token-level policy and value relations. Its cumulative Q-parameterization makes policy and value learning share an optimum while satisfying consistency relations by construction.
2.1. Setup & Basic Results
- The setup uses terminal rewards in {0, -1} and KL-regularized reward maximization; its soft optimum can preserve the reference policy’s diversity among successful responses.The paper frames learning a high-entropy policy as more aligned with generative modeling than hard RL’s deterministic policy.Source: §2.1 · p. 2
2.2. Token-level Policy & Value Functions
- Token-level optimal actions are expressed through the soft Q-value or advantage of the reference policy, linking each token choice to the sequence’s success-conditioned value.The setup defines token-level values for partial responses.Source: §2.2 · pp. 2–3
2.3. Path- & Bellman Consistency
- Path consistency relates Q-values to policy log-probability differences along a trajectory, while Bellman consistency relates a token’s Q-value to the reference-policy-weighted values of next tokens.The presented Bellman relation has no intermediate reward because the setup uses terminal rewards.Source: §2.3 · p. 3
2.4. The Cumulative Q-Parameterization
- The cumulative Q-parameterization maps the initial value and policy log-probabilities to token-level Q-values, so optimal policy and value functions share the same optimum.The initial prompt value is estimated using Monte Carlo sampling from the reference policy.Source: §2.4 · p. 3
- The parameterization satisfies Bellman and path consistency internally for any policy, eliminating those losses at nonterminal tokens; consistency with observed terminal reward still must be learned.The theorem concerns internal Q-value consistency and does not guarantee that the terminal Q-value equals the external reward.Source: §2.4 · p. 4
3. Soft Policy Optimization
SPO learns from online and offline trajectories using a cumulative Q-function derived from the initial prompt value and token-level policy log-probability differences. The section develops terminal-reward learning objectives, Monte Carlo targets, and an asynchronous framework for combining data sources and training losses.
3. Soft Policy Optimization
- SPO is a hybrid online off-policy method that can learn from on-policy trajectories and arbitrary offline data using a cumulative Q-parameterization.The Q-function combines a sampled initial prompt value with cumulative token-level log-probability differences between the policy and reference model.Source: §3 · p. 4
3.1. Learning Objectives
- Learning objectives target consistency between the cumulative terminal Q-value and observed reward, with alternative off-policy losses derived by rearranging that condition.The section compares loss choices and objectives, Monte Carlo targets, and policy gradients.Source: §3.1 · p. 4
3.1.1. LOSS FUNCTIONS
- Squared error and binary cross-entropy are candidate losses, while sigmoid warping applies an injective transform to both sides of an objective equation.Clipping handles predictions or targets above zero in the cross-entropy setting.Source: §3.1.1 · p. 5
3.1.2. OFF-POLICY OBJECTIVES
- The off-policy objectives include terminal Q regression and advantage regression; nonterminal Q regression uses reverse-Q targets adjusted for the quality of later actions.With detached reverse-Q targets, gradients flow into the predicted Q-value at each token; the text notes that cross-entropy errors differ across terms.Source: §3.1.2 · p. 5
3.1.3. MONTE-CARLO TARGETS
- Monte Carlo estimates of soft values can be biased by the logarithm, yet cross-entropy yields an unbiased loss when exponentiated value targets are unbiased.Pivotal Token Search identifies tokens where success probability changes significantly and supplies sparse estimates for Q-value targets.Source: §3.1.3 · p. 5
3.1.4. PROXIMAL POLICY OPTIMIZATION
- The PPO baseline uses GAE from a learned value model and a clipped loss; asynchronous stale-policy rollouts receive importance weights to correct gradients.The correction uses policy probabilities evaluated on workers and weights the trainer’s loss by the ratio of current to rollout-policy probabilities.Source: §3.1.4 · pp. 5–6
3.2. Asynchronous On- & Off-policy RL Framework
- The asynchronous framework separates rollout workers from trainers, which can mix worker trajectories with offline data and select losses by data source.Workers can use alternative decoding methods; trainers periodically send updated weights, and low update intervals were found stable for on-policy losses.Source: §3.2 · p. 6
4. Related Work
Related methods overlap with SPO in soft-RL objectives, consistency learning, or value-model removal, but differ in how they learn and which data they use. The paper positions SPO’s shared policy-value parameterization and off-policy learning as distinguishing features.
Related work
- SPO’s off-policy losses can be viewed as Q-regression or whole-trajectory path-consistency losses, with consistency satisfied on sub-trajectories rather than learned through separate interval losses.Unlike path-consistency learning, SPO learns policy and value through a shared parameterization.Source: §4 · p. 6
- Unlike Soft Q-Learning and transformer Q-learning methods, SPO uses no Bellman backups because its Q-parameterization guarantees internal Bellman consistency.The cited transformer methods need to learn internal Bellman consistency.Source: §4 · p. 6
- DPO requires preference pairs and DRO is offline-only, while SPO directly uses its policy-value representation; GRPO and VinePPO remove the value model but cannot use off-policy data.The paper also contrasts process reward models, which accelerate policy gradients rather than serving directly as policy and value functions.Source: §4 · p. 6
5. Experiments & Results
The experiments compare SPO with PPO on code-generation performance and test whether SPO can benefit from prior experience. Ablations examine loss functions, non-terminal targets, and Monte Carlo targets.
5. Experiments & Results
- The experiments ask whether SPO can benefit from previous experience and learn a better, more diverse policy than PPO.The evaluation uses CodeContests and TACO, with pass@10 measuring whether at least one of ten sampled solutions passes all tests.Source: §5 · p. 7
- SPO’s pass@10 steadily improves during training while PPO’s declines somewhat, and mixed online/offline SPO outperforms both pure-online SPO and PPO.The half-online run remains stable while using 50% offline samples, reducing sampling cost by half.Source: Figure 1 · p. 7, §5 · p. 7
- PPO achieves 8.4 pass@1, versus 6.3 for SPO half-online and 6.0 for online SPO.The PPO baseline’s pass@1 improvement cooccurs with a modest decline in pass@10.Source: §5 · p. 7
5.1. Ablations and Extensions
- Terminal Q-regression with squared loss improves faster than terminal Q-regression with cross-entropy loss or advantage regression with sigmoid warping and cross-entropy loss.The authors report that the loss choice significantly affects the speed of improvement.Source: §5.1 · p. 8
- For non-terminal Q-regression with reverse-Q targets, cross-entropy significantly degrades performance, while squared loss has no substantial effect.The comparison tests terminal versus non-terminal regression with both loss types.Source: §5.1 · p. 8
- Non-terminal Q-regression toward Monte Carlo targets at pivotal tokens significantly improves performance, whereas applying terminal loss to the search rollouts performs like the baseline.The Monte Carlo-target condition adds 50,000 trajectories, corresponding to fewer than 1,600 steps per epoch.Source: §5.1 · p. 8, Figure 2 · p. 8
6. Discussion
The discussion links SPO’s performance to preserving response diversity and describes the cumulative Q-parameterization’s potential. It also notes limits of the fixed reference model and the current offline-data evaluation.
6. Discussion
- PPO rapidly favors responses already probable under the reference policy, while SPO continues improving both pass@1 and pass@10; offline low-probability solutions further enhance this effect.The discussion frames discovering all good responses, rather than only one, as the goal of Soft RL.Source: §6 · pp. 8–9
- Cumulative Q-parameterization unifies policy and value functions, and Monte Carlo targets improved learning and final performance in experiments that remain limited in scale.Combining SPO with a policy-gradient loss remains unexplored.Source: §6 · p. 9
- The current implementation keeps the reference model fixed; updating it would require online Q0 estimation and is considered likely to improve pass@1 and pass@10.The authors cite preliminary experiments using fixed bQ0 as support for the expected improvement.Source: §6 · p. 9
- The full benefit of offline data is expected to emerge as the dataset grows across experiments, although this cumulative procedure becomes hard to reproduce and amortizes inference cost.Source: §6 · p. 9
7. Conclusion
The conclusion presents SPO as a hybrid online off-policy method based on cumulative Q-parameterization. Code-contest experiments show it can use diverse off-policy data, preserve policy diversity, improve pass@10, and reduce training costs.
7. Conclusion
- SPO uses cumulative Q-parameterization to learn from diverse, highly off-policy data and preserve policy diversity, yielding improved pass@10 over PPO in code-contest experiments.The conclusion attributes greater scalability and faster training to fewer model transfers, lower memory use, and reduced inference compute from offline data.Source: §7 · p. 9