COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning

Advantage staleness and two-channel analysis

• Exact bias and variance decompositions show multiplicative coupling between policy-weight and advantage-estimation errors, while squared policy weights amplify conditional advantage uncertainty in gradient variance.The analysis motivates coordinating the channels but does not prescribe an optimal coordination rule.Source: §1 · p. 2

Coupled off-policy correction

• COPC combines token-level ratio masking with two-sided clipped-ratio weighting of TD residuals to produce corrected returns for critic training and actor advantage estimation.Separate channel parameters enable joint calibration.Source: §1 · p. 2

Empirical validation of coordinated correction

• Joint parameter sweeps across staleness levels show that changing one correction strength can improve or degrade performance depending on the other, while strong performance spans a broad parameter region.The experiments use the same parameter pair across tasks without task-specific retuning.Source: §1 · p. 2

Performance, stability, and efficiency

• 2.03 points in avg@32 on tool-integrated mathematical reasoning and 2.77 points in greedy-decoding accuracy on search over the strongest reported asynchronous baseline are COPC’s reported margins.COPC achieves the highest reported performance in both settings.Source: §1 · p. 2
• COPC remains stable throughout search training while most evaluated asynchronous baselines collapse late, and its gains persist at 64-step policy staleness.It adds minimal step-time overhead over asynchronous PPO and retains a 1.7× speedup over synchronous PPO.Source: §1 · pp. 2–3

Asynchronous RL for LLM post-training

• Asynchronous actor–learner systems improve throughput by decoupling data collection from optimization, at the cost of behavior–learner policy mismatch [Espeholt et al., 2018].LLM post-training systems extend this paradigm through distributed rollout generation and asynchronous updates [Noukhovitch et al., 2025, Fu et al., 2025, Wu et al., 2025].Source: §2 · p. 3

Policy-side stabilization in off-policy LLM optimization

• Recent LLM-RL methods primarily stabilize off-policy actor updates with importance-ratio control and related mechanisms, whereas COPC also corrects continuation-dependent returns and advantages.Examples include AReaL and SAO [Fu et al., 2025, Hou et al., 2026], VCPO [Huang et al., 2026], and M2PO [Zheng et al., 2025].Source: §2 · p. 3

Off-policy return and advantage correction

• Retrace and V-trace correct multi-step returns using importance weighting or truncation [Munos et al., 2016, Espeholt et al., 2018], while COPC uses separate advantage-side correction and policy-side masking parameters.COPC studies interaction between these channels, unlike the return-correction methods summarized here.Source: §2 · p. 3

3.1 Asynchronous RL Training

• The learner updates an autoregressive policy using responses generated by a stale behavior policy, and the PPO-style objective combines token-level importance ratios with advantage estimates.The ratio measures learner–behavior policy mismatch at each token, and the advantage estimate is used by the actor.Source: §3.1 · p. 3

3.2 Why Advantage-Side Correction Is Needed

• With no advantage-side correction, the expected corrected-return operator remains tied to the behavior policy; exact ratios shift its fixed point to the learner policy, while clipping trades some of that correction for variance control.COPC without corrected returns and advantages, using standard GAE instead, underperforms COPC on both tool-integrated mathematical reasoning and search.Source: §3.2 · p. 4

3.3 Two-Channel Correction Setup

• The two-channel setup separates controlled actor weights from estimates that account for stale behavior-policy continuations, parameterizing their corrections independently for studying their interaction.Policy-side masking controls the token weight; advantage-side correction replaces the ideal current-policy advantage with a mismatch-adjusted estimate.Source: §3.3 · p. 4

4.1 Bias Coupling

• The bias includes a multiplicative interaction between policy-weight and advantage-estimation errors that can reinforce or offset their direct contributions.Consequently, adjusting either channel’s correction can affect overall bias differently depending on the residual error in the other.Source: §4.1 · p. 5

4.2 Variance Coupling

• Squared policy weights rescale conditional advantage uncertainty in the variance of each token contribution.The full-response decomposition also includes cross-token covariances.Source: §4.2 · p. 5

4.3 Joint Gradient-Estimation Error

• The joint gradient-error decomposition motivates calibrating the two correction channels together because error need not be separable in their parameters.The analysis considers fixed correction settings and a generic group average under the corresponding sampling and advantage-estimation scheme.Source: §4.3 · pp. 5–6, §4 · p. 5
• Across staleness levels 8, 12, and 16, changing one correction parameter can improve or degrade AIME 2025 avg@32 depending on the other parameter.The sweeps support the coordination hypothesis, with preferred configurations following the hypothesized pattern.Source: §4.3 · p. 6, Figure 1 · p. 6

Joint parameterization

• Separate interval controls ηP and ηA set the policy- and advantage-side correction bounds, and the parameters are selected jointly.Smaller η narrows the corresponding interval and represents stronger correction under this convention.Source: §5 · p. 6

Evaluation

• The mathematical-reasoning table reports avg@32, and the search table reports greedy-decoding accuracy; both mark best and second-best results within each backbone group.Their Δ rows report gains of the method over PPO; the supplied captions do not provide cell values.Source: Table 1 · p. 7, Table 2 · p. 7

Actor–critic update

• The actor objective applies a fixed, non-differentiable mask: masked tokens contribute no actor gradient, while retained tokens preserve the importance-ratio gradient path.The masking decision itself is not differentiated.Source: §5 · pp. 6–7
• Corrected returns supply critic targets, while return targets and actor advantages are computed from frozen pre-update quantities and held fixed during optimization.Masked tokens still contribute to return estimation and critic training, and their TD residuals can influence earlier advantages.Source: §5 · p. 7

6.1 Experimental setup

• The experiments compare COPC with asynchronous baselines and a controlled ablation across tool-integrated mathematical reasoning and seven search benchmarks, using shared rollout and evaluation implementations within each task.The ablation COPC w/o RC retains other settings while replacing corrected returns and advantages with standard GAE.Source: §6 · p. 7, §6.1 · p. 8

6.4 Training Stability, Robustness, and Efficiency

• On search, COPC stays stable throughout training while most evaluated asynchronous baselines collapse late; on AIME 2025, it stays stable from s = 8 to s = 64 with comparable final avg@32 performance.The left panel covers search training dynamics; the right covers COPC on AIME 2025 across policy-staleness levels.Source: Figure 2 · p. 8, §6.4 · p. 9

6.2 Joint Parameter Sweeps

• The effect of changing advantage-side correction reverses with policy-side correction across all tested staleness levels: increasing ηA from 0.25 to 0.75 helps at ηP = 0.25 but hurts at ηP = 0.75.A broad range of correction strengths remains competitive, supporting joint calibration without staleness-specific retuning.Source: §6.2 · p. 9

6.3 Main Results and Ablations

• COPC reaches the best result on all five mathematical-reasoning benchmarks, averaging 50.29, and the highest search average accuracy of 44.57, 2.77 points above the strongest asynchronous baseline.The comparisons cover tool-integrated mathematical reasoning and search.Source: §6.3 · p. 9
• Return correction raises performance from 48.11 to 50.29 on mathematical reasoning and from 40.80 to 44.57 on search versus COPC w/o RC.The corrected-return method beats the ablation on all five mathematical-reasoning benchmarks and six of seven search benchmarks.Source: §6.3 · p. 9

6.4 Training Stability, Robustness, and Efficiency

• Across both task settings, COPC adds about 1% step time and reduces throughput by at most 2.5% versus asynchronous PPO, while retaining an approximately 1.7× step-time speedup over synchronous PPO.Table 3 reports training efficiency on tool-integrated mathematical reasoning and search.Source: §6.4 · p. 9, Table 3 · p. 9

7 Conclusion

• COPC combines token-level policy-side masking with advantage-side correction using two-sided clipped-ratio weighting of TD residuals.The correction channels are motivated by advantage staleness from behavior-policy continuations and nonseparable interactions between channel errors.Source: §7 · p. 10
• Across tool-integrated mathematical reasoning and search, COPC improves performance and training stability and remains robust across correction strengths and substantial policy staleness.The conclusion also reports minimal additional overhead and a 1.7× step-time speedup over synchronous PPO.Source: §7 · p. 10
• The paper concludes that asynchronous LLM-RL off-policy correction should be treated as a coupled estimation problem, not solely as policy-ratio control.Source: §7 · p. 10

A Proof of Proposition 1

• The expected corrected-return operator has V eπ as its unique fixed point, established by showing weighted TD residuals have zero conditional expectation and using backward induction for uniqueness.Source: A Proof of Proposition 1 · p. 13
• With a finite time-indexed nonterminal state space, exact iteration of the expected return operator converges to V eπ under Proposition 1's assumptions.Source: A Proof of Proposition 1 · p. 14

B Proofs for Coupling Between Correction Channels

• The coupling-channel proofs fix learner parameters, the value estimator, and behavior-policy sampling, and assume integrability and finite second moments without generally assuming within-response or within-group independence.Source: B Proofs for Coupling Between… · p. 14

B.1 Proof of Proposition 2

• The samplewise update decomposition separates direct policy-side and advantage-side errors from a third term capturing their interaction.The decomposition is formed before expectations, so the proof does not require independence between correction errors or co-sampled responses.Source: §B.1 · p. 15

B.2 Proof of Proposition 3

• The conditional variance decomposition uses conditional measurability and moment assumptions, without requiring independence between the policy-side weight and the advantage estimate.Source: §B.2 · p. 16

B.3 Response-Level Variance Decomposition

• The response-level variance decomposition retains random response-length normalization and captures cross-token dependence and response-length variability without assuming token independence.Source: §B.3 · p. 16

B.4 Proof of Corollary 1

• For the finite-sample update, prompt groups are independent and identically distributed, while arbitrary dependence among responses within each group is retained.Source: §B.4 · p. 17

C.1 Prompt Templates and Reward Design

• Search uses a shaped reward that gives correct answers 1.1 without search or 2.0 with search, while mathematical reasoning uses binary verified correctness without a separate formatting reward or tool-use bonus.Tool calls in mathematical reasoning affect reward only through their contribution to a verifiably correct final answer.Source: §C.1 · p. 18

• Training runs on 8 NVIDIA H200 GPUs, using Qwen3-4B-Instruct for mathematical reasoning and Llama-3.2-3B-Instruct for search.Source: §C.2 · p. 19
• Before RL, Qwen3-4B is fine-tuned on the ReTool SFT corpus and Llama-3.2-3B-Instruct on 2,000 Chain-of-Agents SFT examples.Table 4 summarizes the SFT configurations.Source: §C.2 · p. 19
• Table 5 summarizes default training configurations for both task settings, with method-specific deviations specified separately.Critic settings apply only to methods with a learned value function; prompt templates and reward functions are described in Section C.1.Source: §C.2 · p. 19

C.2.1 Method-specific configurations.

• PPO uses a clipped surrogate objective with standard GAE, while DAPO uses group-relative advantages without a learned critic.DAPO centers and standardizes rewards within prompt groups, dynamically retains groups with nonzero reward variance, and uses token-level loss averaging for mathematical reasoning; overlong reward shaping is disabled.Source: §C.2.1 · p. 19
• Table 4 summarizes SFT configurations, while Table 5 gives shared defaults for the two task settings and asynchronous rollout and optimization.Losses are averaged per response for mathematical reasoning and per token for search, unless otherwise specified.Source: §C.2.1 · p. 19, Table 4 · p. 20, Table 5 · p. 21
• SAO replaces PPO clipping with two-sided importance-ratio masking and samples one response per prompt, yielding 512 distinct prompts per batch.Source: §C.2.1 · p. 20
• AReaL clips the learner-to-proximal ratio and separately weights proximal-to-behavior mismatch, excluding actor-loss tokens with ratios above 5.0.Each batch has 1,024 responses split into two 512-response minibatches; for search, actor learning-rate warmup lasts 25 optimizer updates.Source: §C.2.1 · p. 20
• V-trace uses truncation thresholds of 1 for value correction, trace coefficients, and policy-gradient correction.Its actor uses the V-trace policy-gradient objective without advantage normalization, and its critic uses the corresponding target without PPO-style value clipping.Source: §C.2.1 · p. 20
• COPC uses clipping margins ϵℓ= 0.4 and ϵh = 0.56 with (ηP, ηA) = (0.5, 0.5), giving [0.80, 1.28] for both correction channels.COPC w/o RC sets ηA = 0, recovering standard GAE on the advantage side while retaining ηP = 0.5 and other settings.Source: §C.2.1 · p. 20

C.2.2 Inference configurations.

• Mathematical-reasoning evaluation samples 32 responses per problem and reports avg@32 accuracy on AIME24, AIME25, AIME26, BRUMO25, and BeyondAIME.Sampling uses temperature 0.6, top-p = 1.0, and a 38,912-token budget per response across interaction turns, including tool observations.Source: §C.2.2 · p. 20
• Search evaluation uses greedy decoding and reports normalized exact-match accuracy on seven question-answering benchmarks.It generates one response per question with an 8,192-token maximum response budget; the benchmarks are NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.Source: §C.2.2 · p. 20