Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

• Conventional reasoning post-training separates positive offline demonstrations for SFT from model-generated rollouts for RLVR, excluding negative demonstrations and retiring earlier data.The passage also notes that rollout batches are replaced even when their verifier rewards remain informative.Source: §1 · p. 2
• Fig. 1 reports that SFT→DAPO has a sharper projected path (ratio 1.2207, turning 368.75°) and lower-reward endpoint than CBPO-Unified (1.0205, 58.20°).Source: §1 · p. 2
• CBPO-Unified achieves the best avg@32 and pass@32 on all four benchmarks, while each restricted variant improves aggregate avg@32 over its regime-matched baselines.CBPO-Offline uses demonstrations and reference responses; CBPO-OnPolicy uses current rollouts; CBPO-OffPolicy uses reference responses plus current and historical rollouts.Source: §1 · p. 2
• CBPO applies one regression loss to reward-labeled offline demonstrations and on- and off-policy rollouts using a fixed-reference target independent of the behavior policy.The target combines reference-policy log-probability, scaled verifier reward, and a frozen prompt-level normalizer.Source: §1 · p. 2

KL-regularized policy optimization and fixed-reference regression

• CBPO extends fixed-reference regression across behavior policies and retains a frozen prompt-level normalizer for unpaired RLVR scalar rewards.Earlier work connects rewards, reference distributions, and optimal policies (Levine, 2018; Korbak et al., 2022; Rafailov et al., 2023); A⋆-PO regresses on current-policy rollouts after estimating an optimal value from reference-policy samples (Brantley et al., 2025). Preference optimization (Tang et al., 2024; Ji et al., 2024) and relative-reward regression (Gao et al., 2024) instead use within-prompt comparisons.Source: §2 · p. 2

SFT–RL post-training and update interference

• CBPO trains demonstrations and rollouts throughout training with one verifier-reward regression objective, unlike approaches that coordinate separate SFT and RL objectives.SFT and distillation are common cold-start strategies (Shao et al., 2024; DeepSeek-AI, 2025); DeepSeek-R1 combines supervision with RLVR, DAPO applies RL directly to the base model (Shao et al., 2024; DeepSeek-AI, 2025; Yu et al., 2025), and BRIDGE coordinates SFT and RL through joint bilevel updates (Chen et al., 2026).Source: §2 · p. 2

Offline and off-policy RLVR

• CBPO combines positive and negative offline CoT with fresh and historical rollouts under its fixed-reference target.Prior approaches exploit non-current RLVR rollouts through asynchronous collection or learned value functions (Bartoldson et al., 2025; Yuan et al., 2025; Wang et al., 2026), or reuse off-policy samples and jointly optimize on- and off-policy data (Cohen et al., 2025; Tang et al., 2025; Li et al., 2025; Wan et al., 2026).Source: §2 · p. 2

3 PRELIMINARIES

• The setup defines prompt x, completion y, verifier reward r(x, y), and sequence log-probability; a rollout is on-policy when used in its collection round and off-policy when reused later.At round t, πθt denotes the current policy, and a rollout collected at round k is sampled from πθk.Source: §3 · pp. 2–3

3.1 THREE RL DATA REGIMES

• Offline demonstrations are available before optimization, on-policy rollouts are freshly sampled at the current round, and off-policy rollouts were sampled at an earlier round.Offline completions may come from external policies whose identities are unknown.Source: §3.1 · p. 3

3.2 CONVENTIONAL TWO-STAGE REASONING POST-TRAINING

• Conventional reasoning post-training uses cold-start SFT on positive offline demonstrations, then RLVR on generated rollouts, typically leaving offline data unused (Shao et al., 2024; DeepSeek-AI, 2025).Because SFT increases likelihood regardless of reward, incorrect or low-reward demonstrations are typically filtered rather than used as targets.Source: §3.2 · p. 3

4.1 FIXED-REFERENCE REGRESSION TARGET

• The fixed-reference target depends on verifier reward, reference log-probability, and a frozen prompt normalizer—not the behavior policy—so completions from different policies share one regression loss.The behavior policy affects which completions are collected, but not their assigned targets.Source: §4.1 · p. 4

4.2 CBPO-UNIFIED ACROSS THREE DATA REGIMES

• CBPO-Unified initializes replay with rescored offline demonstrations and reference responses, then adds scored current-policy rollouts while retaining earlier rollouts and offline data for training.Each completion record carries its verifier reward and reference log-probability, with a frozen normalizer cached per prompt.Source: §4.2 · p. 4

4.3 SPECIAL FORMS OF CBPO

• CBPO-Offline uses only offline demonstrations and reference responses; CBPO-OnPolicy uses current rollouts, while CBPO-OffPolicy retains current and historical rollouts without offline demonstrations.CBPO-Unified combines offline demonstrations with on- and off-policy rollouts in an evolving buffer.Source: §4.3 · p. 5

4.4 FINITE-SAMPLE NORMALIZER ANALYSIS

• The frozen log-normalizer is downward biased at finite Kref, and the induced gradient bias is O(1/Kref) under the stated boundedness conditions.Replaying reference samples used in the estimate adds a covariance term that remains O(1/Kref) under bounded score gradients; increasing Kref trades normalizer accuracy against sampling and preprocessing cost.Source: §4.4 · p. 5

5.2 PERFORMANCE ACROSS DATA REGIMES

• Restricted CBPO variants improve regime-matched baselines across all three data regimes, with Qwen3-8B aggregate avg@32 gains of 0.87, 0.78, and 0.04 points.At Qwen3-1.7B, CBPO-Offline exceeds SFT by 1.48/1.67 aggregate avg@32/pass@32 points and DPO by 1.27/1.83; CBPO-OnPolicy scores 41.84/63.02, beating its strongest baseline by 0.47/1.04 points. CBPO-OffPolicy improves TBRM by 0.38 aggregate avg@32 points and improves or matches it on six of eight entries.Source: §5.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
• 119.36 GPU hours suffice for Qwen3-1.7B CBPO-Unified, within 2% of DAPO, REINFORCE++, GSPO, and TBRM; at 8B, its 437.44 GPU hours are below non-PPO on-policy baselines and less than half PPO's cost.Figure 3 shows early rapid improvement and leadership on both metrics across all four benchmarks at the final checkpoint; CBPO-Unified continues improving after other rollout methods plateau or fluctuate.Source: §5.2 · p. 6, Figure 3 · p. 7

5.3 SHARED-TARGET OPTIMIZATION AND DATA REUSE

• CBPO-Unified exceeds SFT→DAPO by 2.33/4.44 points overall in avg@32/pass@32 after retaining earlier rollout rounds, leading reported results on all eight benchmark-metric combinations.Source: §5.3 · p. 7
• Adding negative offline trajectories improves CBPO w/o History by 0.33/1.40 aggregate avg@32/pass@32 points, while adding reference responses improves it by a further 0.38/1.65.These variants use the same positive offline trajectories and fresh current-policy rollouts; retaining earlier rollout rounds further improves aggregate performance by 1.49/2.07 points over FIFO-1.Source: §5.3 · pp. 6–7, Table 3 · p. 7
• Full replay achieves the best avg@32 and pass@32 on every benchmark, while extending finite replay from one to 16 rounds raises avg@32 across all four.For rollouts older than 16 policy versions, high-residual replay outperforms age-balanced replay on both metrics across all benchmarks, whereas low-residual replay underperforms it.Source: Table 4 · p. 7, §5.3 · p. 8
• CBPO-Unified's projected path has L/D = 1.0205 and cumulative turn 58.20°, versus 1.2207 and 368.75° for SFT→DAPO, ending in a higher-reward region.The lower path ratio and cumulative turning persist with five and ten PCA components and change only slightly with layer normalization.Source: Table 5 · p. 8, §5.3 · p. 8

5.4 ABLATION STUDIES

• Increasing qneg from 0% to 33.3% improves aggregate avg@32/pass@32 by 1.10/1.89 points and yields the best result on every benchmark and metric.The trend is non-monotonic: moving to 66.7% lowers aggregate scores by 1.37/2.77 points, while negative-only demonstrations lower them by 2.06/3.25 points; incorrect completions provide contrastive supervision but do not replace successful trajectories.Source: Table 6 · p. 8, §5.4 · p. 8
• With a fixed budget of 40 generated responses per prompt, (Kref, M, Rroll) = (24, 10, 16) scores highest at 44.56 avg@32 and 68.72 pass@32.Across Kref values, aggregate avg@32 rises from 42.36 to 45.27, while pass@32 rises overall from 66.52 to 70.15 with a dip at Kref = 24; allocating 32 responses to reference estimation yields 42.26/66.97.Source: Table 7 · p. 9, §5.4 · p. 9

6 CONCLUSION

• CBPO represents offline demonstrations and on- and off-policy rollouts with one regression loss, enabling continued reuse without behavior-policy likelihoods, importance ratios, a critic, or an objective switch.Verifier rewards, fixed-reference log-probabilities, and frozen prompt normalizers provide target fields for successful and unsuccessful completions.Source: §6 · p. 9
• Across two model scales, combining restricted data regimes performs best, and historical replay further improves CBPO.CBPO also improves each restricted regime compared with its alternatives.Source: §6 · p. 9
• CBPO follows a more direct path toward a higher-reward policy than SFT→RLVR, while balanced negative demonstrations and accurate reference normalization strengthen the unified method.Source: §6 · p. 9

7 LIMITATIONS

• Experiments cover mathematical reasoning with Qwen3 models and rule-based verifiers; generalization to other domains, model families, noisy rewards, and offline distribution shifts remains untested.CBPO also incurs an upfront generation cost to estimate the frozen prompt normalizer.Source: §7 · p. 9

ETHICS STATEMENT

• The work collected no new human-subject data, removed confirmed training-question overlaps with held-out evaluation problems, and used artifacts subject to their licenses and terms.The overlap-screening protocol is described in Appendix B.1.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Reproduction details include model scales, data, rewards, budgets, evaluation sets, metrics, verifier, overlap screening, baselines, decoding, hardware, and seed usage.Reproduction requires access to the cited model and dataset artifacts under their respective terms.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

• Generative AI tools assisted with language editing and code generation, editing, and debugging; authors reviewed and verified all AI-assisted manuscript edits and code changes.Source: AI USE STATEMENT · p. 10

APPENDIX

• The appendix provides theoretical analysis, algorithmic details, data and verifier specifications, training configurations, additional results, diagnostic protocols, and qualitative examples.Source: APPENDIX · p. 15

A.1 BEHAVIOR-SOURCE INVARIANCE OF THE CBPO TARGET

• CBPO’s target and loss omit the generating behavior policy, so identical logged fields yield the same per-sample loss and gradient across behavior sources.The buffer-sampling distribution still determines each record’s weight in the aggregate objective.Source: §A.1 · p. 15

A.2 FINITE-SAMPLE BIAS OF THE FROZEN PROMPT NORMALIZER

• The direct Monte Carlo estimator of Zref(x) is unbiased, but concavity of the logarithm makes log bZref,N(x) downward biased at finite N.Under the stated bounded-reward and bounded-length conditions, the bias is bounded and vanishes as the reference sample count increases.Source: §A.2 · p. 15
• With a fixed collection distribution and finite first-moment score-gradient assumption, the gradient bias from the Monte Carlo log-normalizer is bounded and vanishes as N →∞.The bound assumes the partition-estimation samples are independent of the training record.Source: §A.2 · pp. 16–17
• Reusing reference samples for both normalizer estimation and training induces an O(1/N) dependence term, while leave-one-out estimation removes that covariance term.The remaining bias is the log-normalizer bias for N −1 samples.Source: §A.2 · p. 17
• In the implementation, verifier rewards lie in {0, 0.1, 1.0} ⊂[0, 1], and at β = 0.2 the weight er/β lies in [1, e5].Finite per-completion score gradients and finite replay support ensure the stated first-moment condition at fixed θ.Source: §A.2 · p. 18

A.3 POPULATION CONSISTENCY AND REGIME-DEPENDENT APPROXIMATION

• Under an exact normalizer, realizability, and support coverage, every regime—and any source mixture with combined support coverage—has the same population-optimal policy π⋆.The population risk has minimum zero, and every global minimizer equals π⋆ for ρ-almost every prompt.Source: §A.3 · p. 18
• Without realizability, exact population regression chooses a qg-weighted best fit to log π⋆, so changing the regime’s record distribution can change the selected fit.The section frames subsequent bounds around the resulting objective gap, finite records, estimated targets, and incomplete optimization.Source: §A.3 · p. 19

A.4 FROM TRAINING-RECORD ERROR TO THE KL-REGULARIZED OBJECTIVE

• For any C ≥1, transfer is bounded by C times the training-distribution regression error plus βBκg(C), covering uncovered pairs and density ratios above C.When pg ≤Cqg everywhere, κg(C) = 0; support coverage alone yields the identifiability result in Proposition 1.Source: §A.4 · p. 19
• The regression-error decomposition comprises policy-class approximation, frozen-target error, population-to-archive sampling error, and empirical optimization error.If achieved squared empirical loss is within ηopt of its infimum, εopt,g ≤√ηopt.Source: §A.4 · p. 19
• The regression decomposition is deterministic for the realized data and requires no independence among reference samples, the archive, and the fitted policy.It therefore also covers reuse of reference responses for training.Source: §A.4 · pp. 19–20
• The objective-gap bound remains valid for adaptive collection because its quantities use the same realized archive and regime distribution.With exact targets, exact population optimization, realizability, and the support condition, Proposition 1 gives equality with π⋆.Source: §A.4 · p. 20
• The objective-gap bound separates normalizer estimation, finite-record sampling, function approximation, and optimization, with distribution shift controlled by C and residual coverage mass κg(C).It quantifies how a regime-specific fit can differ from the optimum under the specified objective distribution ρ.Source: §A.4 · p. 20

A.5 FINITE-REFERENCE TARGET ERROR AND FINITE-RECORD BOUNDS

• With probability at least 1−δZ, bounded reference weights yield a uniform log-normalizer error bound over fixed prompts, valid for any supported archive, even one selected using reference samples.The bound assumes 0 < a ≤ W ≤ b < ∞ and uses K independent reference draws.Source: §A.5 · p. 20
• For adaptively collected records, concentration is uniform over a fixed bounded policy class with a finite sup-norm net, so the fitted policy may depend on the collected records.The collection law may depend on earlier policy updates, prompts, and samples.Source: §A.5 · pp. 21–22
• The finite-sample objective bound combines normalizer and record-sampling errors for any fitted policy, without requiring sample splitting when reference responses also serve as training records.It holds under the stated conditions of Propositions 2 and 3, with collected prompts in the fixed prompt set.Source: §A.5 · p. 22

A.6 SOURCE MIXTURES, FIXED ARCHIVES, AND REPLAY

• CBPO-Offline uses offline and reference records, CBPO-OnPolicy uses latest rollouts, CBPO-OffPolicy uses reference and current and historical rollouts, and CBPO-Unified uses all four source groups.OnPolicy target-construction reference samples are excluded from its training mixture.Source: §A.6 · p. 22
• When a population reference component has positive mixture weight and the stated boundedness conditions hold, it gives pg ≤ Cqg with C = eBb/(aαg,ref) and κg(C) = 0.This coverage claim applies to the population reference component; its finite archive is handled by the sampling term.Source: §A.6 · p. 22
• Treating an offline archive as fixed adds no concentration event, while repeated replay optimizes the same empirical risk; collecting new responses changes the sampling and coverage terms.The fixed-archive bound can be combined with fresh reference and rollout records.Source: §A.6 · p. 23

A.7 EXTENSION TO TRUNCATED REFERENCE SAMPLING

• With truncated reference sampling, total normalizer error separates finite-sample error around the decoded law from decoding discrepancy; increasing K removes the former, while total bias need not be negative.The uniform archive bound is εZ,g ≤ uMC + εdec,g.Source: §A.7 · pp. 23–24
• At fixed parameters, a leave-one-out normalizer for K > 1 removes reuse covariance from the gradient estimate, but its expectation still contains ddec.The bound for reused reference responses accounts for both the shared-sample contribution and the quadratic remainder.Source: §A.7 · p. 25

Artifact sources and offline-record preparation

• RLVR-Math-16k draws on Math12k and NuminaMath-1.5-RL-Verifiable; fixed DPSK-CoT trajectories come from DeepSeek-R1-Distill-Qwen-32B.The teacher generated 32 candidates per prompt, with at least three reward-1 candidates retained per prompt; response generation used a 2,048-token maximum, and source and collection-round metadata support diagnostics but do not enter the CBPO loss.Source: B ADDITIONAL EXPERIMENTAL SETUP · pp. 25–26

Prompt construction

• RLVR-Math-16k prompts pair a common system message with a row-specific math problem and are rendered with the Qwen tokenizer chat template, enabling thinking when supported.Table B.1 shows representative user messages and verifier targets from the training split.Source: B ADDITIONAL EXPERIMENTAL SETUP · p. 26, Table B.1 · p. 27

Reward implementation

• The intermediate 0.1 reward is for extracting a boxed final answer, while verifier-confirmed equivalence to normalized ground truth supplies the remaining 0.9.The format component is used only to encourage a parseable final answer.Source: B ADDITIONAL EXPERIMENTAL SETUP · p. 26

Offline DPO

• Offline DPO matches CBPO-Offline’s policies, prompts, optimizer-step budget, and evaluation protocol, using a sigmoid loss with βDPO = 0.1 and a frozen initial reference.It uses full-parameter optimization, pairs responses across label groups, omits prompts without both labels, excludes overlength pairs, and updates on 256 pairs; AdamW uses weight decay 0.1, 10-step warmup, and gradient-norm limit 1.0. The off-policy TBRM baseline uses n = 4; baseline groups and shared controls follow Sec. 5.1.Source: B ADDITIONAL EXPERIMENTAL SETUP · pp. 26–27

On-policy baselines

• PPO, DAPO, REINFORCE++, and GSPO use distinct advantage, clipping, and loss-aggregation settings, and all four disable reward-side and actor-loss KL penalties.PPO uses GAE with λ = 0.999; DAPO uses clipping widths 0.2 and 0.28; REINFORCE++ uses γ = 1 without a critic; and GSPO uses sequence-level ratios with clipping widths 3 × 10^-4 and 4 × 10^-4. Their actor optimizers use weight decay 0.1 and a 10-step warmup.Source: B ADDITIONAL EXPERIMENTAL SETUP · p. 27

Cooperative SFT/RL

• BRIDGE is matched to CBPO-Unified on the initial student, prompts, positive offline trajectories, current-policy generation budget, and student optimizer-step budget, but CBPO-Unified additionally uses negative offline trajectories, reference responses, and historical replay.BRIDGE uses a 0.5 supervised/RL mixing weight and shares student-generated rollouts between the student and auxiliary model; cost accounting includes reference preprocessing and scoring, rollout generation, and all parameter updates.Source: B ADDITIONAL EXPERIMENTAL SETUP · p. 27

B.1 TRAINING–EVALUATION DECONTAMINATION

• Exact matching strips only leading and trailing whitespace; approximate matching uses normalized question text and character 3–5-gram TF-IDF, with cosine similarity of at least 0.80.Notation normalization standardizes selected Unicode and LaTeX forms while retaining numerical values, variable case, and operator distinctions; it does not test arbitrary algebraic equivalence.Source: §B.1 · p. 28
• Semantic retrieval uses Sentence-BERT with all-MiniLM-L6-v2 embeddings and a cosine threshold of 0.85; candidates are assessed for mathematical equivalence.Questions are embedded using 384-dimensional chunk embeddings, with long questions split without truncation; content comparison distinguishes the same problem from changed requests, different subquestions, and different problems.Source: §B.1 · p. 28
• Beyond the 14 confirmed matches, screening found 2 changed-answer pairs and 15 shared-setting pairs with different subquestions; the remaining 120 candidates were different problems, all in MATH500.The changed-answer cases involved two evaluation questions; the shared-setting cases involved eight. No candidates were retrieved for the other three evaluation groups.Source: §B.1 · pp. 28–29, Table B.4 · p. 29
• 151 candidate pairs involved 97 evaluation questions and 149 training texts; 14 pairs involving 13 evaluation questions were confirmed as same-problem matches.One evaluation question matched two differently worded training versions.Source: §B.1 · p. 28, Table B.3 · p. 29
• Excluding the 14 same-problem training questions reduced the prompt pool from 16,268 to 16,254 questions, while the evaluation set remained at 842.The evaluation set comprised 30 AIME25, 272 MinervaMath, 40 AMC23, and 500 MATH500 questions.Source: §B.1 · pp. 28–29, Table B.5 · p. 29

C HYPERPARAMETERS AND TRAINING-SYSTEM SETTINGS

• The Qwen3-1.7B CBPO variants follow the data-source restrictions in Sec. 4.3 while keeping the shared target, generation settings, and optimization budget aligned.Table C.1 records the core configuration.Source: C HYPERPARAMETERS AND TRAINING-SYSTEM… · p. 29, Table C.1 · p. 30
• The response budget uses 16,254 prompts, with 24 reference responses and 16 current-policy responses per prompt, totaling 40.The setup uses six reference-collection epochs and four current-policy epochs, with four responses per prompt per epoch.Source: C HYPERPARAMETERS AND TRAINING-SYSTEM… · p. 29

D.1 HETEROGENEOUS OFFLINE SOURCES

• Mixing three teacher sources gives CBPO-Unified its highest aggregate scores, improving avg@32/pass@32 by 2.18/9.15 points over R1-Qwen-1.5B and 0.67/1.75 points over R1-Qwen-32B.The comparison uses a common retained prompt set of 8,724 prompts; the mixed condition combines the teacher-specific datasets.Source: §D.1 · p. 30, Table D.1 · p. 31
• CBPO-Unified outperforms BRIDGE under all four source conditions, with aggregate margins of 1.73–2.43 points in avg@32 and 4.61–5.81 points in pass@32.Under the mixed condition, the margins are 2.05/5.81 points in avg@32/pass@32.Source: §D.1 · p. 30, Table D.1 · p. 31

D.2 PROMPT-LEVEL REFERENCE NORMALIZER DIAGNOSTICS

• At most one correct reference response occurs for 36.69% of retained prompts, while 18.99% have 24 correct responses.Prompts are grouped by the number of reward-1 completions among 24 reference responses.Source: §D.2 · p. 31
• Under independent reference-response sampling, the diagnostic constructs pointwise exact trinomial bounds on absolute log-normalizer error using test inversion and Buehler limits.The stated coverage guarantee is pointwise in prompt and sample size before grouping by observed reference success.Source: §D.2 · pp. 31–32, Table D.3 · p. 32
• The overall mean bound falls from 2.2693 at K = 8 to 1.7209 at K = 16, 1.4560 at K = 24, and 1.3044 at K = 32.At K = 24, the mean bound is 0.1526 for the all-correct group and 2.9156 for the group with at most one correct response.Source: Table D.3 · p. 32, §D.2 · p. 32
• Across all five groups, gains rose with K but slowed; increasing K from 24 to 32 costs one-third more reference responses for only 0–1 avg@32 points, motivating Kref = 24.This choice is consistent with A⋆-PO’s reported MATH500 plateau around N = 8 reference responses per prompt (Brantley et al., 2025).Source: §D.2 · p. 32

D.3 NEGATIVE-TRAJECTORY COUNT AND TYPE

• Adding 1 negative improved aggregate avg@32/pass@32 by 1.40/2.03 points, while adding a second lowered them by 0.45/1.70 points.The comparison held two positive offline trajectories per prompt fixed and used nested negative subsets.Source: §D.3 · p. 33, Table D.4 · p. 33
• Negative-type comparisons added one qualifying response per prompt to the same two positive offline trajectories, distinguishing verifier rewards and manually reviewed response content.Near-correct responses had largely valid derivations with a localized error; clearly invalid responses had unsupported or incoherent derivations that did not yield a valid solution.Source: §D.3 · p. 33
• Reward-0.1 negatives raised aggregate avg@32/pass@32 by 1.64/2.22 points, near-correct negatives gained 2.02/3.92 points, and clearly invalid negatives lowered them by 0.52/4.03 points.Reward-0 negatives changed the metrics by 0.03/-0.10 points; near-correct negatives improved both metrics on all four benchmarks, while clearly invalid negatives reduced both on all four.Source: §D.3 · pp. 33–34, Table D.5 · p. 34
• Across five training seeds, selected negatives included 35.48% near-correct responses, 40.32% reward-0.1 responses, and 16.13% in both categories.This mixed composition was consistent with the default one-negative configuration outperforming the no-negative reference but remaining below the all-near-correct condition in the seed-42 comparison.Source: §D.3 · p. 34, Table D.6 · p. 34

D.4 REPLAY-SAMPLING COMPARISONS

• Four Qwen3-1.7B CBPO-Unified runs varied only replay-sampling rules under matched initialization, settings, generation and optimizer-step budgets.Current-policy rollouts entered each run’s evolving buffer under a common collection schedule; records were drawn without replacement within minibatches and resampled from the current buffer at each update.Source: §D.4 · p. 34, Table D.7 · p. 35
• Age-balanced sampling allocates rollout records as evenly as possible across four policy-version age groups and samples uniformly within each group.Age-group membership is recomputed at every update.Source: §D.4 · p. 34
• Source-balanced sampling divides each minibatch as evenly as possible among nonempty offline, reference-response, latest-rollout, and earlier-rollout groups.Group quotas differ by at most one record, with uniform sampling within each group.Source: §D.4 · p. 34
• Prompt-balanced sampling selects 2,048 distinct prompts uniformly and draws one archived response per prompt, unlike response-balanced sampling, which draws uniformly from the buffer.Prompt-balanced sampling gives every prompt the same expected training share; response-balanced sampling makes a prompt’s expected share proportional to its eligible responses.Source: §D.4 · pp. 34–35

D.5 WHICH HISTORICAL ROLLOUTS REMAIN USEFUL?

• For rollouts older than 16 policy versions, High reaches avg@32/pass@32 means of 45.16/70.61 versus CBPO-Unified’s 44.49/68.67 (+0.67/+1.94 percentage points).Historical records are split into Low and High residual groups at the median absolute pre-update residual among historical records replayed at each optimizer step.Source: Table D.8 · p. 35, §D.5 · p. 35
• Across benchmarks, avg@32 rises with replay age in High and falls in Low; reward-1 share rises from 20.19% to 74.64% in High but falls from 86.57% to 38.23% in Low.Under squared loss, larger absolute residuals produce stronger gradients with respect to sequence log-probability, supporting the use of residuals to identify older responses that can counter forgetting.Source: §D.5 · pp. 35–36

D.6 CHECKPOINT-WISE TURNING ALONG THE OPTIMIZATION PATHS

• CBPO-Unified maintains small local turns through most training, while SFT→DAPO’s largest turn is approximately 107° at checkpoint 6, where its objective changes.The peak is the largest single contribution to the two-stage trajectory’s cumulative turn.Source: Figure D.2 · p. 36, §D.6 · p. 36
• Across 2, 5, and 10 PCA components, layer normalization changes L/D by at most 0.0060 and cumulative turning by at most 2.28°.The appendix also shows trajectories with five and ten PCA components; Table 5 reports path geometry for the 2-, 5-, and 10-component projections.Source: §D.6 · pp. 36–37, Figure D.3 · p. 37, Figure D.4 · p. 37

D.7 REWARD-TEMPERATURE SENSITIVITY

• Table D.11 is titled as a CBPO-Unified reward-temperature sensitivity study, with β values 0.1, 0.2, 0.4, and 0.8 specified in its caption.Source: Table D.11 · p. 38

E QUALITATIVE EXAMPLES

• The qualitative examples comprise two groups of matched-prompt responses illustrating objective continuity and historical replay.Each response preserves its original content and formatting and is paired with a reward, training stage or collection round, response index, and prompt hash; reward 1 indicates a verifier-matched boxed answer.Source: E QUALITATIVE EXAMPLES · p. 38

EXAMPLE 1: PASCAL’S TRIANGLE

• Verifier reward 1 is reported for both the offline SFT trajectory and CBPO-Unified’s answer 45, while the SFT→DAPO response answers 5 with reward 0.1.The section introduction identifies these as responses to the same prompt; another SFT→DAPO response answers 5 with verifier reward 0.Source: §E.1 · p. 39, EXAMPLE 1: PASCAL’S TRIANGLE · pp. 39, 40, 43

EXAMPLE 2: MODE OF A STEM-AND-LEAF PLOT

• The shown stem-and-leaf plot example identifies 89 as the mode of the test scores.The plot is provided in the prompt, and the response states that 89 appears four times and is the mode.Source: EXAMPLE 2: MODE OF A STEM-AND-LEAF PLOT · pp. 44–45
• For the stem-and-leaf prompt, a CBPO-Unified response identifies 89 as the mode, while an SFT→DAPO response concludes 99.The CBPO-Unified response counts 89 four times; the SFT→DAPO response instead counts 99 four times.Source: EXAMPLE 2: MODE OF A STEM-AND-LEAF PLOT · pp. 46, 48, 51, 52

E.2 HISTORICAL REPLAY

• Archived CBPO-Unified responses are displayed in increasing order of collection round.Source: §E.2 · p. 53

EXAMPLE 1: SIBLING COUNTING

• For the sibling-counting prompt, a model response calculates Harriet’s 2 sisters and 5 brothers and gives their product as 10.Source: EXAMPLE 1: SIBLING COUNTING · pp. 53–54
• Sibling-counting responses differ: a verifier-reward-0.1 response gives 10, while a verifier-reward-1 response gives 12.The first computes 2 × 5; the second computes 2 × 6.Source: EXAMPLE 1: SIBLING COUNTING · pp. 55–57

EXAMPLE 2: LADDER DISPLACEMENT

• For the ladder prompt, verifier-reward-0 and verifier-reward-0.1 responses both report 15 ft after the top slips 4 feet.The reward-0 response calculates a new horizontal distance of 15 ft; the prompt asks how far the foot will slide.Source: EXAMPLE 2: LADDER DISPLACEMENT · pp. 59–61
• The initial height is 24 feet for a 25-foot ladder whose foot starts 7 feet from the building.This follows from the Pythagorean theorem: h_1 = sqrt(25^2 - 7^2) = 24 feet.Source: EXAMPLE 2: LADDER DISPLACEMENT · p. 64
• After the top slips 4 feet, the ladder’s height becomes 20 feet.The calculation is h_2 = 24 - 4 = 20 feet.Source: EXAMPLE 2: LADDER DISPLACEMENT · p. 64
• With the ladder still 25 feet long and its height at 20 feet, the new distance from the building is 15 feet.The passage applies the Pythagorean theorem to obtain d = sqrt(25^2 - 20^2) = 15 feet.Source: EXAMPLE 2: LADDER DISPLACEMENT · p. 64
• The foot of the ladder slides 8 feet, ending 15 feet from the building.The passage distinguishes the new 15-foot distance from the 8-foot displacement from the initial position.Source: EXAMPLE 2: LADDER DISPLACEMENT · p. 64