PaperlayerView exact revision on arXiv

Source-linked AI summary

Sharpening Tax in Post-Training

Changdae Oh, Qi Zeng, Qi Qi, Andrey Zhmoginov, Deren Lei, Yun He, Hoang Phan, Hangoo Kang, Azalia Mirhoseini, Sharon Li

arXiv:2610.01509v1cs.AIcs.LG

TL;DR

Whether RL post-training creates new agentic capabilities or mainly sharpens existing behavior remains an open question, with prior accuracy–coverage evidence concentrated in math and coding. The paper evaluates base and post-trained models on agentic tasks, measures the resulting test-time scalability cost, and tests an adaptive sampler; base models often achieve greater coverage at scale, while PTGS reduces the tax and improves accuracy and coverage.

  • Problem

    Whether RL post-training creates new agentic capabilities or mainly sharpens base-model behaviors remains unsettled, with prior accuracy–coverage evidence concentrated in math and coding.

    Source: §1 · p. 1

  • Method

    The paper compares base and post-trained models on agentic tasks, defines Sharpening Tax for lost test-time scalability, and uses PTGS to adapt sampling temperature to prompt difficulty.

    Source: §1 · p. 2

  • Results

    Base models often reach more tasks with enough rollouts despite post-trained models' single-shot lead; Sharpening Tax is pervasive, while PTGS lowers it and improves accuracy.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    Post-training should be assessed by solution coverage as well as single-shot accuracy, because sharpening can reduce the base model's potential under test-time compute.

    Source: §8 · p. 13

    The checkpoint analysis is observational, not interventional, because representative models do not disclose their exact pre- and post-training data.

    Source: §9 · p. 13

Abstract

from arXiv · show

An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.

1 Introduction

The paper tests whether RL post-training creates new agentic capabilities or mainly sharpens existing behavior, and examines the resulting coverage trade-off and ways to measure and mitigate it.

1 Introduction

  • The central question is whether RL post-training creates fundamentally new capabilities or amplifies rewarding behaviors already present in the base model.The paper frames the latter possibility as distribution sharpening.Source: §1 · p. 1
  • Prior accuracy–coverage evidence was concentrated in math and coding, while agentic tasks test behaviors such as tool use and long-horizon interaction thought to be learned during post-training.The paper treats agentic tasks as a stress test of whether post-training expands the base model's reasoning boundary.Source: §1 · pp. 1–2
  • Post-trained models lead in single-shot accuracy, but harness-equipped base models can surpass them in solution coverage with enough test-time compute.The analysis also finds that post-training shifts task success probabilities toward always-solved or never-solved extremes, limiting gains from parallel rollouts.Source: §1 · p. 2
  • Sharpening Tax summarizes how much post-training shrinks effective reasoning coverage; across 14 model backbones and three agentic benchmarks, the tax is pervasive and estimable from a handful of rollouts.The metric can also predict future tax at larger rollout budgets and other performance metrics.Source: §1 · p. 2
  • PTGS adapts sampling temperature to task difficulty, smoothing hard prompts and sharpening easy ones; it pays a smaller tax while improving accuracy and coverage over baseline in agentic environments.The paper reports this for common RL algorithms including PPO and GRPO.Source: §1 · p. 2

2 Preliminaries

This section defines the pass@k measures used to evaluate agentic reasoning boundaries and describes the benchmarks and checkpoint pairs used in the study.

2 Preliminaries

  • The evaluation uses pass@1 for single-trial accuracy, pass@k for the probability at least one of k rollouts succeeds, and passk for the probability all k succeed.Dataset-level metrics are averages across tasks, using standard unbiased estimators.Source: §2 · p. 3
  • Three benchmarks require multi-turn tool calling and check intermediate actions and state transitions: BFCL v4 multi-turn base split, WebShop, and ACEBench.This setup avoids counting a lucky final answer after an incorrect trajectory as success.Source: §2 · p. 3
  • The study compares 14 open-source base/post-trained checkpoint pairs from Gemma-4, Ministral-3, Qwen2.5, and Qwen3.5, spanning 3B to 35B effective parameters.The exact training recipes are not fully disclosed, so the paper uses “RL post-trained” loosely; thinking mode is turned off for supported post-trained models.Source: §2 · p. 3

3 The Accuracy-Coverage Tension of Post-trained Agent Policy

The section tests whether findings about post-training’s accuracy–coverage trade-off transfer to agentic tasks, where prior coverage evidence was limited. Harness-equipped base models can surpass post-trained models in solution coverage at larger rollout budgets, while post-training concentrates task success rates at the extremes.

3 The Accuracy-Coverage Tension of Post-trained Agent Policy

  • On WebShop, gemma-4-31B base models exceed 85% pass@128 versus 56% for RL models.Post-trained models lead at small rollout budgets, but base-model pass@K curves cross over in most settings.Source: Figure 2 · p. 4, §3 · p. 4
  • A lightweight, model-agnostic harness improves base models on difficult benchmarks but can hurt post-trained models.The harness combines a simple system prompt with relaxed tool-calling and parsing interfaces; results are averaged across eight base models in Table 1.Source: §3 · p. 4, Table 1 · p. 4
  • On WebShop, the Gemma-4 crossover budget falls from k*>128 for 4B to k*≈3 for 31B.Across Gemma-4 scales, larger models pull the crossover to smaller rollout budgets.Source: Figure 3 · p. 5, §3 · p. 5
  • On WebShop with gemma-4-31B, intermediate-success tasks fell from 87.6% to 30.0%, while always-pass rose from 0.0% to 26.0% and always-fail from 12.4% to 44.0%.Post-training shifts task success rates toward both extremes, not only toward tasks that always succeed.Source: §3 · p. 5, Figure 4 · p. 6
  • Post-trained policies stay comparatively close in pass@K coverage and passK consistency, whereas base-model consistency collapses as its coverage eventually surpasses RL.The trade-off appears across model families; Qwen3.5 sharpens most softly and Gemma-4 most steeply.Source: Figure 5 · p. 6, §3 · p. 6

4 Quantifying the Effect of Post-Training Sharpening

The section introduces Sharpening Tax to summarize how post-training changes test-time scalability, then evaluates its prevalence and predictive value. The tax is positive in most model-benchmark cases at large rollout budgets and can be estimated from a small number of rollouts.

4 Quantifying the Effect of Post-Training Sharpening

  • Sharpening Tax is the base-minus-post-trained deficit in scalability; a positive value means post-training reduces test-time scalability.The paper defines raw and calibrated variants, TaxA(K) and TaxS(K).Source: §4 · p. 7
  • At K=128, TaxS(128)>0 in 36 of 42 model-benchmark combinations; the tax is positive at almost every budget for the largest backbones.For the smallest backbones, it is often negative at k≤8 before rising toward zero or above as the budget grows.Source: §4 · p. 7, Figure 6 · p. 7
  • TaxA rises almost monotonically with rollout budget, while calibrated TaxS may rise or fall because of its accuracy-headroom calibration.A positive tax also means successes rely less on retries, through less recovery of failed first attempts or faster saturation.Source: §4 · p. 7
  • From eight rollouts, TaxS(8) predicts TaxS(32) at ρ=0.85 and is the only early predictor reported to strongly predict all three future metrics.The TaxS(8) estimate and validation use disjoint task subsets.Source: §4 · p. 8, Figure 7 · p. 8

5 How to Balance Sampling Efficiency and Solution Coverage?

The section introduces PTGS as a prompt-adaptive way to temper rollout sampling during RL training. In experiments, it reduces the accuracy-coverage trade-off seen with fixed-temperature PPO and GRPO, while retaining exploratory behavior on difficult prompts.

5 How to Balance Sampling Efficiency and Solution Coverage?

  • A global temperature increase improves pass@K but degrades pass@1, motivating per-prompt adaptive tempering during RL.Source: §5 · p. 8
  • PTGS estimates each prompt’s difficulty from a running success-rate posterior and uses it to set the temperature for the next rollout group.Hard prompts are heated for exploration, while prompts with higher estimated success are cooled for exploitation.Source: §5 · p. 9
  • PTGS reduces the tax and improves both pass@1 and pass@128 over PPO and GRPO baselines in both Sokoban and FrozenLake.Table 2 reports Qwen2.5-7B-Instruct results averaged over five runs.Source: §5 · p. 9, Table 2 · p. 9
  • On FrozenLake, PTGS and fixed-temperature PPO reach the same validation success, but PTGS maintains higher output entropy and greater coverage.Fixed-temperature PPO entropy decreases monotonically, whereas PTGS entropy stays higher and rises repeatedly as it heats prompts the policy keeps failing on.Source: §5 · p. 10, Figure 9 · p. 10

6 Theoretical Analysis

The theoretical analysis links Sharpening Tax to the loss of retry-based scalability when post-training pushes task success probabilities toward zero or one. It also explains how PTGS can improve rollout groups available for learning, conditional on heating increasing a difficult prompt’s success probability.

6 Theoretical Analysis

  • Raw scalability A(K) measures expected failed attempts before a first success within the rollout budget; tasks solved immediately or never solved contribute zero.S(K) restricts this quantity to tasks that fail on the first attempt and normalizes it to [0, 1].Source: §6 · p. 10
  • Sharpening tasks to success probabilities of zero or one removes their retry value, so the raw tax equals the sharpened fraction times base scalability.Under the theorem’s assumptions, this holds for every budget K ≥ 2.Source: §6 · p. 11
  • PTGS cools mastered prompts toward consistent success and heats failing prompts toward an intermediate success probability where retries can recover failures.For cooled prompts, mixed groups shift toward all-success rather than all-failure groups.Source: §6 · p. 11
  • When heating raises a hard prompt’s success probability, groups containing at least one success become more frequent, and mixed groups also increase while the prompt remains hard.This gives actor-critic learners more successes to reinforce and group-contrast learners more groups with nonzero advantages.Source: §6 · p. 11
  • The PTGS group-frequency result assumes temperature scheduling raises success probability; it does not claim that higher temperature always helps hard prompts.The paper reports fewer zero-success prompts with PTGS than with a fixed-temperature baseline in an empirical check.Source: §6 · p. 12

7 Related Work

The related work situates the paper among debates about post-training and reasoning boundaries, approaches to preserving policy diversity, and test-time repeated sampling. The paper extends these discussions to agentic tasks and uses repeated-sampling curves diagnostically.

Does RL post-training expand or sharpen the reasoning boundary?

  • Prior evidence on whether RL post-training sharpens or expands reasoning boundaries comes almost entirely from math and coding, not agentic environments.The cited work also notes that pass@K can be inflated by lucky final answers.Source: §7 · p. 12

Preserving diversity in post-training

  • PTGS is a sampling-side diversity method that adapts temperature per prompt through a Beta-Binomial posterior without changing the RL algorithm.The passage distinguishes it from prior sampling-side approaches that schedule temperature across training or within rollouts.Source: §7 · p. 12

Test-time parallel scaling via repeated sampling

  • Unlike work treating repeated sampling as a performance tool, this paper uses the shape of test-time scaling curves to diagnose policy changes after post-training.Source: §7 · p. 12

8 Discussion and Conclusion

The discussion concludes that base models with a light harness can solve many agentic tasks and often reach more tasks than post-trained models with enough rollouts. It argues that post-training should be assessed for both reliability and retained coverage, while whether heavier RL moves beyond elicitation remains open.

8 Discussion and Conclusion

  • In agentic tasks, base models with a light harness often reach more tasks than post-trained models given enough rollouts, though post-training improves reliability.Source: §8 · p. 13
  • The authors argue that sharpening becomes more costly as base models strengthen and test-time budgets grow, because rare behaviors can then be recovered through additional sampling.Source: §8 · p. 13
  • Whether heavier RL truly moves beyond elicitation remains open, and the authors suggest reporting Sharpening Tax alongside accuracy.Source: §8 · p. 13

9 Limitations and Future Work

The appendices document observational limits and the benchmark, serving, harness, and estimation choices behind the sharpening analysis.

9 Limitations and Future Work

  • Undisclosed pre- and post-training data makes the model analysis observational, and the paper does not compare how different post-training methods sharpen policies.The authors identify controlled data and comparisons across RL, supervised fine-tuning, and distillation as future work.Source: §9 · p. 13
  • The analysis covers text-based LLM agents; whether its accuracy-consistency-coverage trade-off extends to multimodal agents remains open.Sharpening Tax also relies on binary success, so open-ended generation would require quality-gated diversity measures.Source: §9 · p. 13

A Experimental Details

  • The appendix outlines reproduction details for sharpening analysis and PTGS, including benchmarks, serving, harnesses, metrics, training, and pseudocode.Source: A Experimental Details · p. 22

A.1 Benchmarks and Environments

  • The benchmarks are process-centric, and the study selects tasks with non-trivial success; BFCL uses one category because comparisons can flip across categories.WebShop counts only fully matching purchases as success; ACEBench excludes tasks requiring an LLM-based user simulator.Source: §A.1 · p. 22

A.2 Model Checkpoints and Serving

  • The study uses open-source base/post-trained pairs, but their undisclosed recipes mean “RL post-trained” refers to post-training effects, not a specific mechanism.Models share benchmark sampling settings; turning thinking mode on improves post-trained pass@k but does not change the reported trend.Source: §A.2 · pp. 22–23

A.3 Harness Design for Base Models

  • A universal lightweight harness converts benchmark conversations to plain-text prompts and parses completions into tool calls without per-model calibration or solved examples.The harness is essential for six of eight base checkpoints on BFCL, while it lowers post-trained performance on average, especially pass@1.Source: §A.3 · p. 25

A.4 Metrics and Uncertainty Estimation

  • Metrics are estimated per task from success counts, then averaged equally across tasks; paired task bootstraps produce 95% intervals for Figure 6.The final pass@K point is noisier because it reduces to whether any rollout succeeded for each task.Source: §A.4 · pp. 25–26

A.5 PTGS Training and Evaluation Setup

The appendix specifies PTGS training and evaluation, details its sampling and tax calculations, and gives proof interpretations of the sharpening metrics.

A.5 PTGS Training and Evaluation Setup

  • PTGS is evaluated by fine-tuning Qwen2.5-7B-Instruct on slippery FrozenLake and Sokoban, then testing checkpoints on 64 unseen tasks with 128 rollouts each.PPO uses the final step-200 checkpoint; GRPO uses the highest-validation-success checkpoint, and results average five runs per setting.Source: §A.5 · pp. 26–27
  • PPO and GRPO baselines use fixed temperature 1.0, while PTGS substitutes a prompt-specific temperature during rollout generation and matching policy-log-probability computation.Temperatures are drawn from a Beta posterior over each prompt’s success rate; the target success rate and temperature range vary by algorithm and environment.Source: §A.5 · p. 26, §A.6 · p. 27

A.6 Pseudocode

  • The Sharpening Tax procedure computes pass@k curves, sums gaps from pass@K into A(K), normalizes to S(K), and takes base-minus-post-trained differences.The listing omits the cumulative-product estimator and paired task bootstrap used for reported estimates and intervals.Source: §A.6 · p. 27
  • PTGS tracks discounted prompt-level successes and failures, samples a success-rate estimate from its Beta posterior, maps it to temperature, and updates the counts from each rollout group.Using that same temperature for policy log-probabilities keeps the update on-policy.Source: §A.6 · p. 27

B Missing Proofs

  • Appendix B restates the theoretical results and proofs, including a decomposition of the tax used in a later appendix analysis.Source: B Missing Proofs · p. 29

B.1 What the Sharpening Tax Measures

  • Scalability has a first-success interpretation: pass@k equals P(H≤k), where H indexes the first successful rollout.The proposition applies to the reported estimates whenever the test-time budget K does not exceed the number of observed rollouts.Source: §B.1 · p. 29

B.2 Ceiling and Saturation Decomposition of the Tax

  • Raw scalability is K times the saturation gap between coverage ceiling pass@K and budget-averaged coverage; the tax splits into ceiling and averaged-coverage gaps.Source: §B.2 · p. 30

B.3 Why Sharpening Charges a Tax

The sharpening model explains how pushing task success probabilities to extremes reduces retry value, while PTGS improves the rollout groups used for learning by adapting temperature to prompt difficulty. The additional results report that post-training bimodalizes task success rates, with different coverage effects across backbone scales.

B.3 Why Sharpening Charges a Tax

  • For every K ≥ 2, sharpening a fraction λ of tasks to success-probability extremes yields TaxA(K) = λ ABase(K) ≥ 0, with strict inequality when λ > 0 and base scalability is positive.The proportionality assumes the sharpened-task selection is independent of base success probability; the tax remains nonnegative if selection depends on that probability.Source: §B.3 · p. 30

B.4 How PTGS Improves Group-Based RL

  • When PTGS raises a hard prompt’s success probability, it increases the chance a rollout group contains a success; for prompts at or below 1/2, mixed groups also become more likely.For mastered prompts, cooling raises success probability while the probability of mixed groups falls; those groups become all-success groups rather than all-failure groups.Source: §B.4 · pp. 31–32

C Additional Results on the Accuracy–Coverage Tension

  • Post-training bimodalizes per-task success rates across families, separates base-model coverage from consistency, and can leave base models with higher coverage at large budgets, especially for larger backbones.The smallest backbones may instead gain coverage when post-training supplies basic skills; the additional-results overview reports these patterns across all families.Source: C Additional Results on the… · p. 32

C.1 Aggregate Results by Backbone Scale

Across scales and model families, post-training’s effects on coverage depend on the base model’s capabilities: it can supply missing skills to small backbones but polarize larger models’ task success. At larger budgets, the added results characterize the tax, its task-level sources, and the limits of global temperature adjustment.

C.1 Aggregate Results by Backbone Scale

  • For the largest backbones, the mean base coverage curve overtakes the post-trained curve around k=22, while for the smallest backbones it never does so within the budget.Small models more often gain coverage as post-training supplies basic skills; large models more often lose tasks than they gain.Source: §C.1 · p. 32, Figure 11 · p. 32

C.2 Results by Model Family and Checkpoint Pair

  • For larger backbones across families, post-training shifts per-task success toward always-solved and always-fail extremes; small backbones on BFCL can instead move tasks out of always-fail.Across model-benchmark combinations, post-training generally gains sampling efficiency while larger backbones more often pay with lost coverage at high budgets.Source: §C.2 · p. 34, Figure 22 · p. 40, Table 6 · p. 41

C.3 Per-Task Scaling Curves and Their Density

  • On Gemma-4’s largest backbone, post-training concentrates task curves at zero or full success, while base-model coverage rises with retries and overtakes post-training at large budgets.On the smallest Gemma-4 backbone, post-training instead moves tasks off the never-solved edge; at k=128 its coverage is 65.2% vs. 48.6% on WebShop and 38.5% vs. 27.0% on BFCLv4.Source: §C.3 · p. 42, Figure 23 · p. 43, Figure 24 · p. 44

C.4 Global Temperature Scaling Cannot Resolve the Trade-off

  • A higher global temperature usually improves post-trained pass@k at larger budgets but can lower pass@1, so one temperature does not improve both accuracy and coverage.The gains are not uniform or monotone in temperature, motivating prompt-adaptive temperature in PTGS.Source: §C.4 · p. 42, Figure 25 · p. 45

D.1 Sharpening Tax for All Fourteen Checkpoint Pairs

  • In every model-benchmark combination, post-trained models reach a larger fraction of their coverage ceiling than base models, while raw sharpening tax generally grows with rollout budget.The table summarizes 42 evaluations; the negative-tax exceptions at k=128 are three small-backbone BFCL settings where post-training repairs tool-call formatting.Source: Figure 26 · p. 46, §D.1 · p. 46

D.2 Decomposition of the Tax by Task Movement

  • The lost-task contribution to TaxA(128) is positive in every combination, while substantial coverage recovery on gained tasks appears mainly for small backbones on BFCL and WebShop.The decomposition separates lost, gained, and curve-shape contributions; task-level movement is shown for the largest backbone of each family.Source: §D.2 · p. 48, Table 8 · p. 48, Figure 27 · p. 49

D.3 Which Tasks Pay the Tax?

The tax is concentrated on tasks where post-training removes rare successes recoverable through retries, while a small pilot estimate can guide policy selection. Additional PTGS results show distinct training dynamics on FrozenLake and Sokoban.

D.3 Which Tasks Pay the Tax?

  • On highest-tax tasks, post-trained policies are deterministic while base policies retain rare successes that retries can recover; lowest-tax tasks show the opposite pattern.The highest-tax curves are flat at 100% or 0% for the post-trained policy, whereas the base policy may rise from a rare success toward full coverage.Source: §D.3 · p. 50, Figure 28 · p. 50

D.4 Tax-Guided Routing between the Base and the Post-trained Policy

  • Tax-based routing from 32 pilot rollouts exceeds post-trained pass@128 in five of six panels by up to 13.6 points, while its pass@1 exceeds the base model in every panel.It beats uniform random routing in every panel and stays close to oracle routing; its pass@1 lies between those of the two individual policies.Source: §D.4 · p. 55, Figure 33 · p. 55

E.1 Training Dynamics of PTGS

  • PTGS pays a smaller tax than PPO at most checkpoints in both environments; it matches PPO’s final validation success on FrozenLake and outperforms it on Sokoban.PTGS maintains higher entropy through training; on Sokoban, PPO validation success declines after step 150 while PTGS continues improving.Source: §E.1 · p. 56, Figure 34 · p. 56, Figure 35 · p. 56

E.2 Ablation on the Temperature Spread τ of PTGS

Across training and frozen-policy tests, PTGS’s prompt-adaptive temperatures improve accuracy and coverage together, while outcomes depend on task difficulty and the temperature spread.

E.2 Ablation on the Temperature Spread τ of PTGS

  • τ = 1.5 gives PPO with PTGS the best final pass@1 and pass@128 in both environments; the spread needed for coverage gains differs by environment.On FrozenLake, τ = 1.2 improves pass@128 and reduces tax, whereas on harder Sokoban, τ = 1.2 barely raises entropy and only τ ≥1.3 improves pass@128.Source: §E.2 · p. 56, Figure 36 · p. 57

E.3 Fixed Rollout-Temperature and Decoding Ablations of the PPO Baseline

  • Fixed rollout temperatures shift the accuracy-coverage trade-off, truncation barely changes PPO, and PTGS alone improves pass@1 and pass@128 together.PTGS has the best pass@128 in both environments and the best or near-best pass@1, with smaller tax than fixed temperatures T ≥1.0.Source: §E.3 · p. 58

E.4 Inference-Time PTGS Application

  • On FrozenLake, every inference-time PTGS setting beats the global-temperature curve on both metrics; on Sokoban, only two settings exceed every fixed temperature by one or two pass@128 tasks.The comparison uses a frozen Qwen2.5-7B-Instruct on 64 tasks per environment.Source: §E.4 · p. 59, Figure 37 · p. 59

F.1 Accuracy of the Difficulty Estimate and Realized Temperatures

  • PTGS’s posterior mean difficulty estimate correlates with realized success at 0.81 on Sokoban and 0.71 on FrozenLake, while underestimating success by about 0.1 on average.The estimate becomes more correlated over time on FrozenLake; its conservatism reflects discounted counts from weaker earlier policies and a prior centered below 1/2.Source: §F.1 · p. 59
  • PTGS heats harder Sokoban prompts but cools on FrozenLake as most prompts become solvable, using varied temperatures across prompts rather than one global shift.On FrozenLake, mean temperature falls from about 1.1 early in training to below 0.9 at the end; within-step standard deviations are about 0.2 to 0.3.Source: §F.1 · p. 60, Figure 39 · p. 60
  • Across both environments, PTGS keeps zero-success rollout groups below PPO for most of training, leaving more groups with a learning signal.A group with no successes among its 16 rollouts carries no success signal for the update.Source: §F.1 · p. 60, Figure 40 · p. 60

F.2 Does PTGS Induce a More Exploratory Policy?

  • On Sokoban, PTGS maintains higher entropy than PPO at T=1.3 or T=1.5; on FrozenLake, PPO needs T=1.5 to reach similar entropy.That PPO temperature is about 60% above PTGS’s mean temperature.Source: §F.2 · p. 61, Figure 41 · p. 61
  • At step 200, PTGS achieves the highest pass@1 and pass@128 in both environments, while globally heating PPO does not match its coverage.On Sokoban, heating PPO to T=1.5 improves both metrics over default PPO, but both remain below PTGS; on FrozenLake, heating lowers pass@128.Source: §F.2 · p. 61, Figure 41 · p. 61

F.3 Case Study: Tasks That PPO Loses and PTGS Recovers

Among tasks PPO loses, PTGS recovers about half, with a much lower reverse rate; the examples show recovery across five training runs in both environments.

F.3 Case Study: Tasks That PPO Loses and PTGS Recovers

  • Among tasks lost by PPO, a PPO w/ PTGS run solves about half, while the reverse rate is much lower.The case studies show PPO failing to solve each task in four of five runs, while PTGS solves each in most runs and masters some.Source: §F.3 · p. 61
  • On the Sokoban example, the base model solves the task in 4 of 128 rollouts; PPO fails in four of five runs, while PTGS solves it in all five and masters it in two.The comparison is at step 200.Source: Figure 42 · p. 62, §F.3 · p. 62
  • On the FrozenLake example, the base model solves the task in 13 of 128 rollouts; PPO fails in four of five runs, while PTGS solves it in four and masters it in three.The comparison is at step 200.Source: Figure 43 · p. 62, §F.3 · p. 62

Related papers

Loading 2610.01509v1…