PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck

Davide Romano, Kanak Raj, Jerrod Parker, Daniele Giofrè

arXiv:2608.18931v1cs.CLcs.AI

TL;DR

Open-ended generation lacks the reliable verification that enabled test-time scaling on mathematics and code. This paper systematically compares five TTS families across five open-ended benchmarks using an exploration–exploitation framework, finding that exploration improves candidate quality while exploitation is the bottleneck and Fusion recovers only about 40% of available quality.

  • Problem

    Test-time scaling has been developed mainly on tasks with straightforward verification, leaving its effectiveness on open-ended generation with nuanced quality criteria insufficiently evaluated.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper compute-normalises five TTS families across five open-ended benchmarks and decomposes each method’s token budget into exploration and exploitation.

    Source: §1 · p. 1, §3.1 · p. 3, §4 · p. 5

  • Results

    The best candidate improves with compute, but exploitation stagnates or regresses; Fusion captures only ∼40% of available headroom, while verifier correlation falls to ∼0.12.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The candidate pool is not the bottleneck; converting it into a strong final output remains the open problem.

    Source: §1 · p. 2

    WritingBench’s apparent Sequential Refinement advantage is confounded by verbosity bias in the benchmark’s evaluation design.

    Source: §6.3 · p. 9

Abstract

from arXiv · show

Test-time scaling (TTS) improves language model outputs by spending additional inference compute - generating multiple candidates, searching over partial sequences, or iteratively refining drafts. These techniques yield large gains on mathematics and code, but have been developed and stress-tested almost exclusively on tasks where verification is straightforward. We conduct the first compute-normalised comparison of five TTS families across five open-ended generation benchmarks spanning medicine, law, finance, general chat, and creative writing - grounded in a unified framework that decomposes the effectiveness of each method's token budget into exploration and exploitation. The answer depends on which side of that decomposition you examine. Scaling exploration works: the best candidate in the pool improves steadily with compute across all settings. What breaks is exploitation - the step that converts a rich candidate pool into a final output. With state-of-the-art generators, reward models correlate at only $ρ_v \approx 0.12$ with true quality, rendering selection near-random regardless of budget. Tree search amplifies this failure through diversity collapse. Refinement helps on one of five benchmarks; its apparent gains elsewhere are confounded. Only synthesis across candidates (Fusion) consistently improves over single-sample baselines, yet still recovers only ~40% of available quality. The candidate pool is not the bottleneck - choosing from it is.

1 Introduction

This first systematic, compute-matched evaluation of five test-time scaling families across five open-ended benchmarks finds that exploration improves candidate pools, but exploitation fails to convert them into strong final outputs. Fusion is the strongest method yet recovers only about 40% of available quality, leaving candidate selection and synthesis as the central bottleneck.

Motivation

  • Verifier-based TTS succeeds on deterministic tasks but breaks on open-ended generation, where nuanced quality rubrics and limited reliable supervision make reward-model ranking nearly random.Methods that bypass verifiers through synthesis or self-critique perform unevenly across benchmarks and model families.Source: §1 · p. 1

Evaluation scope

  • The paper gives the first systematic, compute-matched comparison of Best-of-N, Beam Search, Particle Filtering, Refinement, and Fusion across five open-ended generation benchmarks.These benchmarks span medicine, law, finance, general chat, and creative writing.Source: §1 · p. 1

Central finding

  • ∼40% of available headroom is captured by Fusion, while oracle quality rises with compute but realised quality stagnates or regresses for verifier-based methods.None of compute, reward-model size, or generator size closes the gap between single-sample and oracle quality.Source: §1 · pp. 1–2

Evaluation methodology

  • The bias-corrected oracle estimator removes maximum-score inflation caused by noisy judging, with a material naive-to-corrected gap on every benchmark.The estimator is closed-form per entry and uses empirically estimated judge variance from repeated scoring.Source: §1 · p. 2

Verifier analysis

  • ∼0.12 is the Spearman correlation between reward-model scores and true quality, and both tested Outcome Reward Models fail similarly.For Best-of-N, headroom capture approximately equals verifier correlation, indicating a structural rather than model-specific selection failure.Source: §1 · p. 2

Per-method diagnosis

  • 40–60% of independent-sampling diversity is lost by Beam Search and Particle Filtering, while Sequential Refinement improves genuinely on only one of five benchmarks.Fusion improves over the single-sample baseline on every benchmark for Qwen3.5 but has mixed results on OLMo3, showing synthesis quality varies by model family.Source: §1 · p. 2

2 Related Work

Prior TTS work comprises five main families whose exploration–exploitation tradeoffs depend especially on external verifiers. Existing evaluations remain dominated by deterministic, math-oriented tasks, leaving no systematic compute-normalised comparison on open-ended generation.

TTS methods and verifiers

  • TTS methods span parallel verification, search with PRMs, sequential refinement, candidate fusion, extended thinking, and hybrids [29] [20] [16] [17] [10] [19] [4] [22] [5] [15] [25] [32].Parallel methods generate independent candidates, search explores partial sequences, refinement critiques one candidate, fusion synthesises candidates, and extended thinking uses one reasoning trajectory.Source: §2 · p. 2
  • Parallel and search methods depend on external verifiers: ORMs score complete query–completion pairs, while PRMs score intermediate Chain of Thought reasoning steps [12].These verifiers are trained predominantly on preference data or math-heavy datasets, and Best-of-N sampling is a common evaluation method [14] [18].Source: §2 · p. 2
  • Prior work lacks a systematic comparison of TTS families or a shared-token-budget framework separating exploration from exploitation and testing verifier-quality degradation.The missing framework concerns generating candidates versus scoring, selecting, or synthesising the final output.Source: §2 · p. 2

Evaluation beyond deterministic tasks

  • TTS evaluation is mostly math-oriented and remains limited to deterministic, verifiable tasks [25] [29] [32], motivating compute-normalised comparison on open-ended generation.The paper identifies this as the remaining evaluation gap across the main TTS families.Source: §2 · p. 2

3 Theoretical Framework

The framework analyzes test-time scaling as a fixed-budget tradeoff between exploration and exploitation. It predicts that verifier quality limits gains and that oracle ceilings differ across TTS methods.

3 Theoretical Framework

  • The framework organizes TTS methods along one axis: how each partitions a fixed token budget between exploration and exploitation.This decomposition supplies the two implications governing the analysis: verifier quality constrains exploitation, while method-specific pools determine oracle ceilings.Source: §3 · p. 2

3.1 Budget Decomposition and Quality Metrics

  • Every TTS method partitions a fixed token budget T into exploration tokens T_e and exploitation tokens T_x, with T = T_e + T_x.The framework distinguishes ground-truth, external, and self-verifiers according to how exploitation scores or generates candidates.Source: §3.1 · p. 3
  • Oracle quality is the best true-quality score available in the full candidate pool, whereas realised quality measures the final output produced from that pool.Headroom capture h quantifies the fraction of exploration gains realised: h = 1 reaches oracle quality, h = 0 matches a single sample, and h < 0 indicates harm.Source: §3.1 · p. 3

3.2 Implication 1: The Verifier Bottleneck

  • Verifier correlation directly determines BoN headroom capture: zero correlation eliminates gains from additional compute, while negative correlation makes more compute harmful.Measured verifier correlation remains predictive despite judge noise, because shared attenuation cancels between estimated correlation and headroom capture.Source: §3.2 · p. 4

3.3 Implication 2: Oracle Ceilings Differ by Method

  • Oracle ceilings differ by method: BoN improves monotonically with diminishing returns, Fusion can benefit from synthesis, and SR benefits only when conditioning on prior drafts improves quality.These per-method ceilings are studied empirically in Section 6.Source: §3.3 · p. 4

4 Experimental Setup

The study evaluates five test-time scaling families across five open-ended domains without exact-match verification, using compute-normalized budgets and a shared large-model judge. The setup spans representative generators, reward models, search variants, refinement, budget forcing, and candidate fusion.

Benchmarks

  • Five benchmarks cover medicine, law, finance, general chat, and creative writing, and none supports exact-match or binary verification.The study therefore evaluates open-ended generation rather than tasks with straightforward correctness checks.Source: §4 · p. 4

TTS Methods

  • Five TTS families are represented by Best-of-N, Beam Search, Particle Filter, Sequential Refinement, Budget Forcing, and Fusion, with two tree-search variants separating deterministic and stochastic diversity.BoN uses an external ORM, tree methods use PRM guidance, refinement uses a self-verifier, and Fusion synthesizes N candidates.Source: §4 · pp. 4–5

Models

  • The experiments use OLMo3, OLMo3.1, and Qwen3.5 generators, with ORMs for BoN and VersaPRM-8B for tree search.ORMs were selected using RewardBench 2 performance, while VersaPRM was selected for multi-domain training.Source: §4 · p. 5

Compute Normalisation

  • Compute is normalized by total generator output tokens across generative steps, with Low, Mid, High, and XHigh matching BoN N ∈{2, 4, 8, 16} within 25%.Discriminative reward-model costs are excluded as negligible relative to autoregressive decoding; the experiments used approximately 23,800 GPU-hours.Source: §4 · p. 5

Evaluation

  • Qwen3.5-397B-A17B judges all benchmarks, with validation against native judges yielding Macro F1 0.679 on HealthBench and criterion-level κ = 0.679 on PRBench.Agreement was also measured by QWK 0.564 on WildBench and 0.408 on WritingBench; on identical BoN samples, headroom capture matched the official 7B Critic at 0.19 vs. 0.19.Source: §4 · p. 5

5 Results

Across matched-compute benchmarks, most selection and search methods stagnate despite steadily improving candidate pools. Fusion is the only consistently scaling method, while refinement is unreliable and reward-model selection remains structurally weak.

Search and scaling

  • BoN gains negligibly across an 8× budget increase, while Tree Search never beats the single-sample baseline and Budget Forcing is flat or regresses.The reasoning models appear to exhaust useful thinking within modest token budgets, so Budget Forcing is not scaled further.Source: §5 · p. 5

Exploitation gap

  • Oracle quality rises steadily with compute on every benchmark, but no selection or search method identifies the better candidates.Figure 2 contrasts available oracle quality with realised quality at matched XHigh compute; naive-oracle estimates conservatively understate the exploitation gap.Source: §5 · pp. 5–6, Figure 2 · p. 6

Reward-model selection

  • 0.584 overall at XHigh is produced by both Skywork-Reward-V2 and Llama-3.1-70B-RM, whose near-identical scores indicate structural rather than model-specific failure.The quantified mechanism is analyzed further in Section 6.1.Source: §5 · p. 6

Overall comparison

  • 0.610 overall at XHigh, Fusion improves over the 0.574 single-sample baseline on every benchmark, outperforming the best RM-based method at 0.584.This scaling pattern holds across model scales.Source: §5 · p. 6

Sequential Refinement

  • Sequential Refinement gains +7.3pp on WritingBench and beats Fusion on PRBench, but regresses −2.3pp on HealthBench and −4.5pp on LEXam.WritingBench gains are confounded by verbosity, while WildBench gains come almost entirely from one subtask.Source: §5 · p. 6

6 Analysis … A Compute Band Configurations

Across open-ended benchmarks, exploration scales reliably, but exploitation remains the bottleneck: selection is weak, tree search collapses diversity, and Fusion captures only part of available quality. These findings hold across model families and motivate verifier diagnostics, while remaining subject to evaluation and coverage limitations.

6 Analysis

  • Fusion captures ∼40% of available quality versus BoN’s ∼15%, despite nearly identical oracle candidate pools, making exploitation—not candidate generation—the central bottleneck.At the highest compute budget, Fusion and BoN oracle pools typically differ by only 0.01–0.04, while their realised scores diverge consistently.Source: §6 · p. 6

6.1 Verifier Correlation Explains Selection Failure

  • Open-ended reward-model correlations average only 0.12 for Skywork and 0.11 for Llama across 152 combinations, reducing BoN to near-random selection.The observed relationship between verifier correlation and headroom capture validates the predicted identity h_BoN ≈ ρ_v, with R2 = 0.66 and ρ = 0.81.Source: §6.1 · pp. 6–7, Figure 3 · p. 7

6.2 Tree Search Failure: Diversity Collapse

  • Particle Filtering fails in both exploration and exploitation by collapsing diversity: mean cosine distance is 0.036–0.069 versus BoN’s 0.123–0.124, with particles reaching similarity ≥0.997.Miscalibrated PRM pruning removes promising branches exponentially with generation depth, whereas naive parallel sampling preserves more diversity at equal cost.Source: §6.2 · pp. 7–8, Figure 4 · p. 8

6.3 Generative Methods: Fusion Synthesises, Sequential Refinement Regresses

  • Fusion consistently improves synthesis: it matches or exceeds BoN’s oracle ceiling and beats an independent held-out candidate, indicating higher mean quality rather than variance alone.Fusion generates fewer independent candidates than BoN at equal compute, yet systematically raises candidate quality beyond the shared sub-pool.Source: §6.3 · p. 8, Figure 5 · p. 8
  • Sequential Refinement is benchmark-dependent: it is counterproductive on HealthBench and LEXam, genuinely improves only PRBench, and its WritingBench gain is confounded by verbosity bias.WritingBench shows within-prompt length–score correlation ρ = +0.33 for SR versus +0.20 for BoN, while excluding Coding & Debugging turns WildBench’s aggregate lift into regression.Source: §6.3 · pp. 8–9

Appendix P.

  • Generation capability does not imply exploitation capability: Qwen3.5 closes ∼36% of headroom under Fusion, whereas OLMo3.1-32B-Think produces negative headroom on two of three benchmarks.Within Qwen3.5, Fusion improves over single-sample baselines at every tested scale, but its advantage over random candidates does not grow with model size.Source: Appendix P · p. 9, §6.4 · p. 9

7 Conclusion

  • The study’s compute-normalised comparison finds exploitation, not exploration, is the bottleneck: oracle quality rises with compute, but no method converts more than a fraction into realised quality.RM selection fails structurally, tree search compounds failure through diversity collapse, Fusion captures ∼40% of headroom, and Refinement yields genuine gains on only one benchmark.Source: §6 · p. 6, §7 · p. 9

A Compute Band Configurations

  • The reported compute bands specify method configurations, including Fusion High subsets of 7 for Qwen3.5 and 6 for OLMo3, with some tasks using W = 3, k = 4.Table 5 also defines the reported mean thinking-token and answer-token quantities for BoN samples and marks unrun entries with —.Source: Table 5 · p. 12

B Benchmark Details · C Hyperparameters and Prompt Templates

The evaluation spans five open-ended benchmarks with benchmark-specific subsets, rubrics, judges, and conversation handling. Generation uses standardized sampling and method-specific configurations, including custom refinement and fusion prompts, while preserving documented compute and scoring conventions.

B Benchmark Details

  • The evaluation uses filtered English subsets of LEXam and WritingBench, four PRBench configurations, and WildBench v2, with dataset sizes and domain or category skews explicitly documented.LEXam retains 516 English items, WritingBench 555, PRBench totals 1,650, and WildBench contains 1,024 items.Source: B Benchmark Details · pp. 12–13
  • HealthBench contains 5,000 items partitioned into seven theme-based tasks and scores responses using physician-authored signed-point rubrics normalized to a clipped mean.Each example averages 11.4 criteria, with positive and negative points reflecting desired and undesired behaviors.Source: B Benchmark Details · p. 12
  • WildBench is micro-averaged over all 1,024 items rather than macro-averaged across task groups, shifting absolute scores but preserving relative TTS ordering.The stated comparison is between methods evaluated on the same item set.Source: B Benchmark Details · p. 13
  • HealthBench, PRBench, and WildBench retain multi-turn conversations as ordered role-content message lists for both generation and judging, while LEXam and WritingBench are single-turn.The multi-turn shares are 41.7%, 42.2%, and 27.0%, respectively.Source: B Benchmark Details · p. 13

C Hyperparameters and Prompt Templates

  • All generation uses seed 42, a shared system prompt, family-specific sampling profiles, and method-dependent context lengths; Fusion and Budget Forcing reduce max_tokens to 8,192.Sequential Refinement and Fusion strip thinking blocks before subsequent processing, and Qwen3.5 receives expanded context for refinement and larger Fusion pools.Source: C Hyperparameters and Prompt Templates · p. 14
  • BoN generates and scores independent completions, Beam Search expands and prunes candidate paths, Particle Filter resamples particles, Sequential Refinement iterates feedback and refinement, and Budget Forcing extends thinking.Sequential Refinement uses a maximum three-iteration context window, while Budget Forcing is evaluated only at Low and Mid because higher budgets caused repetitive self-termination.Source: C Hyperparameters and Prompt Templates · pp. 15–16
  • ORMs score complete candidates in one classification pass, while VersaPRM-8B scores search paths per step and aggregates them using the minimum positive probability.BoN shares its 16-candidate XHigh pool across lower bands through bootstrap subsampling, and other methods use native prompts unless custom templates are specified.Source: C Hyperparameters and Prompt Templates · pp. 15–17
  • Sequential Refinement uses feedback, stop-check, and revision prompts capped at the most recent K=3 draft-feedback pairs, whereas Fusion directly synthesizes shuffled candidate subsets.The stop-check is computed but ignored under forced-iteration evaluation; Fusion uses a direct-synthesis variant rather than the unused compare-then-fuse variant.Source: C Hyperparameters and Prompt Templates · p. 16

D Judge Agreement Analysis … Heterogeneous attenuation across benchmarks.

Judge agreement is strong against physician annotations but heterogeneous across benchmarks, while the paper’s theory shows how verifier and judge noise shape test-time scaling estimates. The proposed corrections clarify oracle bias without changing the measured relationship between headroom capture and verifier correlation.

D Judge Agreement Analysis

  • Qwen3.5-397B-A17B reaches Macro F1 0.679 against physician annotations, comparable to GPT-4.1 at 0.709 and o4-mini at 0.692, and above the inter-physician mean of 0.655.The unified judge falls at the 57th percentile of agreement among 184 physicians.Source: D Judge Agreement Analysis · p. 19, Table 12 · p. 19
  • Agreement is strongest for PRBench and LEXam, while WildBench and WritingBench reach QWK values of 0.564 and 0.408 because evaluation criteria vary across domains and task types.Table 13 compares criterion-level agreement between the unified and native judges, using 5% sampling for PRBench and 15% for the other benchmarks.Source: D Judge Agreement Analysis · p. 19, Table 13 · p. 19

E Theory

  • Headroom capture for Best-of-N equals verifier correlation: perfect correlation captures all available quality, zero correlation yields no compute benefit, and negative correlation makes more compute harmful.Self-verifier methods instead depend on the generator’s own distributional quality, with capability—not parameter count alone—setting their headroom ceiling.Source: E Theory · p. 20
  • At per-step pruning error ϵ_prune = 0.33 and depth d = 4, the best candidate survives with probability ≤0.20, so tree search returns a suboptimal candidate more than 80% of the time.This formalizes why Beam Search and Particle Filter can underperform naive parallel sampling when PRM miscalibration is severe.Source: E Theory · p. 20

F Oracle Bias From Judge Noise

  • Judge noise inflates the naive oracle max_i X_i above the ideal oracle max_i µ_i, with the gap growing with candidate count and shrinking when true candidate spread dominates noise.The gap vanishes as σ_J → 0 or τ → ∞, while τ = 0 leaves anσ_J of purely noise-driven apparent winner advantage.Source: F Oracle Bias From Judge Noise · p. 20, §F.1 · p. 21

Ideal per-entry estimator

  • The closed-form oracle-bias correction is exact only for i.i.d. Gaussian candidates, so structural deviations create directional errors: Fusion and Refinement tend toward slight lower bounds, whereas Particle Filter tends toward a slight upper bound.The estimator assumes i.i.d. candidate pools for these algorithms, but correlated trajectories, fused responses, and resampling clusters violate that assumption.Source: Ideal per-entry estimator · pp. 22–23

F.2 Consistency of h ≈ρ_v under judge noise

  • Independent judge noise attenuates measured headroom capture and verifier correlation symmetrically, so the identity h ≈ ρ_v remains valid without oracle correction.The correction is needed for unbiased true-oracle quality estimates, not for the validity of the measured relationship.Source: §F.2 · p. 23, Effect on measured verifier… · p. 23

Heterogeneous attenuation across benchmarks

  • Across benchmarks, differing true-quality spread τ and judge noise σ_J create offsets around the identity line but do not bias its expected slope; the observed slope is 1.198 with R^2 = 0.656.This is consistent with heterogeneous attenuation across benchmarks rather than a failure of the underlying linear relationship.Source: Heterogeneous attenuation across… · p. 24

G Naive Oracle Results … H.3 Qwen3.5-9B Oracle Results

Across naive-oracle and Qwen3.5 results, oracle quality continues to rise with compute while realised quality stagnates, leaving exploitation as the binding constraint. Headroom capture remains bounded by verifier correlation under both oracle conventions.

G Naive Oracle Results

  • Oracle quality rises with compute while realised quality stagnates, so exploitation remains the binding constraint despite the naive oracle’s inflated ceiling.The naive oracle selects partly on judge noise, making the apparent exploitation gap look more severe rather than less.Source: G Naive Oracle Results · p. 24, Figure 7 · p. 24
  • Across 152 task-level points, naive-oracle headroom capture closely follows the identity relation h ≈ ρ_v, matching the bias-corrected analysis.This result uses the uncorrected oracle and measured, non-denoised verifier correlation.Source: G Naive Oracle Results · p. 26
  • Headroom capture is bounded by verifier correlation regardless of whether the oracle and correlation estimate are denoised.The naive-oracle analysis therefore preserves the same exploitation limitation as the bias-corrected convention.Source: G Naive Oracle Results · p. 26

G.1 Headroom capture tables (Unadjusted oracles)

  • Headroom capture h measures the fraction of the oracle gap realised relative to the BoN N=1 single-inference baseline across compute levels and model variants.h = 1 denotes full exploitation, h = 0 matches the baseline, and h < 0 underperforms it.Source: Table 15 · p. 26, Table 18 · p. 27, Table 16 · p. 27

H.1 Qwen3.5-35B-A3B Oracle Results

  • The Qwen3.5-35B-A3B oracle results report oracle quality across five benchmarks and four compute levels, with gains measured against the BoN oracle N=1 baseline.The supplied table caption identifies the comparison scope but does not provide cell values.Source: Table 19 · p. 28, §H.1 · p. 28

H.2 Qwen3.5-9B Realised Results

  • The Qwen3.5-9B realised-results table compares quality across five benchmarks and four compute levels against the single-sample BoN@1 baseline.Its encoding distinguishes improvements from regressions, but the supplied passage does not expose individual cell values.Source: Table 20 · p. 28, §H.2 · p. 28

H.3 Qwen3.5-9B Oracle Results

  • The Qwen3.5-9B oracle-results table reports oracle quality across five benchmarks and four compute levels relative to the BoN oracle N=1 baseline.The supplied passage provides the table’s scope and encoding, not the underlying cell values.Source: Table 21 · p. 29, §H.3 · p. 29

I OLMo3 Results … K Verifier Correlation and Headroom Capture by RM

OLMo3 separates generation from exploitation: candidate quality rises with compute, yet realised outputs can regress, especially for OLMo3-32B. Across models and reward models, headroom capture tracks verifier correlation, confirming selection quality as the bottleneck.

I OLMo3 Results

  • OLMo3-32B Fusion falls from 0.475 baseline to 0.441 at Mid and 0.394 at High, while OLMo3-7B Fusion improves on four benchmarks at High.This contrast shows exploitation depends on synthesis capability rather than generation capability alone.Source: I OLMo3 Results · p. 29, Table 22 · p. 30, Table 23 · p. 30

J.1 Qwen3.5-35B-A3B results

  • Qwen3.5-35B-A3B Fusion captures approximately 40% of headroom and improves every benchmark, establishing the positive comparison for the headroom-capture analysis.The supplied passages identify the corresponding headroom-capture table for Qwen3.5-35B-A3B.Source: I OLMo3 Results · p. 29, Table 26 · p. 31

J.4 Olmo3-7B results

  • OLMo3-7B Fusion improves HealthBench by +3.0pp, LEXam by +2.3pp, PRBench by +1.2pp, and WritingBench by +1.2pp at High, but regresses WildBench by −1.7pp at Mid.Its partial success contrasts with OLMo3-32B and shows the failure is not a simple architecture-wide property.Source: I OLMo3 Results · p. 29, Table 29 · p. 32

J.3 Olmo3-32B results

  • OLMo3-32B Budget Forcing drops overall quality from 0.475 to 0.393 at Mid, while PRBench falls from 0.211 to 0.157 and WritingBench from 0.569 to 0.421.Additional thinking tokens produce degenerate outputs rather than deeper reasoning.Source: I OLMo3 Results · p. 29, Table 28 · p. 32

I.1 Oracle Results Olmo-7B

  • OLMo3-7B oracle quality rises from 0.356 at N=1 to 0.474 at XHigh, with Fusion oracle closely tracking BoN oracle across compute levels.The candidate pool is therefore comparable to Qwen3.5, locating the divergence in exploitation rather than exploration.Source: I OLMo3 Results · p. 29, Table 24 · p. 30

I.2 Oracle results Olmo3-32B

  • OLMo3-32B oracle results provide the corresponding exploration benchmark, separating its poor realised quality from candidate-pool limitations.Tables 24–25 report oracle scores for OLMo3-7B and OLMo3-32B.Source: I OLMo3 Results · p. 29, Table 25 · p. 31

J.2 Qwen3.5-9B results

  • Qwen3.5-9B has a dedicated headroom-capture analysis measuring realised gains against the oracle gap across compute levels.The table defines h = (realized − µ) / (oracleideal − µ), with h = 1 representing full exploitation.Source: Table 27 · p. 31, §J.2 · p. 31

K Verifier Correlation and Headroom Capture by RM

L Deterministic Benchmark Results: MATH-500 and GPQA Diamond

On deterministic benchmarks with binary verification, TTS is more effective than on open-ended QA, but reward-model exploitation remains limited. Stronger generators mainly reduce available headroom rather than improving reward-model exploitation rates.

Benchmark setup

  • MATH-500 [8] and GPQA Diamond [21] provide exact-match and multiple-choice settings with binary verification, the canonical environment for TTS and reward-model methods.The study uses Qwen3.5-35B-A3B, matched compute levels, and the same token-budget normalization as the main experiments to compare method rankings and headroom capture.Source: L Deterministic Benchmark Results:… · p. 33

Results

  • 50% of available headroom on MATH-500 and 15% on GPQA Diamond is captured by RM-based BoN at XHigh, versus ~15% on open-ended QA.The generator reaches pass@1 values of 0.922 on MATH-500 and 0.823 on GPQA Diamond, leaving 3.2 pp and 13.7 pp of headroom to oracle Pass@16.Source: L Deterministic Benchmark Results:… · p. 34, Table 32 · p. 34
  • On weaker reference generations, BoN+Skywork captures ~43% of headroom on MATH and ~22% on GPQA, within a few points of the stronger-generator results.This indicates that reward-model efficacy is approximately benchmark-fixed: stronger generators shrink absolute headroom without improving exploitation rates.Source: L Deterministic Benchmark Results:… · p. 34

M Skywork-Reward-V2 Reproduction on Deterministic Benchmarks … O.2 Native WritingBench Judge: Bias Amplification

Across deterministic and open-ended benchmarks, the experiments validate BoN scaling but show that refinement gains are highly task-specific and often reflect verbosity bias rather than improved content quality. Native WritingBench judging amplifies this length-based bias, confirming it is intrinsic to the benchmark’s evaluation design.

M Skywork-Reward-V2 Reproduction on Deterministic Benchmarks

  • At N=32, realised BoN accuracy reaches ∼0.66 on MATH-500 and ∼0.55 on GPQA, reproducing Skywork-Reward-V2’s reported scaling curves [14].The oracle approaches 1.0 while realised quality plateaus below 0.7, although positive correctness correlation still makes realised quality improve with compute.Source: M Skywork-Reward-V2 Reproduction on… · p. 34, Figure 10 · p. 35

N WildBench Per-Category Decomposition

  • Coding & Debugging contributes 105.5% of WildBench’s realised refinement gain despite comprising 16.6% of items, while all remaining tasks regress by −0.28 points per item.Its mean lift is +27.06 per item; excluding this near-deterministic subtask, Sequential Refinement produces a net regression.Source: N WildBench Per-Category Decomposition · p. 35, Figure 11 · p. 35

O Verbosity Analysis

  • Sequential Refinement increases median response length monotonically across iterations and benchmarks, unlike BoN and Fusion, making apparent WritingBench gains potentially reflect elaboration rather than quality.The steepest length growth occurs on WritingBench and PRBench.Source: §O.1 · p. 36, Figure 12 · p. 36

O.1 Unified Judge: Length–Score Correlation

  • WritingBench has the strongest unified-judge length–score correlations: +0.198 for BoN and +0.352 for Sequential Refinement, with length growth driving its score gains.For Sequential Refinement, CVlen exceeds CVscore on WritingBench, with ratio = 0.36.Source: §O.1 · p. 36, Table 37 · p. 38
  • HealthBench and PRBench remain content-driven under BoN, with CVscore/CVlen ratios of 1.57 and 3.17, whereas Sequential Refinement regressions reflect genuine quality degradation.LEXam is the only benchmark with a negative refinement length–score correlation, ˆρ = −0.060, p < 10−3.Source: §O.1 · pp. 36–37

O.2 Native WritingBench Judge: Bias Amplification

  • Under the native WritingBench judge, length–score correlation rises to +0.370 for BoN and +0.539 for Sequential Refinement, while other benchmarks remain stable.The native judge reduces WritingBench CVscore to 0.034 for BoN and 0.038 for Sequential Refinement, indicating that it discriminates primarily on length.Source: Table 38 · p. 38, §O.2 · p. 38, Table 40 · p. 39
  • Native-judge score variation on WritingBench is almost entirely length-driven, with CVscore/CVlen ratios of 0.25 for BoN and 0.17 for Sequential Refinement.The native WritingBench critic is WritingBench-Critic-Model-Qwen-7B [30], reported to achieve 83% human agreement in pairwise evaluation [30].Source: §O.2 · p. 38, Table 39 · p. 39, Table 41 · p. 39

O.3 WritingBench: Judge Robustness Analysis · P Effect of Model Capacity on Self-Verifier Algorithms

Across model capacity and judge robustness, Fusion is the most reliable self-verifier, while Sequential Refinement remains benchmark-sensitive and judge-dependent. Judge differences do not overturn the main conclusions, with apparent Refinement gains explained by length bias rather than better selection.

O.3 WritingBench: Judge Robustness Analysis

  • BoN headroom capture is judge-invariant at 0.19 for both Skywork and Llama70B reward models, supporting robust conclusions for parallel selection.The comparison rescored identical TTS outputs under the study judge and the official WritingBench Critic model at k=16 across 555 samples.Source: §O.3 · p. 39, Figure 13 · p. 40
  • Sequential Refinement capture rises from 0.58 to 0.75 under the Critic model, but its denominator shrinks from 0.125 to 0.081 while the numerator falls from 0.073 to 0.061.The inflated ratio reflects stronger length bias pulling late realized and oracle scores together, not genuine selection improvement.Source: §O.3 · p. 39, Figure 13 · p. 40

P Effect of Model Capacity on Self-Verifier Algorithms

  • Fusion consistently helps Qwen3.5 across benchmarks with roughly scale-stable gains, whereas Olmo3.1-32b fails and Olmo3-7b often misses BoN@1.The capacity study spans Qwen3.5 models at 4B, 9B, and 27B, plus Olmo variants; increasing compute improves Qwen3.5 performance monotonically.Source: P Effect of Model Capacity on… · p. 40, Figure 14 · p. 41, Figure 15 · p. 42
  • Sequential Refinement gains on 2/5 benchmarks but hurts on 2/5, with no consistent relationship between performance and model capacity.The reported absolute gains and relative error reductions agree on these conclusions.Source: P Effect of Model Capacity on… · p. 40, Figure 14 · p. 41, Figure 15 · p. 42
  • At matched compute, BoN’s oracle beats Sequential Refinement on 3/5 benchmarks, ties it on WildBench, and loses on WritingBench because of judge length bias.WritingBench favors Refinement through the interaction between benchmark length bias and Refinement’s verbosity.Source: P Effect of Model Capacity on… · p. 40, Figure 16 · p. 43
  • At matched candidate counts, Sequential Refinement usually produces a worse oracle pool than i.i.d. sampling: two benchmarks are negative, two show small effects, and WritingBench is strongly positive.Any oracle advantage appears tied to model size, while WritingBench’s large positive effect is attributed to length bias and increasing verbosity.Source: P Effect of Model Capacity on… · p. 40, Figure 17 · p. 44

Q Task Demand Profiles · R Token usage

Demand profiles characterize the skills stressed by each benchmark, but evaluation operationalisation—not demand dimensions alone—determines TTS success. Token-usage tables report completion-token accounting across methods and compute levels, supporting the paper’s compute-normalised comparisons.

Q Task Demand Profiles

  • HealthBench emphasizes MC and CE while largely lacking KN, so its medical tasks stress self-monitoring and rubric-constrained response generation more than factual recall.Sequential Refinement’s −2.3pp regression at XHigh is attributed to saturation of the first draft’s metacognitive engagement.Source: Q Task Demand Profiles · p. 46
  • LEXam emphasizes CE and AS, and its −4.5pp Sequential Refinement regression at XHigh suggests that conditioning on drafts compounds misinterpretations in precise information extraction.Its failure mode is therefore not uniquely associated with MC-dominated tasks.Source: Q Task Demand Profiles · p. 46
  • PRBench’s CE+MC profile supports a +3.5pp Sequential Refinement gain at XHigh because rubric-based scoring rewards elaboration and structured reasoning, unlike precision-anchored HealthBench evaluation.The same demand dimensions can produce regression or gain depending on whether judges reward precision or elaboration.Source: Q Task Demand Profiles · p. 47
  • WildBench combines MC and CE with notable AS, yet Sequential Refinement gains only +2.4pp at XHigh under a holistic checklist judge tolerant of elaboration.This contrasts with HealthBench despite their similar MC dominance.Source: Q Task Demand Profiles · p. 47
  • WritingBench is dominated by CE and CL, but its +7.3pp Sequential Refinement gain is substantially confounded by judge verbosity bias.The demand profile makes refinement plausible, yet genuine skill improvement requires verbosity-controlled evaluation.Source: Q Task Demand Profiles · p. 48
  • Sequential Refinement outcomes diverge across benchmarks sharing dominant MC and CE demands: −2.3pp on HealthBench, +3.5pp on PRBench, and +2.4pp on WildBench at XHigh.These differences show that demand profiles alone do not determine method success.Source: Q Task Demand Profiles · pp. 46–48
  • Fusion improves uniformly across all five demand profiles, indicating structural gains that bypass verifier quality rather than depending on benchmark demand or evaluation style.The profiles therefore characterize task demands without predicting method success on their own.Source: Q Task Demand Profiles · p. 48

R Token usage

  • Token-usage tables report total completion tokens by benchmark for BoN, Particle Filter, and Beam Search across specified compute levels.Table 42 reports BoN n=16 totals and a ±27% compute band, while Tables 43–47 report method-specific totals.Source: Table 42 · p. 49, Table 43 · p. 49, Table 44 · p. 50

Related papers

Loading 2608.18931v1…