PaperlayerView exact revision on arXiv

Source-linked AI summary

Selection, Recombination, or a Fresh Solve? A Candidate-Free Control for Single-Pass Test-Time Aggregation

Guiv Farmanfarmaian

arXiv:2608.18379v1cs.LGcs.AIcs.CL

TL;DR

Test-time aggregation can appear to recover all-wrong candidate pools, but without a candidate-free comparison it is unclear whether candidate context adds value beyond a fresh solve. The paper introduces that control and finds that candidate conditioning helps when multiple candidates are correct but harms when all are wrong, while the one-correct regime remains unresolved.

  • Problem

    Existing baselines leave the net value of candidate context unresolved when every candidate is wrong because they score zero without a correct candidate.

  • Method

    The study compares aggregation with a candidate-free fresh solve under the same token allowance, stratifying conditional effects by the number of correct candidates.

  • Results

    Candidate conditioning helps with multiple correct candidates (Δcand(c2+) = +0.290), harms all-wrong pools (Δcand(c0) = −0.123), and remains unresolved when exactly one candidate is correct.

  • Takeaways & Limitations

    At this scale, all-wrong recovery is lower with candidate conditioning than with a fresh solve, reversing its interpretation as candidate-added value.

  • Takeaways & Limitations

    The evidence is limited to one Qwen3-4B model family, two mathematics benchmarks, and first-answer-truncated candidate fragments.

Abstract

from arXiv · show

When every candidate is wrong, correct-candidate selection is unavailable, yet the aggregation call can still solve the problem afresh. A correct aggregate answer may therefore reflect recombination, fresh solving, or both. For efficient test-time reasoning, the relevant question is whether candidate context adds value beyond the additional generation pass. We introduce the missing candidate-free control under the same maximum output-token allowance and stratify by the number of correct candidates. Across AIME-2025 and HMMT-2025 with Qwen3-4B, candidate conditioning improves accuracy when multiple candidates are correct ($Δ_{\mathrm{cand}}$(c2+) = +0.290), lowers accuracy when every candidate is wrong ($Δ_{\mathrm{cand}}$(c0) = -0.123), and remains unresolved in the one-correct regime. The c2+ and c0 conclusions survive a conservative correction for the adaptive two-benchmark procedure. Under this counterfactual, the interpretation of all-wrong recovery reverses at this scale: conditioning on an all-wrong candidate pool lowers accuracy relative to a fresh solve. Original-format matching and placebo results characterize the failures descriptively but leave their mechanism unresolved. Within a separate structured intervention, explicit answer fields causally steer outputs toward their values; masking yields no measurable accuracy improvement, and equivalence with the original format was not established. The evidence is limited to one Qwen3-4B family, two mathematics benchmarks, first-answer-truncated candidate fragments, and single-pass prompted aggregation.

1 Introduction

The section introduces a candidate-free aggregation control under the same output-token allowance to separate candidate-context value from the additional generation pass. It finds a regime-dependent reversal: conditioning helps with multiple correct candidates but hurts when all candidates are wrong.

  • Interpretation: The all-wrong result reverses the usual recovery interpretation because correct aggregate answers can reflect fresh solving, recombination, or both.All-wrong recovery combines candidate-context effects with the additional generation pass, so performance must be compared with a candidate-free solve.
  • Motivation and control: The candidate-free control compares aggregation with a fresh solve using the same maximum output-token allowance.This comparison measures whether reading candidate context improves on the aggregation-stage generation itself.
  • Experimental design: The study uses a single aggregation pass with AGGREGATE, NO-CANDIDATE, ANSWER-ONLY, VOTING, and ORACLE arms, with generating arms sharing one maximum generation-token allowance.A planted-recombination validation checks the harness, and a pre-specified cross-benchmark decision rule supports the central result.
  • Regime-dependent results: +0.290: Candidate conditioning improves accuracy when multiple candidates are correct (∆cand = +0.290, raw pooled p = 0.004).The evaluation stratifies results by the number of correct candidates.
  • Regime-dependent results: −0.123: Candidate conditioning reduces accuracy when every candidate is wrong (∆cand = −0.123, raw pooled p = 0.012).The conservative correction gives a Holm-adjusted value of 0.0236 under the retrospective correction for the adaptive extension.

2 Method

The method compares five aggregation and control arms on candidate sets stratified by how many candidates are correct, using a shared maximum generation-token allowance for generating arms. Candidate conditioning is evaluated against candidate-free generation, while c0 contrasts are interpreted as net effects rather than evidence that recombination never occurs.

  • Decoding and comparability: All arms decode at temperature 1.0 and top_p 1.0 without seeding, but the comparison is neither compute-matched nor response-length-matched.Realized prompt lengths, output lengths, latency, and total computation differ across arms.
  • Evaluation design: Each K = 4 candidate set is assigned to c0, c1, or c2+ and run through five arms with a shared 16,384-token maximum output allowance.An AIME-2025 sweep also varies K ∈ {2, 4, 8}.
  • Contrasts and stratification: Δcand = AGGREGATE −NO-CANDIDATE measures the net value of conditioning on candidates relative to candidate-free generation.At c0, selection is unavailable, so this contrast isolates the net effect of all-wrong candidate conditioning in a selection-impossible regime.
  • Interpretation: A positive c0 contrast would be consistent with recombination, whereas a negative contrast establishes no net benefit in c0, not the absence of recombination.The contrast can be weighed by other effects of candidate conditioning and does not establish that recombination never occurs.
  • Labeling and analysis: An audited extractor removes <think>, truncates each candidate at its first valid answer, and scores all arms using the same first-answer labeling procedure.The pool’s stored correct answer is never used, so all arms are scored alike.

3 Results

Candidate conditioning helps when multiple candidates are correct but harms when all are wrong, with the c1 regime unresolved and the c2+/c0 conclusions surviving conservative correction. Validation and interventions characterize recombination and displayed-answer effects, but the mechanism of the original-format deficit remains unresolved.

  • Candidate-correctness strata: +0.290 for c2+ and −0.123 for c0 reverse the candidate-conditioning effect, while c1 remains unresolved.The conservative adaptive correction rejects c2+ (adjusted p = 0.0133) and narrowly rejects c0 (0.0236 vs family α = 0.025), but not c1 (0.0755).
  • Planted-recombination validation: 0.767 vs 0.000: AGGREGATE solves planted additive-recombination tasks while NO-CANDIDATE and ANSWER-ONLY do not.These all-c0 constructed tasks were solvable only by combining two candidates carrying half the answer; the pre-specified success threshold was 0.30.
  • Candidate-correctness strata: 0.960 vs 0.887: plurality voting scores numerically higher than aggregation at c2+, but the paired difference is not statistically significant.The paired comparison is 10/4 with p = 0.180.
  • All-wrong failures: 0.845 versus 0.371: wrong AGGREGATE answers at c0 match displayed candidates more often than candidate-free errors on AIME.HMMT shows the same ordering, 0.892 versus 0.506; placebo contexts reproduce much of the deficit, but the mechanism remains descriptive.
  • Structured intervention: −0.0045 [−0.0500, +0.0364]: masking explicit answer fields yields no measurable accuracy improvement in the structured intervention.Changing explicit answer fields causally steers outputs toward their values, but equivalence with the original prompt was not established.

4 Related work

Prior aggregation studies examine recovery without correct-candidate selection, but do not provide a candidate-free aggregation-stage control within candidate-correctness strata. This distinction shows that positive all-wrong recovery does not establish a benefit from candidate conditioning.

  • Single-pass aggregation: GSR synthesizes candidate pools into refined answers and reports at most 9% recovery on all-wrong pools at Nc=0, interpreting this as recombination.Majority voting and best-of-N must fail when every candidate is wrong.
  • Candidate-free control: None of the cited studies compares candidate-conditioned aggregation with a candidate-free generation under the same aggregation-stage allowance and candidate-correctness strata.Existing comparisons include call-matched majority voting, base-model comparisons, or alternative inference-budget allocations.
  • Iterative methods: RSA and TRT are distinct iterative, multi-round methods outside this study’s scope, despite RSA examples of recombining correct intermediate steps from imperfect candidates.TRT is reported as reaching 100% on AIME by iterating.
  • Interpretation of all-wrong recovery: ∆cand = −0.123 shows that positive raw recovery on all-wrong pools does not demonstrate improved accuracy from candidate conditioning.This interpretation is compatible with GSR’s near-zero refinement gap for its untrained prompted aggregator.

5 Limitations

The study’s evidence is limited to one Qwen3-4B model family, one candidate-set seed, and two mathematics benchmarks. The c0 result has a narrow corrected margin and applies only to first-answer-truncated candidate fragments, while benchmark contamination may affect absolute accuracy.

  • Scope: The study covers one Qwen3-4B model family at one principal size, one deterministic candidate-set construction seed, and two mathematics benchmarks.Model decoding was unseeded.
  • Statistical limitation: The c0 effect is significant only in the pooled analysis, with a narrow Holm-adjusted margin of 0.0236 at family α = 0.025.This limits the robustness of the c0 conclusion.
  • Data and candidate limitations: The c0 finding of no net recombination benefit is established only for candidates truncated at each one’s first valid answer, not full traces.AIME-2025 and HMMT-Feb-2025 may also be affected by benchmark contamination, which may inflate absolute accuracy.

6 Conclusion … D RSA: baselines and the Appendix-F recombination example

The candidate-free counterfactual reverses the interpretation of all-wrong recovery: candidate context helps with multiple correct candidates, harms with none correct, and leaves the one-correct regime unresolved. The conclusion is scoped to first-answer-truncated fragments and does not identify recombination rates or deployment-weighted value.

  • 6 Conclusion: Candidate context helps when multiple candidates are correct, harms when every candidate is wrong, and leaves the one-correct regime unresolved.The c0 contrast estimates candidate conditioning’s net effect, not the occurrence or rate of recombination.
  • A Answer extraction and verifier audit: 99.3% of instruct traces continue past their first answer, and last-answer labeling would disagree on ∼20.8% of traces.Correctness uses a single audited first-answer extractor uniformly across arms, with no false negatives found against benchmark golds.
  • A Answer extraction and verifier audit: First-answer-truncated candidate fragments define the c0 scope; full traces and separate arm-output narration analyses lie outside it.The scope follows truncation at each candidate’s first valid answer, while narration is assessed separately with fresh generations.
  • B Extended comparison to GSR, SSA, and TRT: GSR finds a refinement gap scaling with model size and weakly correlated with base capability, while its untrained selfRef@4 average ∆ is ±0.0.Prompt and study differences leave it unresolved whether alternative wording mitigates answer matching in this work.
  • B Extended comparison to GSR, SSA, and TRT: SSA reports that most correct outputs copy candidates already holding the right answer, with little evidence of genuine synthesis in rare gold-absent cases.Its answer-truncation probe nevertheless argues that some synthesis ability remains, limiting a purely selection-dominant interpretation.
  • C What the regimes would imply if observable: At c2+, plurality voting numerically outperforms the alternatives, while a fresh solve scores higher at c0; the c1 comparison remains unresolved.These are conditional point estimates, not deployment recommendations, because correctness-defined regimes are unavailable in practice.
  • D RSA: baselines and the Appendix-F recombination example: On AIME-2025, RSA Pass@1 = 73.18 ± 2.20 versus 68.33 for budget-matched majority voting and 43.91 for the base model.On HMMT-2025, the corresponding values are 47.55 versus 35.00 versus 27.17; per-round improvement is reported only qualitatively.

E Pre-specification and analysis plan

The confirmatory analysis plan was fixed before HMMT data collection, using clustered problem-level sign tests with Holm correction across the three c-bins. Because the AIME c0 result fell in the pre-registered gray zone, the planned HMMT extension was conducted, yielding a pooled negative c0 effect.

  • Primary analysis: The pre-specified primary analysis clustered accuracy by problem and c-bin, then applied exact two-sided sign tests with Holm correction within each Δ-family.Each problem contributed one observation per c-bin: its mean accuracy per arm across sets.
  • Pre-registered extension: p = 0.109 for the AIME problem-level c0 Δcand result triggered the single pre-registered HMMT extension under the gray-zone rule.The plan specified direct reporting for p < 0.05, extension for p in (0.05, 0.15], and set-level reporting for p > 0.15.
  • Pooled result: c0 Δcand = −0.123 after pooling AIME and HMMT, with raw pooled p = 0.0118 before Holm adjustment.The negative c0 direction was exploratory on AIME and confirmatory on HMMT.
  • Chronology and corroboration: Cross-benchmark sign agreement and a pre-HMMT commit support the stated chronology, while the postdated OpenTimestamps anchor provides only prospective tamper-evidence.The commit is treated as corroborating rather than definitive evidence of pre-registration.

F Construction of the validation experiment · G K-sweep and run-health details

The validation benchmark shows that aggregation can recover answers by recombining two candidates whose final answers are both wrong. K-sweep and run-health analyses document stable c0 effects but asymmetric truncation that disadvantages candidate-free arms.

  • F Construction of the validation experiment: AGGREGATE achieved 0.767 versus 0.000 for both NO-CANDIDATE and ANSWER-ONLY on the constructed c0 recombination benchmark.The pre-specified criterion required AGGREGATE to exceed both baselines by ≥0.30.
  • F Construction of the validation experiment: Each validation problem encoded its answer as X+Y across two candidates, both reporting wrong final answers, so recovery required composition.The benchmark used the identical harness and every set was c0.
  • F Construction of the validation experiment: The validation benchmark was designed to test constructed recombination rather than correct-candidate selection.One candidate carried X and the other carried Y, while both final answers were wrong.
  • G K-sweep and run-health details: AIME-2025 c0 Δcand was sign-invariant across K∈{2,4,8}, while raw AGGREGATE accuracy fell from 0.250 to 0.222 to 0.037.The per-K results are reported in Table 6, with run-health details in Table 9.
  • G K-sweep and run-health details: 0 truncations occurred on every instruct and HMMT arm for truncated-no-answer and candidate-starvation counts.The only nonzero run-health counts occurred in RLVR.
  • G K-sweep and run-health details: 37/240 NO-CANDIDATE and 13/240 ANSWER-ONLY RLVR truncations occurred, versus 0 for AGGREGATE.These counts deflate the comparison arms.
  • G K-sweep and run-health details: 41/472 NO-CANDIDATE and 20/472 ANSWER-ONLY instruct truncations occurred in the process-level rerun, versus 3/472 for AGGREGATE.The corresponding rates were 8.7%, 4.2%, and 0.6%.
  • G K-sweep and run-health details: 0/472 truncations lacked an extractable answer across all arms, and first-answer scoring was unaffected by post-answer output.Candidate-free arms spend the budget reasoning from scratch, whereas AGGREGATE mostly reads.

H Extraction-scoring sensitivity analysis

A fresh process rerun shows that extraction asymmetry does not explain the main candidate-conditioning effects: the c0 deficit and c2+ gain replicate, while final-answer scoring changes the c2+ reasoning result. Candidate matching remains broadly stable, and the observed flip asymmetry is small and opposite to the proposed artifact.

  • Rerun validation: The rerun replicated all six per-benchmark Δcand signs and significance, including c0 Δcand −0.118 and c2+ Δcand +0.290 under first-answer scoring.The rerun generated 1,416 outputs under the primary settings, while committed results remained the primary reference.
  • Extraction asymmetry: The flip(AGG) −flip(NoCand) asymmetry was −0.032 [−0.086, +0.027], below the preregistered 0.05 margin and opposite the proposed narration artifact.AGGREGATE used the boxed tier for at least 99% of final-answer extractions.
  • Extraction sensitivity: Final-answer scoring strengthened the c0 Δcand deficit to −0.159 (p < 0.0001) while preserving the c2+ Δcand gain at +0.242 (p = 0.0003).The first-answer values were c0 Δcand −0.118 (p = 0.012) and c2+ Δcand +0.290 (p = 0.0002).
  • Extraction sensitivity: Final-answer scoring changed c2+ Δreas from +0.161 (p = 0.0013) to ≈0 (Holm p = 1.000), overturning its significance.Manual review found 24 ANSWER-ONLY c2+ rows where an earlier intermediate value caused first-answer extraction to miss a correct boxed answer.
  • Matching sensitivity: Matching attribution changed little: pooled c0 matching was 0.885 / 0.417 under first-answer scoring and 0.874 / 0.475 under final-answer scoring.The ordering retained at least 50% of the first-answer magnitude; 15 of 161 matching flags disappeared on final-answer review.

I Long-context placebo control (PlaceboLong) · J Identification notes · K Additional related work

The PlaceboLong control shows that off-topic context reproduces much of the c0 accuracy deficit, while the on-topic increment is not distinguishable from zero. Identification remains limited because causal decomposition and mechanism-specific recombination rates are not uniquely determined, although the candidate-free comparison connects the result to test-time scaling and self-improvement risks.

  • I Long-context placebo control (PlaceboLong): −0.164 [−0.241, −0.091] for PlaceboLong −NoCand shows a significant off-topic-context deficit at c0.The corresponding on-topic increment, AGG −PlaceboLong = +0.045 [−0.018, +0.118], has a confidence interval including 0.
  • I Long-context placebo control (PlaceboLong): Off-topic candidates reproduce much of the c0 deficit, weighing against a content-specific account under the pre-stated classification rule.AGGREGATE–PLACEBO is not statistically distinguishable from zero.
  • I Long-context placebo control (PlaceboLong): 104 of 192 wrong placebo outputs matched an off-topic displayed answer, a row-level rate of 0.542.Displayed-answer matching remains substantial even when candidate content is unrelated to the target problem.
  • I Long-context placebo control (PlaceboLong): Wrong-output matching was measured by whether an arm’s wrong answer equaled one of AGGREGATE’s displayed candidate answers; the paired comparison 33/4, p < 0.0001 was AIME-specific.This defines the matching statistic used in Figure 3.
  • J Identification notes: The placebo equality is algebraic, but topical-content and generic-context interpretations require placebo exchangeability, and internal mechanism components remain unidentified.Arm-level correctness contrasts do not identify a unique recombination rate because latent mixtures can differ.
  • K Additional related work: NO-CANDIDATE provides a fresh-solve comparison using another generation under the same maximum output-token allowance, though realized resources differ across arms.Prompt length, output length, latency, and computation are not equalized by this allowance.
  • K Additional related work: The c0 result illustrates a self-improvement risk: conditioning on wrong prior answers can reduce accuracy relative to a fresh solve.This relates the finding to methods that read and revise prior outputs, including Self-Refine, Reflexion, Tree-of-Thoughts, and STaR.

L Correcting for the adaptive extension … P The reasoning-trained variant

The adaptive correction preserves the c2+ and c0 rejections while c1 remains unresolved. Structured answer-field steering is measurable, masking is not, timing diagnostics invalidate raw-format comparisons, and the reasoning-trained variant remains exploratory.

  • M The structured answer-field intervention: details: Value-specific steering increases matching by +0.068, whereas masking changes accuracy by −0.005, with the structured intervention scoped to all-wrong sets.The corresponding 95% percentile confidence intervals are [+0.023, +0.118] and [−0.050, +0.036], respectively.
  • M The structured answer-field intervention: details: The structured intervention’s bridge check finds FULL-structured minus original-prompt AGGREGATE = +0.0227, with a confidence interval spanning zero.The passage identifies retained reasoning context, upstream answer occurrences, distraction, and mechanism interactions as potential contributors.
  • N Pipeline-rendered timing and resource accounting: Timing and resource tables use separate samples: frozen 30-set marginal timing versus 220-set full-grid token accounting, excluding candidate-generation cost and end-to-end latency.The timing result characterizes marginal post-pool generation under the tested execution.
  • O Raw-format timing diagnostic: Raw-format timing is invalid for pipeline comparison: templated requests had median output length 1,366 tokens, versus 16,384 for raw-format requests, with 27 of 30 raw requests truncated.The preregistered harness omitted the system message and chat template, producing substantially different termination behavior.
  • P The reasoning-trained variant: The reasoning-trained model has CGT 0.936 vs 0.379, placing 200 of 240 sets in c2+ and leaving c0 and c1 too sparse to assess training effects.The c0 sign is preserved, but the three-problem stratum is unpowered.
  • P The reasoning-trained variant: In c2+, the reasoning-trained variant shows ∆cand +0.120 and ∆reas +0.045, but model-class invariance remains unresolved and truncation may inflate ∆cand.NO-CANDIDATE has 37/240 no-answer truncations, ANSWER-ONLY 13/240, and AGGREGATE 0.

Q Full result tables

The full result tables report a consistent three-regime pattern across benchmarks, with additional analyses covering robustness to candidate-pool size, model class, validation criteria, and run health. Several table notes also qualify interpretation through power limitations and truncation-induced bias.

  • Statistical reading: Primary comparisons use paired problem-level exact sign tests, with ties dropped and Holm correction applied within the relevant family.A notation such as “10/4” counts discordant problems favoring the first versus second arm.
  • Cross-benchmark results: Table 5 finds the same three-regime sign pattern on both benchmarks, with no detected AIME↔HMMT heterogeneity in c0, c1, or c2+.The homogeneity p-values are c0 0.91, c1 0.55, and c2+ 0.70; benchmark-specific effects agree in sign.
  • K-sweep: At c0, raw AGGREGATE accuracy decreases as K grows, while ∆cand remains negative for every K.The sign of the candidate-conditioning effect is therefore invariant across the AIME-2025 K-sweep.
  • RLVR: In RLVR, 200 of 240 sets fall in c2+, while c0 and c1 contain only 3 and 2 problems, limiting model-class comparisons.CGT is 244/644=0.379 for instruct versus 395/422=0.936 for RLVR.
  • Validation experiment: The validation experiment uses constructed additive recombination on all-c0 sets and requires AGG to exceed both candidate-free arms by ≥0.30.This is the pre-specified success criterion for the validation table.
  • Run health: Run-health notes show instruct/HMMT had zero truncation issues, whereas RLVR non-aggregate no-answer truncations deflate affected arms and make reported c2+ effects upper bounds.The affected counts are NO-CANDIDATE 37 and ANSWER-ONLY 13 of 240; the instruct headline is unaffected.
Loading 2608.18379v1…