PaperlayerView exact revision on arXiv

Source-linked AI summary

When Verifiers Vote Backwards under Verdict Substitution: Signed Pivotal Value in Correlated Self-Consistency

Yang Shu

arXiv:2609.26144v1cs.AI

TL;DR

Prior work reported negative Bayes-optimal weights for process-reward-model scores but did not explain why an optimizer would vote against its verifier. This paper studies verdict-style ballot replacement with an exact signed-gain accounting, finding both helpful and harmful cases while the harm attenuates under the evaluated answer-identity plurality analysis.

  • Problem

    Prior work reported negative Bayes-optimal process-reward-model weights but did not explain why an optimizer would vote against its verifier.

    Source: §1 · p. 1

  • Method

    The paper derives an exact signed-gain decomposition and evaluates labeled verdict substitution in k=7 panels using state-specific verifier accuracy and pivotal-state composition.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    Across four configurations, results include helpful cases, a replicated harmful role-reversed configuration with a −11.2pp pivotal-region gain, and an exploratory harmful accounting case; answer-identity plurality attenuates role-reversed harm to a non-significant estimate.

    Source: §1 · pp. 1–2, §9 · p. 7

  • Takeaways & Limitations

    Pivotal structure identifies when substitution can matter, while state-specific accuracy and one-vote-state composition—not global accuracy or provenance—account for its sign.

    Source: §1 · p. 2, §9 · p. 7

    The LCB original configuration has only 14 pivotal rows across 9 tasks, so it is treated as an exploratory accounting case rather than evidence for a population effect.

    Source: §7 · p. 7

Abstract

from arXiv · show

Replacing one ballot can change a majority decision only on queries decided by a single vote; this structural fact requires no independence assumption. We study the sign of that change using a labeled, verdict-style intervention: one correctness signal replaces one correctness-indicator ballot in $k{=}7$ self-consistency panels. This diagnostic intervention is not identical to deployed answer-identity plurality. A primary MATH-500 experiment ($n{=}570$) gives a different-model verifier a $+24.2$pp pivotal gain, whereas a role-reversed configuration gives $-11.2$pp; an exploratory code stress test (14 pivotal rows across 9 tasks) gives $-24.5$pp. An exact signed-gain decomposition accounts for all observed signs through the verifier's state-specific accuracy and the composition of the two one-vote tally states, rather than global accuracy or model provenance. Same-source signals lose accuracy on the pivotal stratum (65$\to$44\% in the primary configuration), while error correlations provide a descriptive error-association diagnostic. Controlled degradation and a $k\in\{3,5,7\}$ subset sensitivity analysis probe the stability of the observed pattern around this accounting. Under the evaluated ties-incorrect answer-identity plurality analysis, the structural zero and strong-verifier benefit persist, but the role-reversed harm attenuates to $-0.9$pp and is not significant. The results therefore establish harmful verdict substitution, not universally harmful deployed plurality, and motivate a testable but unverified hypothesis for negative process-reward-model weights.

1 Introduction

Single-ballot substitution can change a majority only on one-vote queries, but whether that change helps is an empirical question. Experiments show both helpful and harmful verdict substitutions, with signed gain accounted for by state-specific accuracy and pivotal-state composition.

Introduction

  • Only one-vote queries can change under single-ballot substitution, and this structural fact requires no independence assumption.The open question is whether a structurally possible change has the correct sign.Source: §1 · p. 1
  • The study’s intervention replaces one correctness-indicator ballot with a correctness signal, so it is a labeled diagnostic rather than a directly deployable aggregator.The ballots require ground truth; answer-identity plurality is tested separately.Source: §1 · p. 1
  • +24.2pp pivotal-region gain came from a different-model verifier in the primary MATH-500 experiment, while a same-source extra draw gave 0.0pp and both signals had zero consensus-region gain.The primary pool contained 570 rows with zero exclusions.Source: §1 · p. 1
  • With DeepSeek-V4-Flash as engine and gpt-4o-mini as verifier, role reversal on the same 190-problem pool yielded a −11.2pp pivotal-region gain (two replications, n=380; naive and cluster bootstrap CIs excluded zero), while execution-graded LiveCodeBench yielded an exploratory −24.5pp estimate.Under verdict substitution, a different-model verifier flipped pivotal decisions in the wrong direction.Source: §1 · pp. 1–2
  • The exact signed-gain decomposition accounts for observed signs using state-specific verifier accuracy and pivotal-state composition, rather than global accuracy or model provenance.A globally accuracy-matched verifier’s accuracy fell from 87% to 36% on pivotal queries.Source: §1 · p. 2
  • The paper combines an exact accounting identity with helpful and harmful configurations, strength manipulation, subset sensitivity, and an answer-identity plurality analysis.Error correlation is treated as a diagnostic association, not a causal explanation.Source: §1 · p. 2

2 Related Work

Related work addresses correlated self-consistency, harmful verifiers, answer-identity voting, and negative PRM weights. The paper distinguishes its single-ballot accounting question from aggregate accuracy analyses and other panel-scoring methods.

Correlation ceilings for self-consistency

  • Bay and Yearick [2026] analyze aggregate sample counts and accuracy under correlated sampling, whereas this paper stratifies by per-query margin.Their effective-sample-size expression is n_eff = n/(1 + (n−1)ρ).Source: §2 · p. 2

Verifiers that hurt

  • He et al. [2026] report a negative verifier effect in multi-agent debate, while this paper supplies labeled counterexamples and an exact accounting decomposition without identifying a causal origin.Jiang [2026] stratifies panels by vote share but uses learned calibration rather than literal ballot substitution.Source: §2 · p. 2

Answer identities versus correctness indicators

  • Cai et al. [2026] study answer-identity trajectory basins; this paper separately re-analyzes math experiments using answer-identity plurality because collapsing wrong answers can alter which queries are contested.The re-analysis distinguishes conclusions that strengthen, survive, or attenuate.Source: §2 · p. 2

PRM aggregation

  • Kuang et al. [2026] report often-negative optimal PRM weights under conditional independence without a mechanism; this paper proposes a hypothesis and states what would falsify it.Source: §2 · p. 2

3 Structural Support Versus Empirical Sign

In binarized k=7 panels, single-ballot substitution can alter the majority only at a one-vote margin; the sign of its accuracy change depends on verifier accuracy in each pivotal state and how often each state occurs.

Structural support

  • Replacing one ballot changes the vote total by at most one, so only m_i=1 rows can change the majority.The panel uses seven temperature-0.8 samples from one model, with correctness indicators as votes.Source: §3 · p. 2
  • Figure 1 is conceptual: it depicts panel formation, ballot replacement, and the distinction between inert wider margins and signed gains on one-vote states.It contains no experimental measurements.Source: Figure 1 · p. 3
  • The structural zero in consensus regions is arithmetic and holds regardless of correlation, signal quality, or effect direction elsewhere.The paper treats the zero across 13 signal–experiment combinations as an implementation audit.Source: §3 · p. 3

Signed value

  • The exact signed gain depends on verifier accuracy in the S=3 and S=4 pivotal states and on their composition, not on one global accuracy.With S=3, substitution helps when a wrong vote is dropped and V=1; with S=4, it hurts when a right vote is dropped and V=0.Source: §3 · p. 3

The trap in the standard control

  • The standard different-model-versus-same-source control confounds provenance with strength because only the same-source signal is accuracy-matched to the panel in expectation.The paper points to negative checks, fixed-provenance strength manipulation, and attempts to give the same-source arm an advantage as ways to separate factors.Source: §3 · p. 3

4 Experimental Setup

The experiments use independently rerun math and code tasks, with specified sampling, grading, and completion procedures. The reported setup includes a significant role-reversed negative estimate, while the small code pivotal subset remains exploratory.

Domains, grading, completeness

  • The math set comprised 190 level-4/5 MATH-500 problems with a simple-numeric reference-answer filter; grading used exact match after final-answer normalization.Code experiments used 60 LiveCodeBench problems graded by sandboxed execution of held-out tests.Source: §4 · p. 3
  • The primary and role-swapped math pools completed all planned rows, while the LCB-original Signal A analysis lost 52 of 300 rows to non-extractable generations.The LCB-original analysis using both signals included 247 rows.Source: §4 · p. 3

Completeness

  • The primary analysis used the full 570-row pool after completing missing verifier verdicts; this raised pivotal gain from +23.3 to +24.2pp.The earlier 44 missing-verdict rows were not missing at random because their panels were harder.Source: §4 · p. 4

Token-budget escalation

  • The role-reversed configuration used token-budget escalation for empty outputs; on the 165-problem subset collectable without escalation, the estimate was −9.9pp with a confidence interval crossing zero.The full pool estimate was −11.2pp and significant.Source: §4 · p. 4

Configurations

  • Task-clustered intervals were [+12.2, +35.5]pp for primary, [−21.1, −1.1]pp for role-reversed, [−42.9, −5.7]pp for LCB original, and [+9.8, +26.4]pp for LCB strong.The LCB-original estimate is exploratory: 14 pivotal rows came from 9 unique tasks.Source: §4 · p. 4

5 Results

Across four verdict-substitution configurations, pivotal gains ranged from positive to harmful, and an exact accounting decomposition attributes their signs to state-specific accuracy and pivotal-state composition. Same-source strengthening, controlled degradation, subset sensitivity, answer-identity plurality, and descriptive association analyses qualify the pattern and its scope.

5.1 Evidence Across Four Configurations

  • +24.2pp pivotal gain for the different-model verifier on primary MATH-500 contrasts with 0.0pp for the same-source draw; both consensus gains were zero.The primary pool contained 570 rows, including 59 pivotal cases.Source: §5.1 · p. 4, Figure 2 · p. 4
  • Role reversal produced −11.2pp across 380 rows, while the exploratory code configuration yielded −24.5pp across 14 pivotal rows and 9 tasks.The code estimate is too small for a broad population claim; a targeted strong-verifier check instead yielded +17.5pp across 240 rows.Source: §5.1 · p. 4

5.2 Exact Signed-Gain Decomposition

  • The exact signed-gain decomposition uses state-specific verifier accuracies and the composition of pivotal states; same-source accuracy fell from 65.4% globally to 44.1% on pivotal queries.The accounting identity audits bookkeeping, not a predictive or causal model; error correlations are descriptive and may reflect shared task difficulty.Source: §5.2 · p. 4, Table 2 · p. 5

5.3 Attempts to Strengthen Same-Source Signals

  • Across five same-source constructions, aggregation and strengthened prompting did not produce a statistically reliable pivotal gain.In math, C7 and C17 accuracies were 64.0% and 63.9%, near the 64.1% single-call accuracy; prompting yielded +3.9pp, non-significant.Source: §5.3 · pp. 4–5, Table 3 · p. 5

5.4 Manipulating Strength Directly

  • +11.5pp pivotal gain remained for a degraded verifier at globally matched accuracy, while one-sided corruption made code gain cross negative at a −10pp gap.Pivotal gains for the different-model verifier were +23.0pp, +21.2pp, and +24.2pp at k=3, 5, and 7; smaller panels reuse banked draws.Source: §5.4 · p. 5, Figure 3 · p. 6

5.5 Robustness: Deployed Answer-Identity Voting

  • Under ties-incorrect answer-identity plurality, consensus gain remained zero and the strong-verifier gain was +15.0pp, while role-reversed harm attenuated to −0.9pp and was not significant.The claim of harmful voting applies to verdict substitution, not universally to deployed plurality.Source: §5.5 · p. 6

5.6 Thirteen Points, One Association

  • The 13-point association is descriptive, not regression evidence, because points share panels; the large LCB-original residual marks accuracy that does not survive pivotal conditioning.The dashed line is a visual aid, not a basis for regression inference.Source: §5.6 · p. 6

6 Discussion

The discussion offers a testable but unverified account of negative PRM weights, a checklist for verifier-value studies, and an arithmetic audit for structural-zero results.

6 Discussion

  • The proposed negative-weight account remains a hypothesis because actual PRMs and weighted score aggregation are untested.The authors propose testing trained PRMs’ conditional pivotal-state accuracies and formally bridging ballot substitution to weighted aggregation.Source: §6 · p. 6
  • Verifier-value experiments should report global and pivotal-state accuracy, measure dependence, and include a reversed configuration without an accuracy advantage.Pooled error association can reflect shared task difficulty, and the checklist notes that omitting role reversal could leave a clean provenance interpretation overturned.Source: §6 · p. 6
  • The consensus-region zero held to floating-point precision across configurations and voting schemes, making any deviation an implementation bug rather than an effect.The authors present this as an arithmetic identity, not a new empirical finding.Source: §6 · p. 6

7 Limitations

The limitations bound the panel-size and regression analyses, flag exploratory-inference and data constraints, and leave broader answer-identity settings untested.

7 Limitations

  • The k=3,5 sensitivity analysis reuses k=7 draws; independently generated smaller panels, larger k, and a systematic temperature sweep remain untested.Exploratory configuration-wise intervals are not multiplicity-adjusted, and role-reversed panels are independent rather than paired draws.Source: §7 · pp. 6–7
  • The 13-point gap–gain plot is descriptive, and shared panels preclude regression inference.Its dashed OLS line is descriptive; marker shape denotes provenance and error bars are naive-bootstrap 95% confidence intervals.Source: §7 · p. 6, Figure 4 · p. 7
  • The code case has only 14 pivotal rows across nine tasks, while answer-identity re-analysis covers math under one normalizer and a ties-incorrect policy.The code pipeline banked pass/fail verdicts but not program text, preventing retrospective behavioral clustering; alternative semantic-equivalence and tie policies are untested.Source: §7 · p. 7

8 Reproducibility and Ethics

The reproducibility package includes raw samples and materials to regenerate reported analyses; the study uses public benchmarks and commercial APIs without human subjects.

8 Reproducibility and Ethics

  • Raw per-call samples, analysis scripts, prompts, model identifiers, sampling details, bootstrap seeds, and a script-to-table map support reproducing reported figures and numbers.Figure 1 is an AI-assisted conceptual schematic checked by the author and encodes no experimental measurements; the authors report no direct ethical risks beyond general dual-use considerations.Source: §8 · p. 7

9 Conclusion

The conclusion separates where substitution can matter from whether it helps, and limits the harmful-plurality claim to the evaluated verdict-style setting.

9 Conclusion

  • An exact accounting identity relates signed pivotal gain to verifier accuracy in the two one-vote states and their composition, rather than global accuracy or provenance.The account describes observed signs without claiming correlation causes them.Source: §9 · p. 7
  • Under the evaluated ties-incorrect answer-identity analysis, plurality preserves the structural zero and strong-verifier benefit but reduces role-reversed harm to a non-significant estimate.The results establish harmful verdict substitution, not universally harmful deployed plurality.Source: §9 · p. 7
  • Testing actual PRMs, weighted aggregation, independently generated panel sizes, and broader answer-identity settings is needed to evaluate the negative-weight hypothesis.Source: §9 · p. 7
Loading 2609.26144v1…