Source-linked AI summary
Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
Sen Xu, Wei Wang, Shixi Liu, Jixin Min, Yingwei Dai, Zhibin Yin, Yirong Chen, Junlin Zhang
TL;DR
Existing reliability signals and whole-trace evaluation can miss decisive semantic errors in reasoning traces. CLR reallocates test-time compute from additional solution sampling to claim-level falsification and targeted verification, generally improving accuracy or token efficiency across evaluated models and benchmarks. Under matched budgets, it raises GPT-OSS-20B accuracy on CMIMC25 from 77.50% to 82.19% with 37.0% fewer generated tokens than Cons@64.
Problem
Existing confidence signals and whole-trace evaluation can miss localized, decision-critical errors, while step-level verification is computationally exhaustive and supervision-intensive.
Method
CLR extracts decision-critical claims from sampled traces, verifies them by searching for disconfirming evidence, and nonlinearly weights traces for consensus aggregation.
Results
CLR improves accuracy, token efficiency, or both across four LLMs and multiple reasoning benchmarks; on CMIMC25, it raises GPT-OSS-20B accuracy from 77.50% to 82.19% with 37.0% fewer generated tokens than Cons@64.
Takeaways & Limitations
Reallocating part of the test-time budget from additional generation to targeted falsification can improve the accuracy–token frontier when count-based consensus is noisy or weakly informative.
Takeaways & Limitations
CLR is not uniformly dominant at every operating point, with less headroom when the base consensus is already reliable.
Abstract
from arXiv · showhide
We propose claim-level falsification as a principle for test-time scaling and instantiate it through Claim-Level Reliability Assessment (CLR), a training-free framework that reallocates test-time compute from additional solution sampling to targeted verification. Since whole-trace evaluation often obscures decisive errors due to signal dilution from routine tokens, CLR condenses each reasoning trace into a compact set of decision-critical claims, thereby isolating its logical anchors. Furthermore, recognizing the inherent difficulty of generating entirely correct solutions under fixed model capabilities, CLR shifts the focus to semantic falsification. This approach exploits a fundamental asymmetry between solution construction and claim refutation. Constructing a valid solution requires a flawless reasoning path, whereas refuting an incorrect claim requires identifying only a single decisive flaw. This targeted search for negative evidence systematically compresses the survival space of high-confidence incorrect traces, effectively suppressing erroneous consensus via nonlinear reliability scoring. Across four LLMs and four reasoning benchmarks under matched budgets, CLR generally improves upon pass@1 and self-consistency. On GPT-OSS-20B/CMIMC25, for instance, CLR exceeds pass@1 by 27.15 percentage-points and raises self-consistency accuracy from 77.50\% to 82.19\% with 37.0\% fewer tokens.
1 Introduction
CLR is a training-free test-time scaling framework that replaces diluted whole-trace reliability assessment with claim-level falsification and targeted verification. Across evaluated models and benchmarks, it improves accuracy, token efficiency, or both under matched model-call budgets.
- Motivation: Existing reliability signals can mistake statistical confidence for logical reliability, while whole-trace evaluation dilutes decisive errors among routine reasoning tokens.Step-level verification can isolate errors but is computationally exhaustive and typically requires process-level supervision or separately trained verifiers.
- Approach: CLR instantiates claim-level falsification by condensing each reasoning trace into a compact set of decision-critical claims for more discriminative reliability assessment.The framework is training-free and designed for consensus-based answer aggregation.
- Inference pipeline: CLR samples K solution traces with claims, then independently verifies each trace’s claims using the original problem and extracted claims.Cons@K denotes self-consistency with K sampled traces, while CLR@K uses K generation calls and K verification calls.
- Results: 12.08 percentage points: CLR@32 raises Gemma-4-12B-it accuracy on HMMT25 from 76.67% with Cons@64 to 88.75%.This result is reported under matched model-call budgets.
- Results: 4.69 percentage points: CLR@32 improves GPT-OSS-20B accuracy on CMIMC25 from 77.50% with Cons@64 to 82.19% while using 37.0% fewer generated tokens.The passage also reports that CLR outperforms Pass@1 in this setting.
2 Method
CLR samples solution traces, extracts decision-critical claims, and assesses them for decisive flaws without using the original trace or final prediction. It converts claim outcomes into nonlinear reliability scores and aggregates equivalent predictions under matched request-count budgets.
- Pipeline: CLR independently samples K traces, each containing a prediction and exactly M ordered decision-critical claims, then assesses those claims in a separate Stage-2 request.Claims exclude generic summaries and prediction restatements, focusing on intermediate conclusions, constraints, decision points, transformations, or evidence links.
- Falsification-Based Claim Assessment: Stage 2 searches each claim for contradictions, counterexamples, errors, missing conditions, unsupported inferences, and cross-claim conflicts, encoding each outcome as refuted or not refuted.The VALID output means only that assessment found no refutation, not that the claim is formally proven correct.
- Reliability Scoring: The fraction of surviving claims becomes a trace reliability score whose nonlinear penalty increasingly suppresses traces as more decision-critical claims are refuted.This heuristic can reduce error-prone traces’ aggregate influence even when they are numerically dominant.
- Reliability Scoring and Aggregation: CLR groups parsed predictions by an appropriate equivalence criterion and selects the candidate with greatest reliability support, with sampling-order tie-breaking.When all traces have equal positive scores, the aggregation reduces to ordinary self-consistency; refuted claims otherwise reduce trace influence.
- Budget and Scope: Each CLR@K flow uses K generation requests and K claim-assessment requests, matching Cons@2K in request count but not necessarily in realized tokens.CLR only reweights parsed Stage-1 candidates and cannot recover a correct prediction absent from those samples.
3 Experiments
CLR is evaluated across four models and four reasoning benchmarks under matched model-call budgets, comparing claim-level falsification with self-consistency. Results show model-dependent accuracy–efficiency gains, substantial correction of erroneous consensus, and benefits from multi-claim assessment.
- Experimental setup: CLR is evaluated on four models—Gemma-4-12B-it, GPT-OSS-20B, GPT-OSS-120B, and Qwen3.5-27B—across HMMT25, HMMT26, CMIMC25, and Apex-shortlist.The primary matched-request comparison is CLR@32 versus Cons@64, with both methods using 64 model calls.
- Matched-budget results: 7.12–12.08 percentage points: CLR improves Gemma-4-12B-it accuracy across all four benchmarks, while increasing generated tokens by 22.2–47.8%.On HMMT25, accuracy rises from 76.67% to 88.75%.
- Matched-budget results: 14.5%: CLR’s largest token reduction for Qwen3.5-27B occurs on Apex-shortlist, while accuracy matches or improves by up to 2.60 percentage points.Qwen3.5-27B already exceeds 90% Cons@64 accuracy on three benchmarks, leaving less headroom.
- Consensus rescue: 16–48%: pooled rescue rates across 16 benchmark–budget settings quantify how often CLR overturns erroneous consensus when a correct candidate is present.The average rescue rate is about 37%, without changing the underlying model or adding a separate verifier.
- Claim-set ablation: 3.13–3.79 percentage points: increasing the claim-set size from M = 1 to M = 3 improves accuracy across all benchmarks, while further gains at M = 5 are task-dependent.Multi-claim settings use 14.1–17.7% more generated tokens than M = 1, and the ablation jointly changes semantic coverage, score resolution, and penalty sharpness.
- Scaling behavior: CLR can move the accuracy–token frontier outward when additional count-based samples provide noisy or weakly informative support, but it is not uniformly dominant at every operating point.The curves can cross at intermediate budgets, and the methods remain closer in near-saturated Qwen3.5-27B settings.
4 Conclusion
The paper presents claim-level falsification as a test-time scaling principle that constrains erroneous reasoning paths through decision-critical semantic anchors. CLR provides an initial instantiation and validation within consensus-based aggregation.
- 4 Conclusion: Claim-level falsification constrains erroneous reasoning paths using decision-critical semantic anchors.This allows reliable paths to exert greater influence.
- 4 Conclusion: Unlike forward-search approaches, the method does not rely on additional sampling or iterative revision to find a correct reasoning trace.
- 4 Conclusion: CLR provides an initial instantiation and validation of claim-level falsification within consensus-based aggregation.
A Experimental Details · A.1 Decoding Configuration
Experiments use each model’s default inference setup, with thinking enabled and default thinking effort, while applying the same sampling configuration to solution generation and claim-level reliability assessment. Model-specific decoding hyperparameters are summarized in Table 4, with other runtime settings left at defaults.
- A.1 Decoding Configuration: Inference uses each model’s default settings unless explicitly stated otherwise.
- A.1 Decoding Configuration: Thinking mode is enabled for all models.
- A.1 Decoding Configuration: Thinking effort remains at each model’s default.
- A.1 Decoding Configuration: Solution generation and claim-level reliability assessment use the same sampling configuration.
- A.1 Decoding Configuration: Table 4 summarizes the model-specific sampling hyperparameters used in the experiments.
- A.1 Decoding Configuration: All other model-specific runtime settings remain at their defaults.
A.2 Raw Counts for Rescue Rate
Table 5 provides the raw counts underlying Fig. 2’s CLR rescue rates. Counts use recoverable question–flow pairs across eight independent flows, with rescued cases reported over recoverable consensus errors.
- Counting unit: N = 8 independent flows determine the denominators, which count recoverable question–flow pairs rather than unique benchmark questions.A pair is recoverable when at least one Stage-1 candidate has the correct final answer but unweighted Cons@K is incorrect.
- Counting unit: The numerator counts recoverable pairs corrected by CLR using the same Stage-1 candidates.This numerator is paired with the recoverable consensus-error denominator.
- Raw counts: Table 5 reports rescued cases over recoverable consensus errors, followed by the rescue rate in parentheses.These raw counts underlie Fig. 2.
B Prompt Templates and Output Formats
Appendix B specifies the prompts, runtime placeholders, XML formats, and fallback rules implementing regular sampling and CLR’s two-stage claim-generation and verification process. Both stages use the same sampling parameters, while Stage 2 outputs ordered binary verdicts after actively testing claims for flaws.
- B.1 Regular Solution Sampling: Base sampling uses a step-by-step prompt that places the final answer within \boxed{}; CLR Stage 1 extends it with claim-generation instructions.The benchmark problem replaces {question}, while {num_claims} specifies the number of claims and verdicts.
- B.2 Stage 1: Solution and Claim Generation: Stage 1 generates a step-by-step solution, places the final answer in \boxed{}, and appends exactly {num_claims} verification claims.Claims must be intermediate mathematical statements with relevance, diagnostic value, non-triviality, and clarity.
- B.2 Stage 1: Solution and Claim Generation: Stage 1 requires the final answer before the claims, followed by exactly {num_claims} XML-formatted <claim> entries enclosed within <claims> tags.The compact {claim_tags} placeholder is assembled dynamically at runtime.
- B.3 Stage 2: Claim-Level Reliability Assessment: Stage 2 presents the extracted claims to a rigorous verifier, which stress-tests them and returns an ordered binary verdict for each claim.The runtime {claims_block} contains the claims extracted in Stage 1.
- B.3 Stage 2: Claim-Level Reliability Assessment: The verifier treats every claim as suspicious and actively seeks contradictions, conflicts, unjustified assumptions, counterexamples, calculation errors, or logical errors.It must not assume truth merely because no immediate counterexample is found.
- B.3 Stage 2: Claim-Level Reliability Assessment: Stage 2 ends with exactly {num_claims} ordered XML verdict tags, each containing one token: VALID or REFUTED, with no text after </verdicts>.The verdict tags are assembled dynamically using the {verdict_tags} placeholder.
- B.3 Stage 2: Claim-Level Reliability Assessment: Before verification, claim line breaks are collapsed and claims are numbered; malformed extraction pads missing claims with empties, while missing or unparseable verdicts default to not refuted.The dynamically assembled claims block lists each claim in order from 1 through {num_claims}.