PaperlayerView exact revision on arXiv

Source-linked AI summary

Does CoT-Pass@k Really Check the CoT? A Multilingual Mathematical Audit

Tarık Tuna Taşaltı, Burcu Hüdaverdi, David Semedo

arXiv:2609.32622v1cs.CL

TL;DR

CoT-Pass@k aims to distinguish sound reasoning from lucky correct answers by requiring an LLM judge to approve the reasoning chain, but that verification assumption had not been tested within the metric or beyond English. This paper audits the metric across multilingual mathematics benchmarks using controlled error edits. The judges often accept corrupted chains, and the metric’s difference from Pass@k is small on current-generation solvers.

  • Problem

    CoT-Pass@k relies on a judge to verify reasoning, but its ability to detect flawed chains had not been audited inside the metric or outside English.

    Source: §1 · p. 1

  • Method

    The paper audits CoT-Pass@k under its own protocol across five English, Turkish, and Portuguese mathematics benchmarks, using deterministic edits that separately corrupt chains and final answers.

    Source: §1 · p. 1, §3 · p. 3, §4.2 · p. 4

  • Results

    Chain corruption barely changes acceptance, and the average Pass@k − CoT-Pass@k difference falls from 19.7 points on an earlier solver generation to 4.1 on the current one.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The audit finds that the tested judges fail to reliably detect reasoning errors and concludes that judged reasoning metrics should be audited against inputs with known correct verdicts.

    Source: §6 · p. 9

    The current-generation solvers reach only 9B parameters, so the observed collapse is not directly verified on larger current models.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Pass@k measures whether a model reaches a correct answer under repeated sampling, but never how: a lucky guess counts the same as sound reasoning. CoT-Pass@k was proposed to close that gap, adding an LLM-as-judge that must assess a solution's reasoning chain before it counts. Its value rests entirely on one assumption: that the judge catches flawed reasoning. That assumption has never been tested inside the metric that depends on it, and never outside English, though the metric's claims concern models used in many languages. We report the first audit of that verification step, run under the metric's own protocol on a multilingual suite of five mathematical benchmarks in English, Turkish and Portuguese, two of them natively written. We corrupt correct solutions with deterministic edits that damage the chain and the final answer separately. We observe that all three judges accept corrupted chains almost as often as clean ones. V4-Flash and Qwen3.6 reject a solution sharply only when its final answer is wrong and accept a wrong answer more readily when the chain agrees with it; the metric's own judge accepts most wrong answers as well. Our study shows that chain-answer agreement dominates the two larger judges' verdicts and that all three fail to reliably detect the tested reasoning errors. Consequently the difference Pass@k - CoT-Pass@k averages 19.7 points on an earlier solver generation but only 4.1 on the current one. What little remains depends on the token budgets on both sides and on the generation mode; raising the generation budget moves Pass@64 by more than fifty points while the difference stays at zero. We close with two checks any judged reasoning metric should pass before its numbers are read as evidence about reasoning.

1 Introduction

The introduction questions whether CoT-Pass@k’s judge actually verifies reasoning, since the metric counts a correct answer only after judge approval. A multilingual audit finds that the judges often accept corrupted chains, and the metric adds little over Pass@k on current-generation solvers.

1 Introduction

  • Pass@k treats lucky guesses and sound reasoning alike, motivating CoT-Pass@k’s judge-based attempt to distinguish them.The audit tests the assumption that the judge can identify flawed reasoning.Source: §1 · p. 1
  • The audit tests CoT-Pass@k under its own protocol with deterministic edits that separately corrupt reasoning chains and final answers across five mathematical benchmarks in English, Turkish, and Portuguese.The suite includes natively written Turkish and Portuguese benchmarks.Source: §1 · pp. 1–2
  • Judges accept corrupted reasoning chains almost as often as clean ones, while V4-Flash and Qwen3.6 respond sharply to wrong final answers and the metric’s own judge accepts most wrong answers.The larger judges’ verdicts are dominated by chain–answer agreement.Source: §1 · pp. 1–2
  • The average Pass@k − CoT-Pass@k difference falls from 19.7 points on an earlier solver generation to 4.1 on the current one.The introduction also reports that raising the generation budget moves Pass@64 by more than fifty points while the difference stays at zero.Source: §1 · p. 2
  • The paper contributes a two-generation comparison, question-level uncertainty analysis, and recommendations for reporting and auditing judged reasoning metrics.It presents the work as the first audit of CoT-Pass@k’s verification step under the metric’s own protocol.Source: §1 · p. 2

2 Related Work

Prior evaluation approaches either score final answers without inspecting reasoning chains or assess judges separately from the metrics that use their verdicts. This paper positions its audit as connecting judge behavior to reported metric values across languages and solver generations.

2 Related Work

  • Pass@k measures coverage as the sample budget grows, while reliability-threshold and posterior-success alternatives still do not inspect the reasoning chain.CoT-Pass@k instead requires an LLM judge to approve the chain before a solution counts.Source: §2 · p. 2
  • Existing Turkish and Portuguese benchmarks do not measure Pass@k, and only PHEB grades written solutions against official rubrics that this suite does not use.MCLM translates AIME 2024 into 55 languages, including Turkish, but scores final answers alone.Source: §2 · p. 2
  • Existing process-reward benchmarks test judges against known step errors, but their verdicts do not become numbers reported by a deployed metric.Other multilingual error-injection studies likewise score models against a known key rather than tracing judge verdicts into a published metric.Source: §2 · p. 2
  • Earlier judge-dependent studies report biases, weak step localization, and credit for plausible-looking reasoning, but they are English-only and stop at the judge.Sun et al. (2026) and Mittal and Arike (2026) are described as closest in method; neither audits a published metric under its own protocol.Source: §2 · p. 3
  • The paper identifies its gap as auditing a published judged metric under its own protocol, following verdicts into reported numbers, and connecting those numbers to token budgets and generation mode.The cited concurrent studies are English-only and assess judges rather than the metric’s output.Source: §2 · p. 3

3 Multilingual Benchmarks

The benchmark suite spans English, Turkish, and Portuguese, pairing translated problems with natively written sets in the two non-English languages. Its construction supports comparisons that separate language changes from translation artifacts, across competition and high-school mathematics.

3 Multilingual Benchmarks

  • Five benchmarks cover English, Turkish, and Portuguese, combining AIME 2026, its Turkish and Portuguese translations, the 2026 Turkish olympiad, and Portuguese PHEB questions.The translated AIME problems were machine-translated and manually audited by native speakers; the PHEB multiple-choice questions were converted to open-ended form.Source: §3 · p. 3
  • Each non-English language has one translated and one native benchmark, allowing language to vary while problems stay fixed and allowing native sets to avoid translation artifacts.All Portuguese material is European Portuguese.Source: §3 · p. 3
  • The suite ranges from competition mathematics to high-school exams that are nearly saturated by current models, and the authors describe the Turkish olympiad as the least exposed benchmark.The Turkish olympiad postdates the public release of the evaluated models; the Portuguese exams predate them all.Source: §3 · p. 3

4 Method

The method defines the solvers, judges, scoring rules, and controlled comparisons used to audit CoT-Pass@k. It separately tests judge responses to injected errors, compares solver generations under a fixed setting, and measures how token limits affect scores.

4.1 Solvers, Judges and Metrics

  • Pass@k counts a question as successful when at least one sampled generation is correct, while CoT-Pass@k additionally requires judge approval of the reasoning chain.At k = n, their difference is the share of questions with a correct solution but no approved one.Source: §4.1 · p. 4

4.2 Error Injection

  • The audit creates 3,600 variants from 720 correct solutions, using deterministic edits for corrupted chains with correct answers, wrong answers with unchanged chains, and wrong answers made consistent throughout the chain.The judges score the clean controls and four error types; comparisons are paired on the same solutions.Source: §4.2 · pp. 4–5
  • Error types separate chain validity from answer correctness: rejection of chain-corrupted solutions with correct answers tests chain assessment, while extra acceptance of consistent wrong answers tests sensitivity to chain–answer agreement.The edits are deterministic string changes, not language-model-generated corruptions.Source: §4.2 · pp. 4–5

4.3 Cross-Generation Design

  • The cross-generation comparison fixes Qwen3.6 as judge, uses a 16k generation budget, and runs the four 64-sample benchmarks through k = 64.The design compares base checkpoints and instruction-tuned non-thinking counterparts; it excludes Portuguese exams and thinking mode from this grid.Source: §4.3 · p. 5

4.4 Token-Budget Measures

  • Solver generations that reach the token limit before stating a final answer are graded wrong, while judge runs that emit no verdict cannot contribute an approval.A counterfactual that counts stopped judgments as approvals bounds the contribution of judge token limits to the Pass@k − CoT-Pass@k difference.Source: §4.4 · p. 5

5 Results

The audit finds that judges largely miss injected chain errors, while CoT-Pass@64 nearly converges with Pass@64 for current solvers. Raising generation budgets increases Pass@64 without changing the difference in the tested budget ladder.

5.1 The Judges Miss the Injected Reasoning Errors

  • 18%, 34%, and 71% of final-answer-error variants are accepted by V4-Flash, Qwen3.6, and R1-distill, while intermediate numeric errors produce no measurable acceptance shift.Changing only the final answer costs V4-Flash and Qwen3.6 75 and 64 points, respectively; the metric’s judge accepts most wrong answers.Source: §5.1 · pp. 5–6
  • 25% and 54% of consistent wrong-answer variants are accepted by V4-Flash and Qwen3.6, showing that chain–answer agreement dominates their verdicts.R1-distill accepts 62% of consistent variants versus 71% of plain wrong-answer variants; none of the judges reliably detects the tested step errors.Source: §5.1 · p. 6

5.2 CoT-Pass@k Collapses onto Pass@k in Current Solvers

  • 19.7 points for earlier-generation solvers versus 4.1 for current-generation solvers is the average Pass@64 − CoT-Pass@64 difference across four benchmarks.Figure 2 displays CoT-Pass@64 and Pass@64 by solver and benchmark under Qwen3.6 at 16k.Source: Figure 2 · p. 7, §5.2 · p. 7
  • 2.7 points versus 2.0 at k = 1 grows to 19.7 versus 4.1 at k = 64 for the earlier and current generations, respectively.The generations diverge as repeated sampling adds correct answers carrying chains the judge rejects in the earlier generation, but not the current one.Source: §5.2 · p. 7

5.3 Raising the Budget Moves the Scores, Not the Difference

  • Gemma4’s token limit can account for an upper-bound 92.7% of its 5.7-point residual in thinking mode, while Qwen3.6 reports a 1.4-point difference on the same solutions.Stopped judgments cannot contribute approval; the collapse itself appears under either judge.Source: §5.3 · p. 8
  • Pass@64 rises from 40.0 to 96.7 on English AIME as the generation budget increases, while Pass@64 − CoT-Pass@64 remains zero in all sixteen tested cells.The high budget is 32k, except 64k on English AIME; the ladder uses the anchor judge with a budget that rises with the solver’s.Source: §5.3 · p. 8

6 Discussion and Conclusion

The audit finds that judges often accept corrupted chains, while chain–answer agreement dominates two larger judges’ verdicts. CoT-Pass@k adds little over Pass@k on current-generation solvers, motivating checks before judged scores are treated as reasoning evidence.

6 Discussion and Conclusion

  • The average Pass@k–CoT-Pass@k difference falls from 19.7 points on an earlier solver generation to 4.1 on the current generation.On current-generation solvers, the metric returns what Pass@k already returns.Source: §6 · p. 9
  • Two checks are proposed: publish judge responses to injected chain errors, and report the judged metric beside its judge-free counterpart across token budgets and generation modes.The difference between the metrics is what the judge adds.Source: §6 · p. 9
  • The metric separates the earlier generation, but harder problems do not restore its meaning because judges still fail to react to corrupted chains.The original study also observed smaller separation on benchmarks the base model already solves.Source: §6 · p. 9
  • On current-generation solvers, CoT-Pass@k conclusions inherit Pass@k evidence, so claims about reasoning require instruments that measure the chain directly.The conclusion says the verification step fails to reliably detect the tested errors and that remaining metric differences depend largely on token budgets and max-token stopping.Source: §6 · p. 9
  • In the tested settings, Turkish and Portuguese do not make verification harder than English; native benchmarks, rather than language alone, shift some rates.The translation approval rates are within 3 points of the original or above it on average over solvers.Source: §6 · p. 9
  • All three judges fail to reliably detect the tested reasoning errors; V4-Flash and Qwen3.6 verdicts are dominated by chain–answer agreement.The metric’s own judge also approves most wrong answers.Source: §6 · p. 9

Limitations

The audit’s limitations concern the realism and coverage of its error edits, the scale of tested solvers and judges, and unresolved benchmark differences. Its findings are bounded to the tested mathematical settings.

Limitations

  • Deterministic string edits make verdicts known by construction and reproducible, but may look like slips rather than broken reasoning.The authors argue that this realism cost would understate, not overstate, how little judges assess if the edits are too mild.Source: Limitations · p. 10
  • Injected chains come from two Qwen3.5 thinking-mode solvers, and non-English problems are assessed with English or mixed reasoning chains.The authors did not manually check whether every edited number affects subsequent steps.Source: Limitations · p. 10
  • Whether an edit in an auxiliary line propagates through later steps was not checked by hand, though the consistent final-answer condition does not require that check.Source: Limitations · p. 10
  • Current-generation solver tests reach 9B parameters, and the judges do not reach any lab’s flagship tier.The original judge’s reasoning was not audited, leaving open whether its verdicts rely on solution form rather than content.Source: Limitations · p. 10
  • The Turkish olympiad’s residual difference cannot be separated from its native language, limited public exposure, or olympiad-level difficulty.The audit covers three languages in mathematics only, so it makes no claim about code or domains without a single verifiable answer.Source: Limitations · p. 10

Ethics Statement

The work evaluates publicly released language models on published mathematical exams for research evaluation, without personal data or human subjects.

Ethics Statement

  • The study uses public models and published mathematical exams for research evaluation; it involves no personal data or human subjects.Translations were audited by an author and volunteer native speakers, and corrupted solutions were used only as judge test inputs.Source: Ethics Statement · p. 10

A Experimental Setup

Solver prompts pair each benchmark problem with a step-by-step instruction in the benchmark’s language. The judge uses the reproduced English prompt, with two specified additions in Turkish and Portuguese.

A.1 Prompts

  • Each solver receives the problem followed by a step-by-step instruction in the benchmark language, with Turkish and Portuguese versions specified for their respective benchmark sets.Source: §A.1 · p. 13
  • The English judge prompt reproduces Wen et al. (2026), fills in the problem and solution, and requires a yes-or-no boxed verdict.Figure 3 reproduces the English prompt.Source: §A.1 · p. 13, Figure 3 · p. 14
  • Turkish and Portuguese judge prompts add a problem-language tag and a task instructing judges to ignore language switching and assess mathematical and logical correctness.The remaining task list, error taxonomy, output format, and verdict convention are unchanged.Source: §A.1 · p. 13

A.2 Configuration

The evaluation uses repeated solver generations and three judgments per correct solution, with fixed judgment budgets subject to stated exceptions. The configuration also defines how injected-error solutions are selected and how verification strategies combine judgments.

A.2 Configuration

  • Solvers draw 64 generations per question, or 16 on Portuguese exams, while judgment budgets are 16,384 tokens except for V4-Flash and the budget ladder.V4-Flash uses 20,480 tokens; in the error-injection study, all three judges use that setting.Source: §A.2 · p. 14
  • Judging every answer-correct generation three times raises evaluation cost substantially: Qwen3.6 and Gemma4 emit 0.82 and 0.92 tokens per solver token, respectively.Across the judge matrix, 133,434 judgments produced 797 million judge output tokens; judging one correct solution costs 10 to 24 thousand tokens.Source: §A.2 · p. 14
  • Each error-injection solution is judged three times, and any-correct, majority-correct, and all-correct combine those verdicts using different approval thresholds.A judgment without a verdict counts as rejection under any-correct and all-correct, but is omitted under majority-correct.Source: Table 1 · p. 15
  • Injected-error originals are correct generations longer than 600 tokens that stopped before the max-token limit and support all four error types.The study adds 32,400 judgments; recorded cluster jobs account for about 620 H200 GPU-hours, a lower bound.Source: §A.2 · p. 15

B CoT Verification Strategies

Across the tested verification strategies, judges largely miss injected reasoning errors, while wrong final answers can sharply reduce acceptance for some judges. The later comparisons examine the judge matrix, sampling uncertainty, acceptance of correct generations, and saturation at k = 64.

B CoT Verification Strategies

  • Under V4-Flash and Qwen3.6, consistent wrong-answer variants are accepted more often than plain wrong-answer variants, while R1-distill orders them oppositely under all three verification strategies.All-correct separates the groups most for V4-Flash and Qwen3.6, but also rejects more unedited controls than any-correct.Source: B CoT Verification Strategies · p. 15
  • 86 to 100% acceptance of intermediate numeric errors across every judge and edit cell shows that changing edit kind or size does not make these errors detectable.Moving the error between 40% and 70% of the chain changes acceptance by at most 3 points; Figure 4 partitions the conditions by edit kind and relative size.Source: B CoT Verification Strategies · p. 15, Figure 4 · p. 16
  • A wrong final answer reduces R1-distill acceptance by 15 to 27 points on every benchmark, whereas the intermediate numeric error shifts none significantly.The intermediate-error comparison has a smallest exact McNemar p = 0.30; R1-distill’s wrong-answer losses have p < 10^-3.Source: B CoT Verification Strategies · p. 17

C.1 The Judge Matrix

  • Tables 2 and 3 report Pass@64 − CoT-Pass@64 by solver under four judges for earlier-generation models and two judges for current-generation models.The supplied prose frames these tables as the per-model form of the finding that the collapse does not depend on the judge.Source: §C.1 · p. 17, Table 2 · p. 17, Table 3 · p. 17

C.2 The k Ladder and Question-Level Uncertainty

  • Qwen3.5-9B-Base has zero difference on all four benchmarks, so its question-bootstrap interval collapses to [0, 0] without implying zero uncertainty.This is a structural limit of resampling an all-zero indicator.Source: §C.2 · p. 17

C.3 Acceptance of Correct Generations

  • Earlier generations’ answer-correct acceptance rates were 57–70%, versus 80–99% for current generations per solver, across the reported judges and benchmarks.On the native Turkish olympiad, the anchor judge approves 39–47% of earlier-generation correct generations and 75–99% of current-generation ones.Source: §C.3 · p. 18

C.4 Saturation and Conditioning on c

  • At c = 1–2, CoT-Pass@64 is 24 versus 68 and acceptance is 23% versus 66%; at c = 33–64, they are 83 versus 100 and 74% versus 96%.At every fixed c, the current generation retains more questions and more correct generations; saturation explains only how its remaining rejections become a difference of zero.Source: §C.4 · p. 18

C.5 Language of the Chains

Chain language varies chiefly by solver, and within-solver comparisons do not show a consistent acceptance advantage for English or mixed chains. The judge matrix also samples judgment-language outputs, while the max-token table tracks stops and the resulting metric difference.

C.5 Language of the Chains

  • Qwen3.5 base checkpoints write 79 to 93% of non-English benchmark chains in English, while instruction-tuned solvers of both generations use the benchmark language at least 97% of the time.The language classes largely coincide with solver identity, limiting cross-solver comparisons by chain language.Source: §C.5 · pp. 18–19
  • Within Qwen3.5 base cells, English and mixed chains are accepted within 3 points of one another; Qwen2.5-32B on the Turkish olympiad is accepted 48% in English and 35% in Turkish.Acceptance rates cannot be read across solvers as a language effect because chain language tracks the solver.Source: §C.5 · p. 19
  • On the Turkish benchmarks, all four judges write 78 to 100% of sampled judgments entirely in English and none in the benchmark language, regardless of chain language.R1-distill is an exception in its hidden reasoning: it thinks mostly in Turkish in 40 to 47% of judgments, while its visible verdict stays in English.Source: §C.5 · p. 20, Figure 9 · p. 20

D.1 Stops at the Max-Token Limit

  • Table 4 reports, for each benchmark and solver, the shares of solver generations and judge judgments that stop at the max-token limit, together with the difference reported by each judge.Source: §D.1 · p. 20

D.2 Judge Budgets and the Difference

The paper separates the judge-verdict component of the Pass@64 − CoT-Pass@64 difference from the contribution of judgments stopped at the token limit. Its budget and mode comparisons are bounded by the conditions under which the runs were completed and reported.

D.2 Judge Budgets and the Difference

  • For four Qwen2.5 solvers on the same four benchmarks at the 16k solver budget, crediting stopped judgments as approvals leaves at least 15.2 of the anchor judge’s 19.7 points; minima are Gemma4 12.5/19.9, R1-distill 18.6/20.1, and V4-Flash 20.1/21.5.Source: §D.2 · pp. 20–21

D.4 Generation Mode and Language

  • Turning thinking on moves every chain out of the benchmark language, and raising the generation budget shifts mixed chains further toward English.The language comparison uses the same thirty AIME problems in English and translated Portuguese and Turkish, with both modes and the 16k budget; the passage cautions that language comparisons depend on mode and budget.Source: §D.4 · p. 22, Figure 13 · p. 23, Table 6 · p. 23

D.3 The Budget Ladder

  • The budget ladder pairs the same solver and benchmark at 16,384 tokens and raised budgets of 32,768, or 65,536 on English AIME.The ladder uses thinking mode and reports stopping shares, Pass@1, and Pass@64; Gemma4 runs could not be completed because it stopped at its own limit so often that serving slowed.Source: §D.3 · pp. 21–22, Table 5 · p. 23

Related papers

Loading 2609.32622v1…