PaperlayerView exact revision on arXiv

Source-linked AI summary

The Answer Is Not the Argument

Will Yeadon, Sergio Juárez, Paul Mackay, T. J. Dowling, Elise Agra, Oto-obong Inyang, Arin Mizouri, Craig P. Testrow

arXiv:2609.00264v1cs.AI

TL;DR

The paper asks whether trusted answers improve independent verification of chain-of-thought reasoning or mainly expose incorrect conclusions. It evaluates naturally generated, independently adjudicated physics traces under answer-information conditions and finds that certification improves aggregate monitoring chiefly through conclusion-consistency checking. The study cautions that ordinary physics errors, answer-conditioned ground truth, and limited online comparability constrain broader safety conclusions.

  • Problem

    The paper asks whether giving monitors a trusted reference answer improves reasoning verification or mainly exposes incorrect conclusions.

  • Method

    The study evaluates 237 step-numbered solutions to 79 HLE physics questions from three frontier models using independently labelled answers and first false steps, with eight monitors tested across answer-information conditions.

  • Results

    Certification raised mean balanced accuracy from 0.637 to 0.796 and exact first-error localization from 0.261 to 0.379, with recall gains concentrated on wrong-answer traces rather than critical traces.

  • Takeaways & Limitations

    Trusted-answer access improves conclusion-consistency checking more than independent verification of the supporting argument.

  • Takeaways & Limitations

    The errors are ordinary model mistakes in physics rather than adversarial sabotage, and the reference standard itself had access to the reference answer.

Abstract

from arXiv · show

Chain-of-thought monitoring is proposed for AI oversight, yet evaluations often provide monitors with a trusted reference answer. We ask whether answer access improves reasoning verification or mainly exposes incorrect conclusions. We collected 237 step-numbered solutions to 79 Humanity's Last Exam physics questions from three frontier models, with no inserted errors, and independently labelled final-answer correctness and the first false step. The reference standard combined physicist annotations, an independent LLM debate, and source-masked adjudication. This yielded 24 critical traces in which the answer was correct but the trace contained a genuine error. 8 LLM monitors evaluated traces blind, with an unverified or certified answer, or after a blind commitment. Certification raised mean balanced accuracy from 0.637 to 0.796, while exact first-error localization rose from 0.261 to 0.379. Certification changed recall (the fraction of error traces flagged as erroneous) from 0.653 to 0.951 on wrong-answer traces but from 0.521 to 0.438 on critical traces; the contrast had the same direction for all 8 monitors (question-bootstrap 95% CI [+0.256, +0.506]). After blind commitment, monitors shown the answer newly flagged 93.8% of previously passed wrong-answer traces as erroneous, but only 18.0% of critical traces. Answer access therefore improves conclusion-consistency checking rather than independent verification of the supporting argument. For AI safety, these traces provide a benign analogue of reward hacking: an acceptable output does not establish that the process producing it was sound. Although the errors studied here were ordinary and mostly non-load-bearing rather than adversarial, trusted-answer evaluations may similarly overstate monitoring capability when acceptable outputs conceal unsound reasoning.

I. INTRODUCTION

The study tests whether reference-answer access helps monitors verify reasoning or mainly identify conclusions that disagree with the answer. It uses naturally generated, independently labelled physics traces and varies answer information only during monitoring.

  • Motivation: Chain-of-thought monitoring lets one language model inspect another model’s written reasoning as a possible layer of AI oversight.Its value depends on how reliably it can detect failures under deployment-relevant conditions.
  • Motivation: Correct-answer traces containing flawed reasoning are the key test because an acceptable conclusion can conceal an unsound process.They provide a benign analogue of reward hacking, where output quality does not reveal process quality.
  • Dataset: The study filters Humanity’s Last Exam physics questions to text-only, exact-match, genuinely multi-step items, leaving 79 questions.The filtered generating models reached correct final answers on 65% of traces, preserving both positive and negative cases.
  • Study design: 237 step-numbered solutions from three frontier models are independently labelled for final-answer correctness and first false step.Eight monitors evaluate each trace while the reference answer is absent, unverified, certified, or revealed after blind commitment, with additional controls.
  • Study design: The evaluation separates error detection from first-error localization and distinguishes wrong-answer traces from correct-answer traces that conceal errors.This decomposition tests argument verification separately from answer checking.

B. Ground truth

The ground-truth process labels both final-answer correctness and the earliest genuine false step, producing a critical set of correct-answer traces with flawed reasoning. Because localization judgments disagree substantially, the final standard combines expert and independent model review through masked adjudication.

  • Ground-truth labels: A critical trace has a correct final answer but a genuine error in its numbered reasoning, which conclusion-only monitoring would always pass.The study treats these traces as the central test of independent argument verification.
  • Ground-truth labels: The first error step is the earliest numbered commitment to a false claim about the problem, relevant physics or mathematics, or prior reasoning.Exploration that is tested and discarded is not itself an error; unsupported plausibility is not labelled false solely for incomplete justification.
  • Adjudication: The reference standard combines physicist annotations with an independent LLM debate and source-masked adjudication.Physicists covered 174 of 237 traces, while the independent model arm annotated every trace.
  • Adjudication: The two review arms agreed on final-answer correctness for 95.6% of jointly covered traces and on error presence for 79.3%.When both named a step, exact agreement was 51.4%, motivating explicit adjudication of localization targets.
  • Final dataset: The resulting 237-trace standard contains 130 clean correct-answer traces, 83 wrong-answer traces, and 24 critical traces.Sixteen of the 24 critical traces carry labels set by author adjudication.

C. Monitoring conditions

Monitoring conditions vary the epistemic status and timing of the reference answer while keeping the generated trace fixed in whole-trace evaluations. The study measures flagging, first-error localization, and uncertainty with clustered resampling and cross-monitor consistency tests.

  • Information ladder: BLIND provides only the problem and complete numbered trace, without a reference answer.It is the answer-blind baseline for monitoring reasoning directly.
  • Information ladder: HINT adds the reference answer as an unverified candidate, whereas CERT presents it as correct with an explicit warning that correctness does not imply sound reasoning.These conditions isolate the effect of answer information and its epistemic framing.
  • Controls: REVISE replays the monitor’s blind verdict before revealing the certified answer, while RCTRL asks for reconsideration without revealing the answer.The pair separates certification from the effect of being prompted twice.
  • Online monitoring: STEP reveals one trace step at a time and takes the union of per-step error decisions as the trace verdict.It hides future context, segments the trace, and requests a decision at every step.
  • Metrics: Balanced accuracy averages sensitivity and specificity, while exact localization L0 requires both flagging an error and naming its correct first step.Cell-restricted recall is reported separately for wrong-answer and critical traces.
  • Uncertainty: Question-level cluster bootstrapping resamples all three generator traces together across 2,000 resamples, and sign tests assess consistency across monitors.The design avoids treating traces sharing a question as independent observations.

III. RESULTS

Certification substantially improves error detection, but its benefit is concentrated on traces whose wrong conclusions reveal that an error exists; localization improves much less, and monitor identity remains influential.

  • Overall detection and localization: 0.637 → 0.796 mean balanced accuracy from BLIND to CERT, while exact localization rises only from 0.261 to 0.379.The detection metric improves more than the metric requiring the correct erroneous step.
  • Overall detection and localization: Certification makes monitors flag more accurately in both directions, with mean specificity rising from 0.655 to 0.757 and sensitivity from 0.619 to 0.836.The increase is not explained simply by flagging more traces.
  • Monitor variation: Monitor identity matters more than condition: within-condition balanced-accuracy spreads reach 0.272 under CERT and 0.358 under HINT.Several stronger monitors under BLIND outperform weaker monitors given certification.
  • Answer-diagnostic traces: +0.299 mean recall on wrong-answer traces from BLIND to CERT, with all eight monitors reaching 0.855–0.988 under CERT.Certification closes 75.9% of the distance between blind recall and perfect recall on this cell.
  • Answer-diagnostic traces: -0.083 mean recall on critical traces from BLIND to CERT, with the confidence interval including zero and declines for six of eight monitors.The wrong-answer versus critical-trace gain difference is +0.382, positive for all eight monitors.
  • Answer-diagnostic traces: HINT lies between BLIND and CERT for most monitors on wrong-answer traces, indicating contributions from both answer presence and certification status.BLIND → HINT adds an unverified answer; HINT → CERT changes its epistemic status.

C. Where error flags land

Certification shifts flags downstream and improves raw localization, but chance correction shows that relaxed within-one-step scoring does not imply near-correct localization; clearer labels preserve the ladder ordering.

  • Where flags land: 0.261 → 0.379 exact localization from BLIND to CERT accompanies fewer missed traces and more flags placed after the true first error.Missed traces fall from 0.383 to 0.165, while after-error flags rise from 0.209 to 0.347.
  • Tolerance and chance: A flag within one step of the adjudicated label is counted correct in the relaxed criterion because adjacent steps were often both justifiable as the first error.The wider window also increases the probability that a lucky guess succeeds.
  • Tolerance and chance: +0.136 at BLIND and +0.185 at CERT are the chance-rate gains from exact to within-one-step scoring, exceeding observed gains of +0.111 and +0.139.Relaxing the tolerance widens the target and makes random placement more likely to succeed.
  • Tolerance and chance: 0.199 exact versus 0.201 within one step under BLIND, and 0.306 versus 0.321 under CERT, for the chance-corrected skill score.Both tolerances preserve the monotonic information-ladder conclusion.
  • Label ambiguity: 0.382 under BLIND to 0.632 under CERT on the 19 traces with unanimous first-error-step selection.This clearer-error subset has higher absolute localization, but the information-ladder ordering remains; it is a robustness check rather than a noise-corrected full-dataset estimate.

D. Revision after commitment

After blind commitment, revealing the answer preserves improved localization but mainly changes monitors’ judgments on traces whose conclusions expose errors. Online stepwise monitoring performs worse than retrospective whole-trace evaluation, though its gap is not a calibrated deployment penalty.

  • Revision after commitment: 0.418 exact localization is achieved by REVISE, with balanced accuracy of 0.783, slightly below CERT because REVISE flags more overall.REVISE has mean sensitivity 0.861 and specificity 0.705; its scores are reported net of the reconsideration control.
  • Revision after commitment: The largest discrimination gap requires withdrawing flags on clean traces while retaining them on critical traces, but some monitor estimates have fewer than ten observations.Among monitors with at least ten observations in both cells, gemini-3.1-pro separates the conditions most clearly, whereas qwen3-32b and llama-3.3-70b show negative gaps.
  • Revision after commitment: 93.8% of previously passed wrong-answer traces receive a new flag after answer revelation, versus 18.0% of previously passed critical traces.Every monitor with an interpretable denominator shows this asymmetry, supporting answer-based conclusion consistency rather than reliable reasoning verification.
  • Stepwise monitoring: 0.614 balanced accuracy under STEP is below BLIND’s 0.637, while STEP’s exact localization is 0.196 versus 0.261 and its miss rate is 0.422.STEP exposes reasoning one prefix at a time and requests a decision at every step.
  • Stepwise monitoring: STEP combines hidden future context, trace segmentation, and repeated decisions, so its lower performance is evidence of greater online difficulty rather than a calibrated deployment penalty.False-positive risk accumulates with trace length, and STEP is neither cost- nor information-matched to whole-trace conditions.
  • Stepwise monitoring: 82% of clean traces are flagged by deepseek-v4flash under STEP, while it exactly locates only 28% of erroneous traces.Its specificity is 0.178, illustrating the false-positive cost of repeated prefix-level decisions for weaker monitors.

F. Analysis of the critical set

The 24 critical traces mostly contain exploratory, supplementary, or robust-conclusion errors rather than substantive failures in the supporting argument. Examples show that written traces can contain incorrect claims that are repaired, irrelevant, algebraically canceled, or accidentally compatible with the correct answer.

  • Critical-set composition: 8 exploratory or repaired errors, 7 supplementary errors, and 7 robust numerical or algebraic errors make up the 24 critical traces.Only two traces contain a substantive failure in the argument supporting the correct answer.
  • Critical-set composition: No error category shows a positive CERT-BLIND change in recall, and the two-trace substantive category is not interpreted separately.The critical-set result is therefore not attributed solely to repaired or exploratory cases.
  • Examples: A temporary claim that a complementary channel’s Choi rank was one is later repaired when the model calculates its Choi matrix as Id with rank d.The trace ultimately returns to the correct bound min(n, rd), leaving the false claim present but absent from the final derivation.
  • Examples: An incorrect fixed-k description of a transmitted field changes its phase description but not the requested amplitude.Spatial-interface matching fixes frequency, so the exact boundary-driven field is not a single fixed-k mode.
  • Examples: An erroneous Jacobi-factor Gauss sum appears in both sectors and squares to one, leaving the final expression (N1N2/n) unchanged.The intermediate algebra is wrong, but the extraneous factor cancels across the two sectors.
  • Examples: Two traces reach correct answers without valid supporting arguments, including a bridge-mute explanation that identifies amplitude a2 instead of the requested resonance frequency f2.The answer remains correct because added mass also lowers f2.

IV. DISCUSSION

Trusted answers improve detection chiefly when incorrect conclusions reveal that something went wrong, not when correct conclusions conceal flawed reasoning. The discussion argues that evaluations should separate conclusion checking from process verification and should report answer-blind, critical-set, and localization results.

  • Implications for evaluation: 0.796 balanced accuracy under CERT can coexist with 0.379 exact localization and 0.438 critical-trace recall.Aggregate evaluation metrics combine process verification with conclusion checking and can obscure this distinction.
  • Implications for evaluation: Answer-blind evaluation, separate recall for conclusion-revealing and critical traces, and localization reporting are proposed as inexpensive procedural corrections.Critical-trace recall at fixed specificity is suggested when a single figure of merit is required.
  • Broader connection: Across two evaluator roles and task settings, reference standards shift LLM evaluators toward consistency checking against the reference rather than verification.A corrupted reference previously caused correct work to be marked wrong; here a genuine reference inflates apparent capability without visibly damaging aggregate accuracy.
  • Safety relevance: Correct conclusions provide no evidence that reasoning is sound, creating a structure shared by the studied benign errors and possible reward hacking or deliberate sabotage.The ordinary errors here are mostly non-load-bearing and are not themselves examples of reward hacking or sabotage.
  • Safety relevance: Weaker monitors incur the largest false-positive costs in STEP, concentrating the failure mode where a trusted monitor is weaker than the overseen model.The discussion connects this operating constraint to weak-to-strong control.
  • Limitations: The experiment does not establish monitoring performance against adversarial reasoning or provide a quantitative bound on sabotage detection.Most studied errors were not load-bearing, and none was optimized to evade scrutiny.

B. Limitations and Future Work

The study is constrained by adjudication dependence, uneven human annotation, physics-only ordinary errors, and several monitor-comparison limitations. Future work must address the costly creation of specialized ground-truth standards.

  • Limitations: The 24-trace critical cell makes cross-monitor direction more informative than any single effect magnitude, but its labels are especially adjudicator-dependent.Localization targets exist for only 16 of 24 critical traces because the independent arms disagreed.
  • Limitations: Only 6 traces outside the pilot were scored by more than one person, preventing estimation of annotator effects in the human arm.
  • Limitations: Answer access during reference-standard construction could omit inconspicuous errors from the critical set and leave some supposedly clean traces mislabeled.The multi-arm review and source-masked adjudication reduce dependence on individual judges but do not remove this shared conditioning.
  • Limitations: The evidence concerns ordinary, mostly non-adversarial model mistakes in physics, so extrapolation to sabotage remains analogical and scope-limited.
  • Future Work: Ground-truth construction was the main practical bottleneck: localizing first errors in physics derivations across 237 traces required months of specialized work.The authors note that this information-ladder design could be transferred to mathematics benchmarks with existing step-level labels.

V. CONCLUSION

The conclusion reports that certified answers improve monitoring metrics, especially for wrong-answer traces, but do not establish independent verification of reasoning. It recommends answer-blind and critical-trace evaluations alongside localization.

  • V. CONCLUSION: 0.637 to 0.796 mean balanced accuracy and 0.261 to 0.379 exact first-error localization followed provision of a certified reference answer.These results aggregate 237 naturally generated physics traces evaluated by 8 LLM monitors.
  • V. CONCLUSION: +0.299 recall on wrong-answer traces contrasted with -0.083 on critical traces whose flawed reasoning reached a correct answer.The contrast had the same direction for all 8 monitors.
  • V. CONCLUSION: The study concludes that trusted-answer access improves conclusion-consistency checking more than independent verification of reasoning.
  • V. CONCLUSION: Process-oversight evaluations should report answer-blind performance, critical-trace recall, and localization rather than treating reference-assisted aggregate gains as verified arguments.
  • V. CONCLUSION: Each trace received independent final-answer and first-error labels, allowing identification of correct-answer traces that nevertheless contain genuine reasoning errors.The earliest false assertion was used even when it did not propagate or was later repaired.

Appendix C: Monitor coverage

Table IV documents monitor coverage and the common subsets used to compare information-ladder conditions, while the prompt templates specify shared and condition-specific monitoring inputs.

  • Appendix C: Monitor coverage: Table IV reports valid verdict counts and per-monitor common subsets for information-ladder comparisons.
  • Appendix C: Monitor coverage: BLIND, HINT, and CERT share the same trace, rubric, and output request, while their prompts differ in the information supplied to the monitor.

1. REVISE, RCTRL and STEP

REVISE and RCTRL replay the blind exchange before adding a final user turn, whereas STEP evaluates each available prefix separately; coverage comparisons use valid common subsets.

  • 1. REVISE, RCTRL and STEP: REVISE replays the monitor’s complete BLIND exchange before appending a final user turn, while RCTRL is its no-answer reconsideration control.
  • 1. REVISE, RCTRL and STEP: STEP makes a separate call for every prefix ending at step k, withholding later steps and the reference answer.The trace-level flag is the union of per-step decisions, and the earliest flagged step becomes the prediction.
  • 1. REVISE, RCTRL and STEP: The common subset intersects valid trace identifiers across BLIND, HINT, CERT, and REVISE for each monitor.Cell-restricted recall comparisons use this fixed subset.
  • 1. REVISE, RCTRL and STEP: All three generating models used the same numbered-solution prompt, and the STEP condition reviews the solution one step at a time.
  • 1. REVISE, RCTRL and STEP: BLIND, HINT, and CERT share the same output request, including has_error, first_error_step, and error_confidence fields.
  • 1. REVISE, RCTRL and STEP: BLIND withholds the reference answer, whereas HINT presents an unverified proposed final answer that monitors must treat only as orientation.
Loading 2609.00264v1…