FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research

2 Introduction

• Existing oversight studies largely assess binary solution validity, leaving less evidence about whether judges identify the actual errors behind their rejections.Because accurate error labels are costly, understanding the specific flaw is important for assessing whether oversight approaches will generalize.Source: §2 · p. 2
• Critique research on simpler tasks found that model critiques helped evaluators identify about 50% more flaws, but complex domains require expert-validated errors for direct alignment tests.Saunders et al. (2022) studied topic-based summarization and synthetic tasks; later work also explored critique evaluation methods (Sun et al., 2024).Source: §2 · p. 2
• Debate studies found higher truth-identification accuracy than single-agent consultancy and accuracy gains with more capable debaters, though this trend may fail on harder problems.Kenton et al. (2024); Khan et al. (2024); Michael et al. (2023).Source: §2 · p. 2
• Debate performance declines can reflect weaker judging or agents increasingly persuading judges of incorrect claims, so known flaws help distinguish these mechanisms.Difficulty may also increase model and judge uncertainty, cognitive shortcuts, and concerns about obfuscated arguments (Barnes et al., 2020; see also Michael et al. (2023) and Brown-Cohen et al. (2023)).Source: §2 · p. 3
• In prover-verifier games, detailed error annotations could test verifier generalization across difficult domains and help determine whether a helpful prover resorts to specious arguments as difficulty rises.The latter concern is conditional on future work confirming that human legibility decreases with problem difficulty.Source: §2 · p. 3
• FindTheFlaws provides diverse expert-validated correct solutions and flawed solutions with annotated errors, enabling tests of genuine flaw identification rather than non-scalable heuristics.The datasets are intended to help track progress and test whether controlled-experiment improvements extend to harder problems.Source: §2 · p. 4

3.1 Scalable oversight and related benchmarks

• Kenton et al. (2024) found debate generally outperformed consultancy across diverse tasks, but these challenging datasets lacked ground-truth error locations or explanations except for modified PrOntoQA.The absence of error annotations limits assessment of whether correct-answer debaters rely more on plausible falsehoods for difficult questions.Source: §3.1 · pp. 4–5

3.1.1 Benchmarks focused specifically on LLM critique/evaluation abilities

• CriticEval tests four critique tasks across nine domains, while several related critique benchmarks cover narrower task or domain ranges.CriticEval includes feedback, response comparison, revision, and feedback-quality evaluation.Source: §3.1.1 · p. 5

3.2 Process vs. outcome supervision

• 78% of MATH test questions were solved by Lightman et al.'s process-supervised model, and PRM800K provides 800,000 step-level correctness labels.Whether process supervision generalizes from mathematics to domains such as medicine, law, language, or biology remains an open question.Source: §3.2 · p. 5

3.3 Hallucination benchmarks

• FindTheFlaws differs from hallucination benchmarks by focusing on difficult, expertise-dependent errors; FELM is a close comparison, but two-thirds of its prompts are less specialized tasks.FELM also uses expert annotators, spans multiple domains, and provides error locations and explanations.Source: §3.3 · p. 6

3.4 Synthetic flaw generation

• FindTheFlaws flaws were not generated through adversarial training: some were adversarially selected, while others were introduced by models and reviewed or revised by experts.The dataset includes expert annotations of error types; GPQA Diamond Plus contains the most extensively expert-revised flawed solutions.Source: §3.4 · p. 6

4.1.1 Modified TheoremQA

• Manual review yielded 95 Modified TheoremQA questions with flawed solutions, including four whose calculation errors still produced correct answers.Validated expert solutions were restructured into numbered steps before GPT-4 generated corrupted versions and described each error and its location.Source: §4.1.1 · p. 7

4.1.2 Adversarial MedQA

• Adversarial MedQA selected 319 questions to challenge GPT-4, then marked 222 reliable and 97 unreliable based on clinician agreement with the answer key.The questions included 255 GPT-4 errors and 64 correct answers; informing Clinician 1 that the model performed poorly was associated with higher key agreement on the final 170 questions (57.6% vs. 45.6%, p = .034).Source: §4.1.2 · pp. 7–8

4.1.3 CELS

• CELS contains 452 model-generated arguments across four domains, with expert reviewers labeling sentence-level reasoning problems and flagging 20 questions as inferior.Medical and Lojban sentences were judged by two reviewers and legal sentences by three; the 20 flagged questions comprise two medical and 18 Lojban questions.Source: §4.1.3 · pp. 8–9

4.1.4 GPQA Diamond Plus

• GPQA Diamond Plus modifies 191 Diamond questions with expert-validated stepwise correct solutions and deceptive flawed solutions annotated at the first error.Experts revised or rewrote flawed drafts to make errors subtle; seven questions were marked unreliable, and answer choices were changed in 19 cases.Source: §4.1.4 · pp. 9–10

4.2.1 Solution-level tasks

• Solution-level evaluation tests whether models classify a solution as correct or flawed and, for flawed solutions, whether they identify the expert-annotated error.Error equivalence is assessed with Claude 3.7 Sonnet; Modified TheoremQA and GPQA Diamond Plus target the first error, while MedQA uses expert error descriptions.Source: §4.2 · p. 12, §4.2.1 · p. 12

4.2.2 Python650 tasks

• Python650 evaluation includes standard solution judgments, argument checking, and an alternative meta subset restricted to flawed arguments that classify solutions correctly but miss actual errors.Standard Python650 evaluates only classification; Meta Python650 evaluates argument accuracy and error explanations, while the alternative subset omits classification scoring because of its selection criteria.Source: §4.2.2 · pp. 12–13

4.2.3 CELS tasks

• CELS adapts evaluation to sentence-level labels, testing whether models classify each sentence correctly and identify expert-annotated errors in flawed sentences.Source: §4.2.3 · p. 13
• AMBIGUOUS sentences are excluded, and scores account for clustering when sentences from the same question are reused.Each sentence is treated as a sample; clustering is accounted for when reporting task scores in Section 5.3.Source: §4.2.3 · p. 13

5.1 Solution-level results

• On Adversarial MedQA, almost all models outperform the single-clinician baseline, though multiple clinicians might score higher.For the other two datasets, error-grading baseline analogs use agreement on the first error when o3-mini receives correct and flawed solutions; the comparison suggests models can miss errors they find when given correct information.Source: §5.1 · pp. 13–14
• Top models are relatively close on match but vary more on error grading, where GPT-4o performs much worse relative to peers.Figure 1 covers Adversarial MedQA, Modified TheoremQA, and GPQA Diamond Plus; displayed match results use F1 and error-grading results use accuracy.Source: §5.1 · p. 13, §5 · p. 13, Figure 1 · p. 14

5.2 Python650 results

• Meta-Python650 scores are extremely high for the expert baseline and o3-mini, followed by Claude 3.7 Sonnet; grading is lower, and Alt Meta-Python650 drops significantly.Manual inspection suggests models and annotators often miss subtle errors when flawed reasoning still leads to the correct classification.Source: §5.2 · pp. 14–15, Figure 2 · p. 15

5.3 CELS results

• CELS match-all performance clusters more than on other datasets; grading varies, with the expert baseline far ahead in Lojban and the Claude family leading in Law.For Surgery, grade-all performance is similar across models and the expert baseline; o3-mini performs unusually poorly on Law.Source: §5.3 · p. 15

6.1 Key findings

• Match-task performance did not always predict error-grading performance, showing that detecting an error and identifying its specific nature are distinct capabilities.For example, models performed similarly on CELS Lojban match tasks, while o3-mini outperformed the other named models on error grading; GPT-4o also declined on GPQA Diamond Plus while Claude 3.7 Sonnet and o3-mini remained strong.Source: §6.1 · pp. 15–16
• Performance varied across models and datasets: Claude 3.7 and o3-mini were generally strong, while Claude models excelled over o3-mini on CELS Law error grading.Some task/dataset combinations also had clear performance tiers useful for automated sandwiching experiments.Source: §6.1 · p. 16
• Top models matched or outperformed experts on several datasets, but all models performed significantly below human baselines on CELS Lojban, which remains unsaturated.The discussion also notes that low agreement among human experts on Alt Meta Python650 may reflect ambiguity in identifying faulty reasoning that reaches a correct conclusion.Source: §6.1 · pp. 16–17

6.2 Implications for scalable oversight

• The datasets support testing whether oversight helps weak judges assess stronger systems, whether methods find genuine errors rather than brittle heuristics, and how performance changes across domains.They also provide baselines for tracking oversight methods; match/error-grading gaps may reveal scaling limitations as problem difficulty increases.Source: §6.2 · p. 17

6.3 Saturation

• Even saturated datasets can test oversight with weak judges and intermediate models, while unsaturated datasets may better indicate generalization to future capabilities.For CELS, the authors prioritized quality by requiring two experts to agree on sentence labels; multi-annotator data also support conservative performance-ceiling estimates.Source: §6.3 · p. 18

6.4 Limitations and future work

• Introduced flaws may be artificial, and their distribution may differ from errors produced by deceptive or debate-trained models; adversarial training is proposed for future expansions.Source: §6.4 · pp. 18–19
• Using Claude 3.5 Sonnet to match model-identified errors with expert annotations may favor certain explanation styles, while prompt phrasing may affect performance.The authors call for research on prompts that reduce phrasing sensitivity and improve consistency.Source: §6.4 · p. 19
• Expert agreement varies across datasets, and current frontier models meet or exceed conservative expert estimates on most; harder-domain benchmarks are needed to test future oversight reliability.The expert analogues for GPQA Diamond Plus and Modified TheoremQA highlight ambiguity in locating errors even when correct and flawed answers are known.Source: §6.4 · p. 19

• FindTheFlaws supports evaluation of model-assisted oversight by testing whether evaluators can spot errors in long-form reasoning across complex domains.Source: §7 · p. 19
• For some models, detecting that an error exists did not entail identifying or explaining its specific nature, a distinction relevant to oversight of invalid or deceptive reasoning.Source: §7 · p. 19
• Model strengths varied by domain: Claude models excelled on CELS Law error grading, where o3-mini performed relatively poorly, supporting evaluation across diverse domains.Source: §7 · p. 19

• GPT-4o results use slightly fewer samples than other models because some outputs were not well formatted.Source: A Performance metrics · p. 24
• Table 5 reports match-task metrics for solution-level datasets, with CORRECT as the positive class and FLAWED as the negative class.Source: Table 5 · p. 24
• Tables 7 and 8 report error-grading performance as the percentage of flawed-sample explanations judged to match or diverge from human expert analyses.Source: Table 7 · p. 26, Table 8 · p. 27

B.1 CELS

• CELS annotators marked each LLM-generated argument sentence for errors, with two reviewers for Surgery and Lojban and three for Law; a flaw required two reviewers’ agreement.Reviewers agreed that no flaw was present when none identified an issue; other cases were treated as lacking agreement.Source: §B.1 · p. 27
• 84.1% of Lojban sentences, 79.9% of Surgery sentences, and 80.3% of Law sentences had reviewer agreement; Cohen’s κ was 0.658 for Lojban, 0.577 for Surgery, and Fleiss’ κ was 0.518 for Law.Within-argument mean agreement was 85.1% for Lojban, 79.7% for Surgery, and 80.6% for Law; Law’s pairwise Cohen’s κ values were 0.520, 0.534, and 0.507.Source: §B.1 · p. 27
• Model evaluations used only sentences on which reviewers agreed, with other sentences filtered out; for the first 20 Surgery arguments, validator majority vote agreed with the reviewers on 91.1% of the 168 agreed sentences.The reviewer agreement rate across all 204 Surgery sentences was 82.4%; Table 3 also reports validator-reference metrics for the first 20 Surgery arguments, the first 20 Lojban arguments, and all Law arguments.Source: §B.1 · pp. 27, 30
• Table 9 reports CELS sentence-level match-all performance metrics, while Table 10 reports Claude 3.5 Sonnet’s ratings of whether model explanations for flawed sentences match or diverge from expert analyses.Both tables use 95% confidence intervals calculated with a cluster-based block bootstrap; Table 9 defines correct solutions as positive samples and flawed solutions as negative samples.Source: Table 9 · p. 28, Table 10 · p. 29
• On CELS Surgery, top models, especially o3-mini, were within validator error bars on all metrics except precision; on Law, Claude 3.5 Sonnet was within the human validator’s error bars on all metrics, while o3-mini performed poorly.On Lojban, all models were below the human validator on accuracy, recall, and F1, and also performed more poorly on match and error grading; on Law and Surgery, at least one model matched or exceeded validator performance on match and error grading.Source: §B.1 · p. 30

B.1.1 Python650

• On Standard Python650, Annotator 2’s baseline was indistinguishable from o3-mini across accuracy, specificity, precision, recall, and F1.The baseline used Annotator 1’s labels as the gold standard; the annotators agreed on 96.6% of the 648 questions (Cohen’s κ = .932).Source: §B.1.1 · p. 30
• On Meta Python650, one model—either o3-mini or Claude 3.7—was not clearly inferior to the human baseline on both match and error-grading tasks.For most of the dataset, Annotator 1 supplied ground truth and Annotator 2 the baseline, with those roles reversed when Annotator 1 disagreed with the answer key.Source: §B.1.1 · p. 30
• On Alt Meta Python650, baseline and model error-grading performance was exceptionally poor, indicating evaluation ambiguity when arguments reach the correct answer with flawed reasoning.For most cases, only one annotator identified a flaw; that annotator’s comments and labels were used as ground truth, prioritizing Annotator 1 when both identified a flaw.Source: §B.1.1 · p. 30

B.1.2 Adversarial MedQA

• Claude 3.7 Sonnet and o3-mini outperformed the external-validator baseline on virtually every metric and on both match and error grading.The reported baseline uses external-validator performance.Source: §B.1.2 · p. 30
• Clinician annotators agreed on 60.8% of the 319 full-dataset questions (Cohen’s κ = 0.510) and 51.8% of the reliable subset (Cohen’s κ = 0.397).The reliable subset included questions on which at least one annotator agreed with the answer key.Source: §B.1.2 · p. 30
• Agreement between the reliable subset and external clinical validators was 53.6% (Cohen’s κ = 0.418).Each external validator was assigned one of the 319 questions.Source: §B.1.2 · p. 30

B.1.3 Modified TheoremQA and GPQA Diamond Plus

• No match-task baseline or related metrics were reported for Modified TheoremQA or GPQA Diamond Plus because obtaining additional annotators or validators was costly and difficult.For GPQA Diamond Plus, questions experts judged incorrect or indeterminate were excluded.Source: §B.1.3 · p. 31
• For error grading, Claude 3.5 Sonnet served as a proxy for uncertainty in experts’ first-error locations, but it was shown both correct and flawed solutions.On Modified TheoremQA, this proxy baseline exceeded all models on standard error grading; on GPQA Diamond Plus, Claude 3.7 and o3-mini were within its error bars.Source: §B.1.3 · p. 31
• The error-location labels mark the experts’ chosen first errors, although reasonable people may differ about which misleading or ambiguously correct claims count as errors.The authors caution that the proxy baseline captures only one source of uncertainty and benefits from seeing both correct and flawed solutions.Source: §B.1.3 · p. 31

C Prompts

• Templates usually include the question and proposed solution; Meta Python650 also includes the solution judgment, while o3 prompts omit evaluation-process sections.CELS templates are separate to support sentence-level labels.Source: C Prompts · p. 32
• Solution-level and Python650 templates compare submitted explanations with standard explanations, requiring explicit agreement about identified errors and their locations.The comparison is based on explicitly stated information, not assumptions.Source: C Prompts · p. 33
• CELS prompts ask for sentence-by-sentence CORRECT or FLAWED classifications with comments, followed by a whole-solution correctness judgment.The prompts also direct evaluators to examine subtle errors, edge cases, and contradictions.Source: C Prompts · pp. 34–35
• The CELS grade-all template marks explanations EQUIVALENT or DIFFERENT per sentence, comparing whether they identify the same errors.Matching tags alone are insufficient if the explanations do not describe the same errors.Source: C Prompts · pp. 35–36

D Examples from datasets

• The appendix provides a Lojban CELS example because Surgery and Law examples share its structure, and excludes GPQA Diamond Plus to limit online exposure.The stated concern is that GPQA Diamond is frequently used as a performance benchmark.Source: D Examples from datasets · p. 38
• In the Python650 example, skipping a repeated height can omit one of the top three values, contrary to the LLM argument that the check does not affect correctness.The expert commentary notes that duplicate heights may still need to be listed separately.Source: Table 15 · p. 38, D Examples from datasets · pp. 39–40
• The Lojban CELS example flags a claim implying fixed SVO order, while commentary describes flexible ordering and disputes ordinary subject-object terminology.Another annotation identifies the full first argument and predicate, noting that the top-level bridi has one sumti.Source: D Examples from datasets · p. 43