PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity?

Obinna I. Ekekezie

arXiv:2609.12162v1cs.AIcs.CL

TL;DR

Draft-verify-revise pipelines can create deictic disagreements when stages interpret the same context-dependent expression from different standpoints, yet the revise-stage model’s ability to resolve them has not been tested in a controlled setting. The paper tests that judgment with synthetic stimuli and finds that wrong verdicts often rely on surface cues rather than operational reasoning.

  • Problem

    The paper asks whether an LLM can reliably judge which reading of a context-dependent expression is correct when pipeline stages resolve it from different standpoints.

    Source: §1 · p. 2

  • Method

    The experiment isolates the binary judgment using synthetic stimuli and compares primary and ablation conditions that remove the grader’s error-classification labels.

    Source: §1 · p. 2, §2.5 · p. 5

  • Results

    On incorrect-draft stimuli, the meta-evaluator articulated the operational interpretation on every wrong transparent trial and on 60.78% of wrong opaque trials (544 of 895), yet let it govern almost none.

    Source: §4 · p. 14

  • Takeaways & Limitations

    Context engineers should make each context-dependent expression’s intended referent explicit in the prompts and context supplied at each pipeline stage.

    Source: §5 · p. 18

    Rationale analysis covers only wrong verdicts, so the reasoning behind correct verdicts is unobservable and that blind spot widens as accuracy rises.

    Source: §4.5 · p. 17

Abstract

from arXiv · show

Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute. One LLM drafts, a second critiques the draft and provides feedback, and a third uses that feedback to revise the draft into the final output. As context cascades between stages, LLMs at different stages can resolve a context-dependent expression such as "previous" differently. When that happens, the expression undergoes a deictic shift, a change in what it refers to. This phenomenon was studied with a synthetic dataset of 10 base examples, each rendered in three conditions. Holding the shared components constant, the conditions varied whether the draft stage LLM (the assistant) or the verify stage LLM (the grader) resolved the expression correctly, and how much independent reasoning the revise stage LLM (the meta-evaluator) needed to determine which reading was correct. Six models from three providers were tested across 21 reasoning effort configurations using e-values for sequential testing, in a primary experiment and an ablation experiment that removed error classification labels from the grader's feedback. A separate LLM analyzed the meta-evaluator's stated rationale for each wrong verdict. Balanced accuracy (the unweighted mean of sensitivity and specificity) ranged from 0.156, below chance, to near-perfect. GPT-5.2 rose from 0.156 without reasoning to 0.942 at its highest reasoning effort level, while Gemini 3 Pro stayed above 0.94 at every level. Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh reasoning effort for roughly 5% of the cost per trial. When the meta-evaluator erred, it tended to rely on surface cues rather than operational reasoning. Context engineers implementing draft-verify-revise pipelines should be wary of deictic shifts and make the intended referent explicit at each stage.

1. Introduction

Draft-verify-revise pipelines pass context across stages, where context-dependent expressions can shift referents and create disagreements that the revise-stage model must resolve. This paper isolates that judgment in a controlled experiment and specifies the roles, stimuli, models, and analyses used to study it.

1. Introduction

  • A deictic shift can make pipeline stages read the same expression differently, leaving the revise-stage model to settle a disagreement that may otherwise create false confidence.The paper frames this as a controlled gap in prior testing and measures a binary verdict rather than requiring a revised draft.Source: §1 · p. 2

2.1 The Phenomenon

  • In the dosage example, the assistant treats “previous” as the dose before its proposed change, while the grader follows the field label, so resolving the disagreement requires tracking temporal standpoint.The paper treats “previous” as a deictic operation on the now-then axis, although it is not a prototypical deictic term.Source: §2.1 · pp. 2–3

2.2 Three-Role Architecture

  • The meta-evaluator renders a binary verdict on the grader’s claim using a complete prior transcript, while all stimulus text before that stage remains fixed across trials.This design makes performance variation attributable to meta-evaluator reasoning rather than variation in earlier stages.Source: §2.2 · p. 3

2.3 Stimulus Design

  • Ten base examples were rendered in three conditions that vary whether the assistant or grader is correct and whether the grader’s operational reasoning is visible.The transparent and opaque incorrect-draft conditions share the same draft and answer, making their accuracy gap a diagnostic of reasoning visibility versus independent reasoning.Source: §2.3 · pp. 3–4

2.4 Models and Configurations

  • Six models from three providers were tested across 21 reasoning-effort configurations, with the stimulus-authoring model flagged because familiarity with its outputs could affect its results.All models are closed-weight, and “smaller” versus “larger” designations are inferred from provider pricing tiers and naming conventions.Source: §2.4 · pp. 4–5

2.5 Primary and Ablation Experiments

  • The ablation repeats the 21 configurations and 30 stimuli after removing four grader error-classification fields while retaining its analysis, root-cause text, and evidence.The tested hypothesis was that labels could distract the meta-evaluator from the grader’s substantive analysis.Source: §2.5 · p. 5

2.6 Statistical Approach

  • Balanced accuracy is the primary metric because raw accuracy rewards always accepting the grader under the dataset’s 1:2 class imbalance; sequential e-values govern stopping and further comparisons.Cells are tested against chance in batches, while additional tests compare conditions, reasoning effort levels, model sizes, and the primary and ablation experiments.Source: §2.6 · pp. 5–6

2.7 Rationale Analysis

  • A separate LLM classifies stated rationales for wrong verdicts, distinguishing whether operational reasoning appeared and whether it governed the verdict.On correct-draft trials, articulation alone cannot distinguish operational reasoning from matching the “Current” field label, so the analysis uses whether the interpretation governed judgment.Source: §2.7 · pp. 6–7

3. Results

Across the primary and ablation experiments, independent models ranged from below-chance to high balanced accuracy, with Gemini 3 Pro low effort leading both. Accuracy varied by condition and model; scaling effort raised some scores but did not guarantee better accuracy or lower cost, while wrong verdicts often reflected reliance on surface cues.

3.1 Experiment Overview

  • Each experiment tested 21 configurations over 30 stimuli, yielding 630 cells and 12,600 trials; most cells reached a statistical boundary after one iteration.The four authoring-model configurations may reflect familiarity with their own stimulus patterns.Source: §3.1 · p. 7

3.2 Performance: Balanced Accuracy

  • 0.965 and 0.971 were Gemini 3 Pro low-effort balanced accuracies, the top independent configuration in the primary and ablation experiments, respectively.GPT-5.2 with no reasoning scored 0.156 in the primary experiment; the independent-model range narrowed from 0.809 to 0.741 in the ablation.Source: §3.2 · p. 7

3.3 Per-Condition Accuracy: Specificity and Sensitivity

  • The transparent–opaque accuracy gap was significant for 11 of 21 configurations in the primary experiment and 12 in the ablation.Transparent sensitivity reached 1.000 in 14 primary and 18 ablation configurations; opaque sensitivity was more demanding because grader analysis supplied no surface cues.Source: §3.3 · p. 8, Figure 2 · p. 9

3.4 Scaling Effects

  • Four of six primary-experiment model-size comparisons were significant, all favoring the larger model; the OpenAI pair was significant in neither test.The ablation replicated the OpenAI pattern, while comparisons quantified over measured configurations rather than matching provider-specific effort labels.Source: §3.4 · p. 10

3.5 Failure Mode Analysis

  • On correct-draft errors, the operational interpretation governed only 9 of 1,310 wrong verdicts; on transparent errors, field-label matching governed 176 of 178.On opaque errors, the interpretation was articulated in 544 of 895 cases but governed only 23 verdicts; the transparent–opaque articulation gap mirrored the accuracy gap.Source: §3.5 · pp. 10, 11, 13

3.4 Scaling Effects

  • In the primary experiment, GPT-5.2 ranged from 0.156 without reasoning to 0.942 at xhigh, while Gemini 3 Pro’s range was 0.024.No adjacent reasoning-effort transition was significant in either experiment; mean reasoning tokens generally rose with effort.Source: §3.4 · pp. 8, 10, Figure 3 · p. 11
  • Gemini 3 Pro low effort reached 0.965 balanced accuracy at $0.0030 per trial, outperforming GPT-5.2 xhigh at 0.942 and $0.0556.Gemini 3 Flash medium reached 0.894 at $0.0022; three effort transitions reduced balanced accuracy.Source: §3.4 · p. 10, Figure 4 · p. 12

3.5 Failure Mode Analysis

  • Removing error-classification labels coincided with opaque articulation rising from 60.78% to 68.72% and governing from 2.57% to 6.02%.Misreport flags appeared in 21.0% of primary and 25.3% of ablation opaque wrong-verdict trials, but an audit suggested those rates were overstated.Source: §3.5 · p. 13

4. Discussion

The discussion attributes many wrong verdicts to field-label matching, while treating training-based explanations and feedback-label distraction as hypotheses. It also emphasizes model-specific scaling tradeoffs and limits on what the synthetic stimuli and rationale analysis establish.

4. Discussion

  • On incorrect-draft errors, meta-evaluators often stated the operational reading but set it aside for field-label matching; on correct-draft errors, label matching governed verdicts and operational reasoning usually did not surface.The operational reading appeared on 60.78% of opaque wrong verdicts in the primary experiment, but governed almost none of them.Source: §4 · p. 14

4.1 Field Label Matching Behaviors May Be an Artifact of Pre-/Post-Training

  • The discussion hypothesizes that pre- or post-training may reinforce surface label matching, but testing the proposed training-data explanation requires access to unavailable evaluated-model data.The authors suggest TURF as a possible test, while noting that the six evaluated models’ post-training data are not public.Source: §4.1 · pp. 14–15

4.2 The Structure of the Grader’s Feedback Can Influence the Meta-Evaluator’s Reasoning

  • Removing the grader’s four error-classification fields raised opaque articulation and governing rates among wrong verdicts, but only 3 of 63 condition-level accuracy comparisons significantly improved.The pattern points toward distraction rather than reduced deference, but the fields were removed together, so a field-wise ablation is needed to distinguish mechanisms.Source: §4.2 · p. 15

4.3 Pathological Deference to the Grader’s Feedback Alone Cannot Account for the Meta-Evaluator’s Behaviors

  • Field-label matching explains wrong verdicts in both directions better than pathological deference alone, although deference may contribute to a minority of wrong acceptances.Wrong rejections of the grader’s correct claim occurred on 21.3% of primary-experiment opaque trials; the extreme GPT-5.2 no-reasoning configuration made errors in both directions.Source: §4.3 · pp. 15–16

4.4 Practical Implications for Model Selection and Configuration in Draft-Verify-Revise Pipelines

  • Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh for roughly 5% of the cost per trial; optimal effort varied by model, and the scaling results cover only 10 base examples.The 600 trials per configuration repeat those examples rather than representing 600 independent stimuli, so the scores do not establish performance on other stimuli of the same kind.Source: §4.4 · p. 16

4.5 Limitations

  • The synthetic minimal pairs test only “previous,” and all examples align the operationally correct target with the “Current” field, leaving field matching confounded with correctness.The rationale analysis cannot reveal what produced correct verdicts; separating operational reasoning from label matching would require stimuli where those targets diverge.Source: §4.5 · pp. 16–17
  • Binary verdicts and forced-binary rationale flags limit interpretation: grader reasoning can conflict internally, and recorded verdicts can contradict stated rationales.Some opaque wrong verdicts may object to the grader’s argument rather than its conclusion; how often verdict and rationale diverged was not measured.Source: §4.5 · pp. 17–18
  • The rationale probe’s audit did not establish classification correctness or reliability, and the experiment tested neither tailored prompts nor production of a revised draft.The audit used an Anthropic model, while each rationale was classified once; results might differ if the meta-evaluator also had to produce a revision.Source: §4.5 · pp. 17–18

5. Conclusion

The experiment isolated a deictic ambiguity across pipeline stages and raises questions about whether revise-stage LLMs can resolve such disagreements reliably. Errors tended to reflect surface cues rather than operational reasoning, while model performance and cost varied substantially.

5. Conclusion

  • The results bring into question whether revise-stage LLMs can reliably resolve disagreements when pipeline stages assign different referents to the same expression.The experiment isolated one such temporal ambiguity using 10 base examples rendered in three conditions.Source: §5 · p. 18
  • When the tested LLM erred, it tended to rely on surface cues, such as field labels, rather than operational reasoning.Source: §5 · p. 18
  • The 10-example, three-condition experiment establishes that this failure occurs and what it looks like, but not how often it arises in applicable pipelines.Source: §5 · p. 18
  • Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh reasoning effort for roughly 5% of the cost per trial.Source: §5 · p. 18
  • Context engineers should make each expression’s intended referent explicit in prompts and context supplied at every stage.The paper notes that feedback structure can also influence revise-stage reasoning and potentially revision quality.Source: §5 · p. 18

Data and Code Availability

The study’s dataset, code, and analysis materials are openly available in a repository, with the exact results version archived on Zenodo. Raw provider batch outputs are not included in the repository.

Data and Code Availability

  • The dataset and source code supporting the findings are openly available under the MIT License.Source: Data and Code Availability · p. 18
  • The repository includes the synthetic datasets, per-trial outcomes, rationale analysis, and analysis code used to produce the paper’s figures and tables.Source: Data and Code Availability · p. 18
  • The exact version used for the results is archived on Zenodo, but raw provider batch outputs are available only from the corresponding author upon reasonable request.Source: Data and Code Availability · p. 18

Funding

The work received no external, institutional, or commercial funding.

Funding

  • The work received no external, institutional, or commercial funding.Source: Funding · p. 19

Use of Generative AI

The author used generative AI tools throughout the research process while retaining responsibility for the study and its decisions. The supplementary stimulus specification describes the meta-evaluator materials, a worked example, and the controlled differences among conditions.

Use of Generative AI

  • Claude Code and Claude Cowork assisted with research design, implementation, analysis, figures, tables, and manuscript editing, while the author made final decisions and remained responsible for the study.Source: Use of Generative AI · p. 19

SM1. Stimulus Specification

  • The stimulus specification reproduces the meta-evaluator prompt and response schema, provides the example inventory, and identifies components held constant or varied across conditions.Source: SM1. Stimulus Specification · p. 21
  • Each meta-evaluator trial requests a rationale and a binary correct-or-incorrect score through a provider-specific structured-output schema.Source: SM1. Stimulus Specification · p. 21
  • In example_04, the three conditions vary whether the draft’s 15mg or 20mg contingency is operationally correct and whether grader feedback supplies usable reasoning.The operational reversion target is the dose in effect immediately before the proposed titration.Source: SM1. Stimulus Specification · pp. 21–22
  • Across all 10 examples, most feedback evidence and classification fields remain constant across conditions; the ablation removes four error-classification fields.Source: SM1. Stimulus Specification · p. 22

SM2. Staged Design

The staged design set decision rules before data collection and assembled matched 21-configuration primary and ablation experiments through five stages.

SM2. Staged Design

  • Stage 1 tested each model at its lowest and a selected upper-bound reasoning effort level, yielding 12 configurations.GPT-5.2 used high rather than xhigh as its upper bound at this stage; xhigh was added later for cost management.Source: SM2. Staged Design · p. 22
  • Stage 2 added eight intermediate-level configurations only if independent-model results met both preregistered performance and profile-difference criteria.The criteria required an independent model to exceed chance at its upper bound and a model with at least three effort levels to show a meaningfully different profile at its two bounds.Source: SM2. Staged Design · p. 22
  • Stage 3 was a preplanned error-classification ablation using the primary experiment’s results as its comparison baseline.It re-ran the 20 configurations from Stages 1 and 2; Stage 5 added GPT-5.2 at xhigh on the ablation dataset.Source: SM2. Staged Design · pp. 22–23
  • Stages 1, 2, and 4 formed the 21-configuration primary experiment; Stages 3 and 5 formed the matched 21-configuration ablation.The ablation re-ran 20 configurations, while Stage 5 added GPT-5.2 at xhigh to align with the primary experiment.Source: SM2. Staged Design · pp. 22–23

SM3. Batch Inference Infrastructure

The infrastructure standardized batch submissions across providers, while the statistical methods used sequential e-values and paired comparisons tailored to each testing question.

SM3. Batch Inference Infrastructure

  • Each example was expanded to 20 trials per configuration and submitted through provider batch APIs; costs reflect February 2026 prices and a 50% batch discount.Provider-specific reasoning settings were translated to native parameters, and the reported cost comparisons are a price snapshot; interactive use would cost approximately twice as much.Source: SM3. Batch Inference Infrastructure · p. 23
  • Provider defaults governed most sampling settings, so within-cell trial variation reflects defaults in effect during the February 2026 collection window.Only Claude Haiku 4.5 with reasoning enabled had a configured temperature or top-p value; Anthropic reasoning-token counts were derived from output totals and visible-text token counts.Source: SM3. Batch Inference Infrastructure · p. 23

SM4. E-Value Testing Details

  • Per-cell verdict tests compared observed matching rates with a 0.5 chance null using a Bernoulli likelihood ratio and a seeded alternative of 0.7.Later batches use the preceding hit rate, clamped away from the null and 1; evidence thresholds support rejection at 20 and futility at 0.05.Source: SM4. E-Value Testing Details · p. 24
  • Accuracy estimates used two-sided, time-uniform confidence sequences obtained by inverting a beta-binomial mixture test martingale.The mixture uses a Jeffreys Beta(0.5, 0.5) prior; the probe-auditor alignment analysis instead reports a one-sided lower bound.Source: SM4. E-Value Testing Details · p. 24
  • Condition comparisons used paired, sequential testing-by-betting e-values, with an equal-weight mean as the global statistic and a double threshold of 20 for each claim.The construction is an exact closed test for the three condition comparisons within one configuration, not a simultaneous guarantee across configurations or experiments.Source: SM4. E-Value Testing Details · p. 25
  • Adjacent effort-level tests paired trials across neighboring settings and class-weighted differences to test whether balanced accuracy declined at higher effort.Complete pairing gives equal weight to sensitivity and specificity; transitions with no surviving paired rounds are aborted, while a transition with no valid trials on one side yields neutral evidence.Source: SM4. E-Value Testing Details · pp. 25–26
  • Within-provider model-size tests assessed whether every larger-model level beat the smaller model’s worst configuration and whether some level beat its best.The tests quantify over configurations rather than selecting a matched effort level; universal claims use a minimum statistic, while existential claims use a mean of minima.Source: SM4. E-Value Testing Details · p. 27
  • Cross-dataset tests paired outcomes between the primary and ablation experiments and tested both possible directions of difference.They used separate condition-level and configuration-level families, with adjusted e-values divided by 2K to control error while reporting a direction.Source: SM4. E-Value Testing Details · pp. 27–28

SM5. Rationale Analysis Protocol and Probe Validation

The section describes a rationale-analysis probe for classifying wrong meta-evaluator verdicts and audits its classifications against independent readings. The audit finds strong agreement on most flags but limits what that agreement can establish.

Probe specification

  • The probe classifies seven features of each rationale: five possible misattributions and two flags distinguishing whether operational reasoning was expressed and governed the verdict.Its inputs include the rationale and verdict, scenario values, the assistant’s fallback value, grader feedback, ground truth, and descriptions of the operational and field-label readings.Source: SM5. Rationale Analysis Protocol and… · pp. 28–29

Eligibility and parse handling

  • The probe includes only valid trials with wrong verdicts, yielding 2,383 eligible primary trials and 2,024 ablation trials from 12,600 each.The authors leave extending the governing flag to correct verdicts for future work because distinguishing independent reasoning from deference would require a new flag.Source: SM5. Rationale Analysis Protocol and… · p. 29
  • Each rationale was classified once, with no reported measure of classification stability under reruns; no eligible trial was dropped for parsing in either experiment.Non-schema-conforming responses were repaired or resubmitted as described, and every classification marking operational reasoning as governing also marked it as expressed.Source: SM5. Rationale Analysis Protocol and… · p. 29

Probe audit

  • The post hoc audit compared the probe with one independent auditor per trial, so it cannot measure agreement between two independent auditors on the same trial.The pools covered 240 trials per experiment, sampled from wrong-verdict trials; they included unequal numbers across conditions.Source: SM5. Rationale Analysis Protocol and… · pp. 29–30

Audit results

  • On opaque stimuli, auditors judged the probe correct on 72 of 90 trials in the primary experiment and 67 of 81 in the ablation, while grader-claim misattribution calls were often over-flagged.Across both experiments, auditors read 31 of 32 misaligned opaque grader-claim trials as faithful disagreements; therefore, the reported opaque misreport rate is an upper bound.Source: SM5. Rationale Analysis Protocol and… · p. 30

Misattribution flags

  • For Claude Haiku 4.5 with reasoning off, the assistant-fallback flag identified four of 22 wrong opaque verdicts in the primary experiment, while the grader-claim flag fired on 15.The four flagged assistant-fallback cases are a lower bound because an additional audited trial showed an unflagged misread; the grader-claim count is an upper bound.Source: SM5. Rationale Analysis Protocol and… · p. 31

Supplementary Exemplars

The exemplars illustrate selected rationale behaviors and failure modes, rather than estimating how often those behaviors occur.

Supplementary Exemplars

  • The 11 selected exemplars cover failures to express operational reasoning, setting it aside for field-label matching, opaque-stimulus errors, and comparisons with grader error labels present or removed.They also include cases where the stated rationale and recorded verdict disagree; selection was by reading rationales, not sampling, so the examples establish nothing about frequency.Source: Supplementary Exemplars · p. 32

Supplementary Exemplar S1. A Fallback to the Current Value Rejected by Its Label Alone

The two exemplar titles contrast a fallback rejected by the field label with a case where the current value is conceded as correct in practice.

Supplementary Exemplar S1. A Fallback to the Current Value Rejected by Its Label Alone

Supplementary Exemplar S3. A Real-World Rollback to the Current Value Conceded, Then the Labeled Previous Value Required by the Task

These exemplar titles present several ways the operational and field-label readings of “previous” shape judgments about drafts and grader feedback.

Supplementary Exemplar S3. A Real-World Rollback to the Current Value Conceded, Then the Labeled Previous Value Required by the Task

Supplementary Exemplar S4. Reverting Recognized as Returning to the Current Value, Then the Labeled Previous Value Accepted as a Reasonable Reading

Supplementary Exemplar S5. Citing the Labeled Previous Value Treated as Adherence to the Requirement

Supplementary Exemplar S7. The Draft Quoted Naming the Labeled Previous Value, Then Credited with Naming the Current Value

Supplementary Exemplar S8. The Ambiguity Conceded, and the Grader Rejected Because the Error Classification Labels Are Not Warranted

The passage labels this item Supplementary Exemplar S8.

Supplementary Exemplar S9. With the Error Classification Fields Removed, the Ambiguity Conceded and the Grader Rejected Because the Labeled Previous Value Is Not a Factual Error

The passage labels this item Supplementary Exemplar S9.

Supplementary Exemplar S10. The Rationale Rejects the Grader’s Mistaken Claim; the Recorded Verdict Accepts It

The passage labels this item Supplementary Exemplar S10.

Supplementary Exemplar S11. The Rationale Rejects the Grader’s Correct Claim by the Label Alone; the Recorded Verdict Accepts It

These exemplars show verdicts diverging from operational reasoning, including label-driven judgments and contradictions between stated rationales and recorded verdicts.

Supplementary Tables

The supplementary tables document model-level results, ablation and reasoning-effort tests, cost comparisons, audits of meta-evaluator interpretations, robustness checks, and the stimulus inventory. They also report high probe-auditor alignment across the reported conditions, with Table S17 showing no audited trial shifting against the probe.

Supplementary Tables

  • Balanced accuracy ranged from 0.156 for GPT-5.2 with no reasoning to 0.965 for Gemini 3 Pro at low effort in the primary experiment, and from 0.230 to 0.971 in the ablation.These were the bottom and top scores among independent models in each experiment.Source: Supplementary Tables · p. 47, Table S1 · p. 47
  • Four of 21 configuration-level ablation tests and three of 63 condition-level tests were significant, all favoring ablation; the condition-level effects were all on opaque stimuli.Configuration-level differences were +0.042 for Gemini 3 Flash minimal, +0.047 for GPT-5-mini minimal, +0.074 for GPT-5.2 none, and +0.115 for GPT-5.2 high effort. No transparent comparison was significant; 14 of 21 configurations had transparent accuracy of 1.000 in both experiments.Source: Table S2 · p. 49, Table S5 · p. 52
  • Of 479 primary and 495 ablation cells rejected in the e-value diagnostics, 419 (87.5%) and 439 (88.7%) were perfect 20-of-20 cells at the upper log e-value bound.The condition-comparison table notes that every value below 1 concerned opaque versus correct-draft stimuli; Claude Opus 4.6 returned exactly 1.000 for transparent versus opaque in every configuration in both experiments.Source: Table S3 · p. 50, Table S4 · p. 51
  • None of 15 adjacent reasoning-effort transitions showed significant degradation in either experiment, and the largest balanced-accuracy change was GPT-5.2’s improvement from none to low effort.That improvement was +0.434 in the primary experiment and +0.425 in the ablation. Pairwise comparison significance counts varied from 36 to 42 in the primary experiment and 35 to 41 in the ablation across betting caps; the reported 0.50 cap yielded 38 and 41, respectively.Source: Table S6 · p. 54, Table S15 · p. 66
  • GPT-5.2 at xhigh effort achieved 0.942 balanced accuracy at $0.0556 per trial, while Gemini 3 Flash at medium effort achieved 0.894 at $0.0022 per trial in the primary experiment.The configurations cost $0.0509 and $0.0021 per trial, respectively, in the ablation experiment; batch prices were half provider list rates.Source: Table S7 · p. 56
  • In both experiments, the every-level model-size test was significant for Anthropic and Google but not OpenAI, while the some-level test was significant for Anthropic and Google but not OpenAI.GPT-5.2 with no reasoning scored 0.156 versus GPT-5-mini minimal’s 0.307 in primary balanced accuracy, and 0.230 versus 0.354 in ablation; GPT-5-mini high effort scored 0.903 and 0.895, respectively.Source: Table S8 · p. 58, Table S9 · p. 59
  • On incorrect-verdict opaque trials, articulation was 60.78% in the primary experiment and 68.72% in ablation, while governing rates were 2.57% and 6.02%.Misattribution flags were driven mostly by the grader-claim flag on opaque trials, firing on 20.8% and 24.8%; Claude Haiku 4.5 off had the highest primary opaque governing rate at 15.17%, versus 0% for GPT-5.2 with no reasoning. These rates describe different analyses and denominators.Source: Table S11 · p. 61, Supplementary Tables · p. 61, Table S12 · p. 62
  • The study used 10 base examples, each rendered in correct-draft, transparent, and opaque conditions; an independent audit found at least 93.8% probe-auditor alignment outside opaque grader-claim flags.In that exception, alignment was 80.0% (72 of 90 audited trials) in the primary experiment and 82.7% (67 of 81) in ablation. The ablation removed the grader’s four error-classification fields.Source: Table S14 · p. 65, Table S16 · p. 67
  • No audited trial in either experiment shifted against the probe in the two-phase blind-then-informed audit.The table partitions outcomes into stable agreement, shifted toward probe, stable disagreement, and shifted against probe.Source: Table S17 · p. 69
Loading 2609.12162v1…