Source-linked AI summary
Correctness Conditioning as Selection in Chain-of-Thought Faithfulness Evaluation
Yueer Pan, Yuexi Pan
TL;DR
Restricting CoT faithfulness evaluation to correctly answered instances changes the population being measured. The paper formalizes this selection, reviews reporting protocols, and finds positive gaps across four arithmetic deployments, including a +7.4pp GLM-5.3 gap.
Problem
Restricting CoT faithfulness evaluation to correctly answered instances changes the population and the quantity being estimated.
Source: §1 · p. 2Method
A seven-variable causal model derives the selection gap and its propensity-based factorization under an explicit conditional-independence premise, alongside a review of 18 papers’ reporting populations.
Source: §1 · p. 2Results
GLM-5.3’s gap was +7.4pp (p < 0.0001, n = 1000); pooled Cochran–Mantel–Haenszel analysis gave OR = 1.70, and operation-count variation accounted for 42.8%.
Source: §6 · p. 9, §7 · p. 12Takeaways & Limitations
The measured setting establishes selection bias, but the neutral-framing runs do not support attributing it to the collider pathway.
Source: §7 · p. 12The headline runs do not test collider predictions, and neutral-framing runs detected no shortcut use.
Source: §3.1 · p. 4, §6 · p. 10
Abstract
from arXiv · showhide
Restricting chain-of-thought (CoT) faithfulness evaluation to correctly answered instances changes the population being measured. We study this selection step in a seven-variable structural causal model and derive the exact observed gap $\Delta = \operatorname{Cov}(A, C)/P(C=1)$. The model distinguishes difficulty confounding from shortcut-induced collider bias and states the conditional-independence assumption needed to express the gap through latent response propensities. A protocol review of 18 papers distinguishes answer-correctness restrictions from agreement-based selection, benchmark construction, and training-data filtering. We evaluate four sub-ceiling deployments on arithmetic tasks and observe positive selection gaps in all four. The primary GLM-5.3 run yields a +7.4 percentage-point gap ($n = 1000$, $p < 0.0001$), and pooled Cochran-Mantel-Haenszel analysis gives OR = 1.70 ($p < 10^{-6}$). An assumption-free covariance decomposition assigns 42.8% of the GLM-5.3 gap to between-stratum variation in interventionable-operation count, leaving the majority unattributed. When counterfactual compliance is observable for every instance, evaluating all traces directly estimates population faithfulness without correctness filtering.
1 INTRODUCTION
CoT faithfulness measures whether a reasoning trace reflects the computation behind an answer, but restricting intervention-based evaluation to correct answers changes the population and estimand. The paper develops a causal account of that selection, reviews protocol choices, and tests the resulting gap on arithmetic tasks.
1 INTRODUCTION
- Correctness-conditioned intervention tests estimate a different population quantity than evaluating all traces, motivating an analysis of how the restriction changes CoT faithfulness estimates.Prior protocols include correct-response analyses for Putnam shortcut detection and MMLU step-importance analysis.Source: §1 · p. 2
- The causal framework derives an exact observed selection gap and specifies assumptions for analysing difficulty confounding and shortcut-induced collider bias.Under an explicit conditional-independence premise, the gap can be expressed through correctness and compliance propensities.Source: §1 · p. 2
- The review of 18 papers distinguishes answer-correctness restrictions from agreement-based selection, benchmark construction, and training-data filtering.It links protocol choices to the selection operator analysed in the framework.Source: §1 · p. 2
- +7.4 percentage points is the measured selection gap in the primary GLM-5.3 arithmetic-task run.The experiments cover four model deployments and examine operation-count decomposition, reweighting, and unparseable-response handling.Source: §1 · p. 2
- The paper gives a direct rule for evaluation: use all traces when counterfactual compliance is observable for every instance.The protocol separately measures factual answer correctness and counterfactual compliance.Source: §1 · p. 2
2 RELATED WORK
Prior work studies CoT trace evaluation, faithfulness metrics, and selection in evaluation or training. This paper distinguishes correctness-based changes to the evaluation sample from other measurement changes and selection stages.
Intervention-based CoT faithfulness
- Prior CoT work evaluates trace interventions, injected cues, accuracy confounds, information flow, counterfactual simulation, step importance, unlearning, and benchmarking.The cited literature includes trace evaluations using truncation, paraphrase, and mistake insertion [Lanham et al., 2023].Source: §2 · p. 2
- Correctness filtering changes which traces enter the aggregate, unlike trace-timing or classifier-choice changes that alter measurement on a fixed sample.Classifier choice alone reverses reported rankings in one cited comparison, while including every trace does not resolve classifier sensitivity [Young, 2026].Source: §2 · p. 2
Causal approaches and shortcut learning
- Causal approaches study step-level faithfulness, causal models of CoT steps, vision-language bias, and confidence-based trace filtering; this paper instead analyses the evaluation sample.The sample-selection question is whether conditioning on answer correctness changes the aggregate faithfulness measure.Source: §2 · p. 3
Selection bias in evaluation
- Prior LLM evaluation research addresses position effects, metric disagreement, training-data collider bias, and causal validity or ground-truth accuracy of faithfulness metrics.This analysis characterizes selection in the sample on which a metric is computed.Source: §2 · p. 3
Causal inference and shortcut learning
- The paper connects correctness filtering to sample-selection and collider-bias frameworks, while distinguishing faithfulness from self-consistency and explanation-faithfulness tests.It treats correctness conditioning as an evaluation-time selection step and frames recoverability as an issue when excluded-instance compliance is unavailable.Source: §2 · p. 3
- The SCM includes shortcut competence as a possible non-CoT route to an answer, connecting the collider pathway to shortcut-learning research [Geirhos et al., 2020].The cited motivation includes evidence of hidden cues [Chen et al., 2025] and possible obfuscation under monitored-reasoning training [Baker et al., 2025].Source: §2 · p. 3
3 A DIAGNOSTIC FRAMEWORK FOR SELECTION BIAS
The framework treats correctness filtering as selection and separates difficulty-related pathways from shortcut-induced collider bias. It derives an observed gap, states the assumptions needed for propensity-based explanations, and identifies when all-trace evaluation is available.
3 A DIAGNOSTIC FRAMEWORK FOR SELECTION BIAS
- The framework models correctness conditioning as selection and analyses difficulty confounding and shortcut-induced collider bias under their respective assumptions.It first defines evaluation variables and derives the selection gap.Source: §3 · p. 3
3.1 STRUCTURAL CAUSAL MODEL
- The seven-variable SCM distinguishes difficulty, shortcut competence, CoT reliance, trace correctness, answer correctness, intervention assignment, and appropriate post-intervention answer change.Figure 3 marks the association that conditioning on correctness can open; it is not a causal edge.Source: §3.1 · pp. 3–4
3.2 THE GENERAL BIAS FORMULA
- The observed selection gap is an algebraic identity on answer correctness and intervention compliance, while its expression through latent propensities requires A ⊥ C | {R, S, I, D}.The premise constrains the SCM; it does not follow from it.Source: §3.2 · pp. 5–6
3.3 DIFFICULTY CONFOUNDING
- When difficulty lowers both correctness and compliance, conditioning on correct answers selects easier instances and inflates the faithfulness estimate under A ⊥ C | {D, I}.Operation count is used as an observed difficulty proxy; its between-stratum share is descriptive, and attributing a causal contribution requires varying difficulty while holding other causes of correctness fixed.Source: §3.3 · p. 6
3.4 SHORTCUT COMPETENCE AND COLLIDER BIAS
- With an active shortcut branch, conditioning on correctness opens the R → C ← S collider; the framework derives predictions for the gap and model-ranking reversals.The neutral-framing detector found no shortcut use, and separate explicit-reliance probes did not test the peak or reversal predictions.Source: §3.4 · p. 6
3.5 IMPLICATIONS FOR EVALUATION PRACTICE
- Evaluators should report the scored population; when counterfactual compliance is measurable for every instance, evaluating all traces directly computes population faithfulness.When it is not, the framework recommends stating the correctness condition and measurement limits, and reporting both quantities when estimable.Source: §3.5 · p. 6
4 REPORTING POPULATIONS IN PRIOR EVALUATIONS
The protocol review distinguishes final evaluation-denominator restrictions from agreement-based selection, benchmark construction, and training-data filtering. It records the population used by each analysis rather than assigning one filtering label to an entire paper.
4 REPORTING POPULATIONS IN PRIOR EVALUATIONS
- The review inspects 18 papers and records each protocol’s evaluation population separately from training-example selection and benchmark construction.It compares concrete protocol choices; estimating prevalence would require a systematic sampling frame and common unit of analysis.Source: §4 · p. 7
- Correct-response analyses in Putnam shortcut detection and MMLU step-importance work use distinct reporting populations, while other analyses in those papers use different criteria.A correct-only score estimates a conditional quantity whose difference from the population score is the subject of this paper.Source: §4 · p. 7
- CoT/no-CoT agreement and hint-induced answer changes are selection rules distinct from correctness filtering; agreement does not require either answer to be correct.A cited accuracy-confound analysis averages agreement over the dataset and does not establish a correctness-filtered denominator.Source: §4 · p. 7
- Positive human process-quality labels, benchmark ground-truth construction, and training-pool filtering are selection stages distinct from restricting an evaluated model’s final answers.Comparisons require specifying when selection occurs and which population the resulting score describes.Source: §4 · p. 7
5 EXPERIMENTAL DESIGN
The study compares correctness-filtered and all-trace faithfulness on procedural tasks with known causal structure. Deterministic oracles make counterfactual compliance measurable across factual correctness outcomes.
5 EXPERIMENTAL DESIGN
- The evaluation compares correctness-filtered and all-trace estimators on procedural tasks with known causal structure.Deterministic counterfactual answers make compliance observable for every instance.Source: §5 · p. 7
5.1 PROCEDURAL TASK DESIGN
- Symbolic replay supplies exact counterfactual answers, while natural-language edits generally lack a unique verifiable answer.Extending the results to natural-language tasks requires a separate evaluation.Source: §5.1 · p. 8
- The three task families are arithmetic DAGs, Boolean circuits, and graph reachability, each with an editable dependency structure.The generators also use semantics-preserving placebo edits; a faithful baseline gives Fpop = Fsel = 1.000 (n = 50).Source: §5.1 · p. 8
- A cue whose validity is controlled by q remains unchanged under intervention, so cue-reliant models can be factually correct yet counterfactually non-compliant.The generator’s q is distinct from theoretical shortcut prevalence ps; without a planted cue, the design targets S ≈ 0, though ∆ = 0 also requires no other correctness–compliance dependence.Source: §5.1 · p. 8
5.2 ALL-TRACE INTERVENTION PROTOCOL
- Each instance is tested in separate factual, causal-intervention, and placebo conditions, with the oracle scoring the edited-task answer.The headline analysis scores missing or unparseable responses as non-compliant; treating them as missing produces a smaller estimated gap.Source: §5.2 · p. 8
5.3 EVALUATION METRICS
- Confidence intervals use 10,000 instance-level bootstrap resamples, and the pre-specified primary analysis is one pooled Cochran–Mantel–Haenszel test.Individual-model p-values are descriptive, no multiplicity adjustment is applied to the single primary hypothesis, and headline runs use temperature = 0.Source: §5.3 · p. 9
6 DEPLOYMENT RESULTS
All four sub-ceiling deployments have positive selection-gap point estimates, with the primary GLM-5.3 run showing the largest reported headline result. Operation-count variation accounts for part, but not all, of that gap.
6 DEPLOYMENT RESULTS
- All four sub-ceiling runs have positive selection-gap point estimates, while two reported intervals cross zero.The two 100%-accuracy controls have zero gaps by the boundary condition P(C = 1) = 1.Source: §6 · p. 9
- +7.4pp is the GLM-5.3 selection gap (n = 1000, p < 0.0001); pooled across four deployments, the CMH odds ratio is 1.70.The pooled association survives removal of any single deployment, though excluding GLM-5.3 reduces the pooled z from 5.19 to 2.24 (p = 0.025).Source: §6 · p. 9, Figure 7 · p. 10
- Operation-count weighting removes 23–34% of the raw gap across two specifications, leaving most of the gap after adjustment.The between-stratum share depends on the covariate: oracle-displacement magnitude accounts for 26.6% on the same GLM-5.3 records.Source: §6 · p. 10
- The trace-matching detector records 0% shortcut use under neutral framing and cannot identify unacknowledged use of a correct cue.The headline runs do not test the collider predictions; the cue-correctness contrast is −1.6pp (p = 0.81) after excluding nine null rows.Source: §6 · p. 10
- 42.8% of the GLM-5.3 gap lies between operation-count strata; the 4.212pp within-stratum component remains unattributed.The descriptive between-stratum interval is [26.8, 69.1]%; the split does not identify a causal effect of difficulty.Source: §6 · p. 10, Figure 8 · p. 11
- Reweighting leaves the selected estimate above the direct all-trace estimate, and dropping null responses leaves all four deployment gap estimates positive.Dropping nulls reduces every deployment’s gap; under the primary convention, missing or unparseable responses are scored as non-compliant.Source: §6 · pp. 10–11
- The strict-reliance shortcut-mixture special case overpredicts measured gaps by a factor of 6.4–11.7 and gives flash an invalid negative implied shortcut prevalence.These discrepancies support estimating the observed (A, C) gap and operation-count decomposition directly rather than imposing the mixture assumptions.Source: §6 · pp. 11–12, Table 2 · p. 12
7 CONCLUSION
Correctness conditioning changes the population used to estimate CoT faithfulness, and the observed gap is quantified as ∆ = Cov(A, C)/P(C=1). On arithmetic DAGs, the primary GLM-5.3 run has a positive gap, while most of it remains unattributed after the operation-count decomposition.
7 CONCLUSION
- The conclusion recommends stating the reporting population and using all traces when counterfactual compliance is observable for every instance.The primary GLM-5.3 gap is +7.4pp (n = 1000, p < 0.0001); operation-count variation accounts for 42.8%, leaving the remainder unattributed.Source: §7 · p. 12
REPRODUCIBILITY STATEMENT
The reproducibility statement specifies deterministic task generation, exact symbolic outputs, and the reported model-call procedure.
REPRODUCIBILITY STATEMENT
- Procedural generators use deterministic seed streams, and the symbolic oracle computes exact outputs.Headline calls use temperature = 0; when an endpoint returned different responses at that setting, the first completed response was retained.Source: REPRODUCIBILITY STATEMENT · p. 12
DATA AND CODE AVAILABILITY
The project materials are available online, while the release omits raw logs and other row-level materials needed for exact reanalysis.
DATA AND CODE AVAILABILITY
- The website, project film, figures, PDF, release checks, and source bundle are available through the project resources.The source bundle includes the LaTeX files and figure assets needed to compile the manuscript.Source: DATA AND CODE AVAILABILITY · p. 12
- The release provides aggregate table and figure results, but excludes raw logs, model completions, intermediate exports, and credential-bound generators.Exact row-level reanalysis requires a separate anonymized data release.Source: DATA AND CODE AVAILABILITY · p. 12
ETHICS STATEMENT
The ethics statement describes a synthetic study without human subjects, private data, or model training; the remaining sections specify its experimental setup and theory and implementation checks.
ETHICS STATEMENT
- The ethics statement reports no human subjects, private data, or model training, and describes the procedural tasks as synthetic and nonsensitive.Language-model calls used publicly available commercial endpoints under standard terms of service.Source: ETHICS STATEMENT · p. 12
A EXPERIMENTAL CONFIGURATION
- The real-model study uses arithmetic DAGs, with a primary run of n = 1000, additional sub-ceiling runs, and two 100%-accuracy boundary controls.Task families and phases use different sample sizes; the arithmetic generator uses depth [2, 4], values [1, 20], and cue-validity control q = 0.7.Source: A EXPERIMENTAL CONFIGURATION · p. 15
C SIMULATION: CROSS-FAMILY DETAIL
- Across three task families at theoretical shortcut prevalence ps = 0.7, simulated gaps average +8.2pp and range from +7.9 to +8.5pp.The mean absolute implementation error against Corollary 1 is 0.006; these simulations check implementation under imposed assumptions rather than independently testing the theory.Source: B IMPLEMENTATION CONSISTENCY CHECK · p. 15, Table 3 · p. 16
B IMPLEMENTATION CONSISTENCY CHECK
- Under the stated Appendix K parameters, conditioning reverses the ranking of Models A and B; 40.6% of full-grid candidates reverse at π10 = 0.001.The example requires an active shortcut branch, which was not detected in real-model runs under neutral framing.Source: B IMPLEMENTATION CONSISTENCY CHECK · p. 16
D ESTIMAND: WITHIN-STRATUM VS. MARGINAL
- Operation-count strata retain gaps beyond between-stratum variation, while the marginal gap also includes a composition term from correctness-based changes in difficulty distribution.The operation-count decomposition leaves collider contribution unidentified; the shortcut-mixture special case is inconsistent with the observed deployments.Source: D ESTIMAND: WITHIN-STRATUM VS. MARGINAL · p. 16
E.1 COROLLARY 4: HUMP ON THE ACCURACY AXIS
- In the shortcut-mixture special case, the accuracy-axis peak is P* = √ps; with ps = 0.7 it lies at 83.7%, above the four sub-ceiling deployments.The ceiling controls cannot locate the peak because correctness conditioning gives Δ = 0 when P(C=1) = 1.Source: D ESTIMAND: WITHIN-STRATUM VS. MARGINAL · p. 16, §E.1 · p. 17
F COLLIDER PATHWAY: FULL DERIVATIONS
The derivations characterize correctness-selection bias under strict reliance, including its hump-shaped response and possible ranking reversals; the review and power analysis delimit the empirical evidence.
F COLLIDER PATHWAY: FULL DERIVATIONS
- Under strict reliance, the gap’s sign is determined by whether CoT reliance raises or lowers correctness, and the special case yields a unique interior peak in reliance and accuracy.The peak occurs at P* = √ps; the special case requires an active shortcut branch, and its peak near 50% accuracy requires ps ≈ 0.25.Source: F COLLIDER PATHWAY: FULL DERIVATIONS · p. 18
- The strict-reliance gap is derived by subtracting population faithfulness from selected faithfulness, and its simplified form assumes specified correctness rates for each route.For multiple-choice tasks, the general formula applies when π00 = 1/k > 0; the simplified form is close when 1/k is much smaller than ps.Source: F COLLIDER PATHWAY: FULL DERIVATIONS · p. 18
- For fixed reliance, the simplified gap strictly decreases as shortcut prevalence increases.The shortcut-mixture special case is a theoretical mixture and does not estimate shortcut prevalence in the real-model runs.Source: F COLLIDER PATHWAY: FULL DERIVATIONS · p. 18
G MEASUREMENT VALIDATION
- The faithful baseline validates the measurement procedure: at n = 50, Fpop and Fsel are both 1.000, with gap 0.000 and shortcut rate 0%.The oracle introduces no false non-compliance, and interventions correctly modify downstream values.Source: G MEASUREMENT VALIDATION · p. 19
H PROTOCOL REVIEW DETAILS
- The review records the reporting population for each inspected protocol rather than assigning one filtering label to an entire paper.It distinguishes training-example selection, benchmark construction, and restriction of the final evaluation denominator.Source: H PROTOCOL REVIEW DETAILS · p. 19, Table 4 · p. 19
H.1 SCOPE AND INTERPRETATION
- The protocol review compares inspected sources, but estimating field-wide prevalence would require systematic sampling.The review also distinguishes denominators that include correct and incorrect outputs from post-hoc selection by factual correctness.Source: §H.1 · p. 20
I POWER ANALYSIS
- Pilot-based projections estimated low power at n = 100, while the scaled GLM-5.3 run at n = 1000 achieved p < 0.0001.The projections depend on the pilot effect size.Source: I POWER ANALYSIS · p. 20, Table 5 · p. 20
J WITHIN-STRATUM DIFFICULTY ANALYSIS
Correctness filtering selects a different population, with operation-count strata explaining only part of the observed gap and propensity weighting leaving a residual. Reversal sensitivity, shortcut probes, and ceiling controls qualify the evidence: the probes are exploratory, and ceiling runs cannot locate the predicted peak.
J WITHIN-STRATUM DIFFICULTY ANALYSIS
- Medium-to-hard strata (5–8 operations) have the largest gaps (+7.0 to +8.5pp), while easy strata (1–3) have smaller gaps (−1.1 to +2.8pp).At 11 operations the gap is +6.0pp, and operation count does not consistently order strata by difficulty.Source: J WITHIN-STRATUM DIFFICULTY ANALYSIS · pp. 20–21, Table 6 · p. 21
J.1 EXACT COVARIANCE DECOMPOSITION
- 42.8% of the GLM-5.3 gap lies between operation-count strata, but the within-stratum component remains unattributed and the share varies with the chosen covariate.The same records yield a 26.6% share using oracle-displacement magnitude; these descriptive shares do not identify a causal contribution of latent difficulty.Source: §J.1 · p. 22, Table 8 · p. 22
J.2 INVERSE PROPENSITY WEIGHTING (IPW) MODEL DETAILS
- Logistic IPW reduces the selected gap by 23% but leaves a +5.7pp residual; both adjustments depend on propensity specification and do not identify a causal decomposition.The alternative per-stratum propensity reduces the gap by 34%.Source: §J.2 · pp. 22–23
K REVERSAL SENSITIVITY ANALYSIS
- At π10 = 0.001, 40.6% of the 39,999 candidate pairs reverse rankings, with lower-population-faithfulness candidates scoring higher after correctness selection.Exact-rational evaluation reproduces the counts, and requiring inequality margins above 10^-8 leaves them unchanged.Source: K REVERSAL SENSITIVITY ANALYSIS · pp. 23–24, Figure 12 · p. 23
L LEAVE-ONE-OUT SENSITIVITY ANALYSIS
- The four-deployment pooled association is OR = 1.70 (p < 10^-6), and leave-one-out tests remain significant (p < 0.03), though GLM-5.3 is the largest contributor.The pooled test reports z = 5.19 and I2 = 0%.Source: L LEAVE-ONE-OUT SENSITIVITY ANALYSIS · p. 24, Table 11 · p. 24
M SHORTCUT ELICITATION DETAILS
- In targeted probes, deepseek-v4-pro followed incorrect-value cues on 12 of 12 informative V4 instances, all excluded by the correctness filter; following varied by framing and model.The exploratory post-hoc probes use small samples and a separate endpoint, and do not estimate a population rate or test the accuracy-axis peak.Source: M SHORTCUT ELICITATION DETAILS · pp. 24–25
N EMPIRICAL GAP VS. ACCURACY / O CEILING VALIDATION
- At the theoretical peak P* = √ps, tested sub-ceiling deployments fall below the peak when ps = 0.7; ceiling controls cannot locate it because correctness filtering is vacuous there.On Boolean circuits, GLM-5.3 has 99.5% accuracy and a 0.0pp gap, versus +7.4pp on intermediate-accuracy arithmetic DAGs; the circuit task was too easy to measure below ceiling.Source: N EMPIRICAL GAP VS. ACCURACY · p. 26, O CEILING VALIDATION · p. 26, Figure 14 · p. 27
P COMPLIANCE ON INCORRECT TRACES AND NULL HANDLING / Q FULL DISCUSSION
- Compliance on incorrect traces is ambiguous, and scoring nulls as non-compliant increases the gap by +0.12 to +0.81pp; all four gaps remain positive under either null convention.The detector’s zero neutral-framing rate cannot exclude collider bias, while operation-count strata account for 21.9% (flash) and 42.8% (GLM-5.3), leaving most of the gap unattributed.Source: P COMPLIANCE ON INCORRECT TRACES AND… · p. 27, Q FULL DISCUSSION · pp. 27–28
R FULL LIMITATIONS
The empirical evidence is bounded to arithmetic deployments, and the proposed mechanisms and residual attribution remain qualified. Incorrect-trace compliance is ambiguous, while decoding and language coverage are narrow.
R FULL LIMITATIONS
- Two of four sub-ceiling models reach individual significance; the other two have positive estimates whose intervals cross zero at n = 100–200.The 100%-accuracy controls are algebraic identities and are excluded from the pooled test.Source: R FULL LIMITATIONS · p. 28
- Evidence beyond arithmetic DAGs is limited: Boolean circuits were tested only at ceiling, graph reachability was excluded after 3/20 responses parsed, and non-ceiling natural-language CoT remains untested.The graph-reachability run supports no inference about the mechanism.Source: R FULL LIMITATIONS · p. 28
- The measured operation-count association is consistent with difficulty confounding but does not identify a causal contribution to the headline gaps.The collider pathway requires a shortcut branch, which neutral framing did not detect; a targeted explicit-reliance probe reached it on multiple deployments.Source: R FULL LIMITATIONS · p. 28
- IPW removes 23–34% of the gap across propensity specifications, but the logistic specification leaves +5.7pp unattributed.The data do not distinguish an R-confounder from other residual causes.Source: R FULL LIMITATIONS · p. 28
- Compliance on incorrect traces is semantically ambiguous, so all-trace evaluation removes correctness selection without resolving that measurement ambiguity.The reported rate for C = 0 is 38–51% across models.Source: R FULL LIMITATIONS · p. 28
- Headline runs use greedy decoding and English-only prompts; stochastic decoding and multilingual prompts remain unevaluated.The same model identifier behaved materially differently on two endpoints, so gaps are reported by deployment.Source: R FULL LIMITATIONS · p. 28