PaperlayerView exact revision on arXiv

Source-linked AI summary

What Was Said, Not What Was 'Thought': Type-6 Logic for CoT Verification

Adrian de Wynter

arXiv:2609.38420v1cs.LOcs.AIcs.CL

TL;DR

Long CoT traces are difficult to verify, and answer correctness can miss incoherent reasoning steps. The paper introduces Type-6 logic and a graph-based verifier to assess trace coherence. Across tested methods, their failure decisions were not interchangeable, and Type-6 was the most-agreed-with method.

  • Problem

    Long CoT traces are difficult for humans to verify, while answer-correctness measures can miss incoherent reasoning steps.

    Source: §1 · pp. 1–2

  • Method

    Type-6 models trace commitments and updates, and its verifier reconstructs a graph to check reasoning paths against Type-6 rules.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    No tested method pair produced interchangeable failure decisions; Type-6 was most-agreed-with, correlating with PRMs (ρ = 0.31) and LLM-judges (ρ = 0.25).

    Source: §7 · p. 9

  • Takeaways & Limitations

    Type-6 evaluates the interpreted trace rather than the generator’s knowledge, and requires no gold labels.

    Source: §1 · p. 2

    Type-6 assesses internal coherence, not the truth of conclusions or authorial intent, and should not be used as a factuality checker or correctness certificate.

    Source: ETHICS STATEMENT · p. 9

Abstract

from arXiv · show

We introduce Type-6 logic, a variant of dynamic epistemic logic augmented with two operators (uncertainty and recurrence), designed to model the inferential dynamics of contemporary large language model (LLM) chain-of-thought (CoT) reasoning. Type-6 accounts for common LLM reasoning pathologies such as unlicensed revision, enthymemes, loopbacks, and unverifiable/incorrect claims. We propose a verifier based on Type-6 logic that builds a graph out the trace, and checks it against Type-6's axioms and inference rules. We evaluate our framework on LLM-generated CoTs four splits spanning formal and informal reasoning. Our verifier detects structurally unsound reasoning steps that surface-level heuristics miss, and allows for easy visualisation of the model's reasoning process. In our corpus, our verifier shows that derived contradiction is the most common hard-fail category in CoT, and that only about 3\% of the propositions of a trace have impact on the final derivation. Ablation studies show that other verification methods (LLMs-as-judges, other neurosymbolic approaches, etc.) cannot be considered interchangeable: for example, agreement between LLMs-as-judges and LINC is $\kappa \approx 0.034$, and this persists within a method across underlying models. Type-6, however, is the most agreed-with method amongst the ones we tested. We prove our verifier runs on average-case linear time; and release our logic specification and artefacts.

1 INTRODUCTION

CoT traces can be difficult to verify, and answer correctness or surface-level coherence measures can miss unsound steps. Type-6 treats the trace as an interpretable reasoning artefact and applies explicit rules to its reconstructed structure.

Motivation

  • CoT is widely used to induce complex problem-solving, but its long traces are often intractable for humans to verify.Automated methods often conflate whether an answer is correct with whether the reasoning steps cohere.Source: §1 · p. 1
  • A trace can reverse a commitment without derivation or reach a conclusion from premises it never establishes, even if its final answer is correct.Answer-correctness and NLI scorers can miss these pathologies of reasoning-as-written.Source: §1 · pp. 1–2

Approach

  • Type-6 models trace commitments and updates, then checks a multigraph reconstruction against its rules to produce coherence, sequential-verification, and per-path reports.Contradictions are logged rather than suppressed, and the agent’s commitments are retained.Source: §1 · p. 2

Contributions

  • The framework distinguishes hard and soft contradictions, unresolved residual signals, reasoning avoidance, and disagreement across reasoning paths.Each category is mapped to concrete verifier output.Source: §1 · p. 2

Evaluation and findings

  • Type-6 was the most-agreed-with tested method, while the methods’ failure decisions were not interchangeable.It correlated with LLM-judges (ρ = 0.25) and PRMs (ρ = 0.31) more strongly than those methods correlated with each other; only about 3% of trace propositions lay on the main derivation chain.Source: §1 · p. 2

2 BACKGROUND AND RELATED WORK

Prior CoT evaluation includes outcome, step-level, judgment-based, and reference-free coherence methods, as well as structured symbolic approaches. Type-6 is positioned as an artifact-level approach that represents informal reasoning and modal commitments.

Scoring chain-of-thought

  • Outcome-based scorers assess final-answer correctness, while process reward models score individual steps but require trained models and do not generalise well beyond their domain.Judgment-based scorers are flexible but have known position and style biases and are difficult to audit at the step level.Source: §2 · p. 2

Neurosymbolic and structured-intermediate methods

  • LINC translates premises and conclusions into first-order logic and uses a theorem prover, but such approaches typically do not cover informal reasoning or modal commitments.Type-6 preserves these signals alongside structural markers and per-path decompositions.Source: §2 · pp. 2–3

Consistency, hallucination, and self-verification

  • Self-checking and verifier-generator methods target the generating LLM, whereas Type-6 evaluates the generated trace as an artifact.Hallucination detection shares the artifact framing but focuses on factual accuracy rather than inferential structure.Source: §2 · p. 3

Formal and informal foundations

  • Type-6 uses update semantics indexed by the trace, retains knowledge and belief operators, and has a four-valued truth space that avoids assigning subjective credence to the agent.Its paraconsistent design logs conflicting commitments rather than deriving arbitrary consequences.Source: §2 · p. 3

3 TYPE-6 LOGIC

Type-6 tracks proposition states as a trace unfolds, distinguishing truth value, commitment strength, and doubt. Its update and back-propagation rules govern revisions, contradictions, and the verification of paths through a reconstructed trace.

3.1 OPERATORS AND STATE

  • Each proposition has a state Σ(π) = (τ, σ, δ), tracking truth value, commitment strength, and doubt as the trace is read.The truth values are T, Uc, Uk, and F; commitment strength is uncommitted, belief, or knowledge.Source: §3.1 · p. 3, §3.2 · p. 4
  • Modal operators B and K encode commitment strength, while doubt, loopback, and pivot operators govern doubt or trace structure.Operators are read left-to-right, so ? K π and K ? π express distinct configurations.Source: §3.1 · pp. 3–4

3.2 LANGUAGE AND UPDATE RULES

  • Statements use unary or connective-led forms; modal updates set commitments, while THEN closes a constraint chain and triggers back-propagation.Back-propagation uses extended strong Kleene semantics; unclear Uc operands make the enclosing derivation abstain.Source: §3.2 · pp. 4–5, Table 1 · p. 4

3.3 UPDATES AND FAILURE MODES

  • A revision is licensed only when the intervening segment contains a jump or loopback and a constraint targeting the revised proposition.Licensing captures effort, not correctness; a conclusion with no constraint-directed derivation is flagged as reasoning-avoidant.Source: §3.3 · p. 5
  • Hard contradictions violate update or back-propagation rules; softer contradictions may reflect legitimate epistemic moves, while unresolved doubt or unknowability at the conclusion is a residual signal.Residual signals become hard when their proposition lies on the direct implication chain to the answer.Source: §3.3 · p. 5, Table 2 · p. 6

3.4 FROM TYPE-6 TO TYPE-6-BASED VERIFICATION

  • Verification reconstructs a graph from the trace and applies Type-6 semantics along candidate source-to-sink paths.Comparing terminal states identifies cross-path disagreement and under-resolution; residuals on a path’s direct implication chain are elevated to hard signals.Source: §3.4 · p. 5

4 METHODS AND EXPERIMENTAL SETUP

The verifier annotates a CoT, constructs a graph, and produces reports from its paths and sequential state. Its evaluation uses a reviewed corpus of formal and informal reasoning traces and compares the system with several baseline methods.

4.1 TYPE-6-BASED VERIFICATION

  • The verifier annotates a raw CoT, builds a multigraph, and produces reports without filling missing warrants or overruling committed values.The implementation operates in five stages and runs in quasilinear time when accounting for CoT sparsity.Source: §4.1 · p. 6

4.2 CORPUS CREATION AND VERIFIER REVIEW

  • The 996-trace corpus spans multiple-choice and argumentation examples; trace-length medians were 66, 54, 44, and 52 line-statements across CoT-Logic, DebateLab, MMLU, and handcrafted subsets, averaging 55.5.Three Type-6-trained annotators reviewed the traces in full over ten months; raw pairwise agreement was κ=0.969 and 0.929 for the two criteria, while LLM operator-and-proposition assignment accuracy was 97.2%.Source: §4.2 · pp. 6–7

4.3 MEASUREMENT

  • The comparison covers five methods, including ROSCOE, LINC, PRMs, LLM-as-judge, and a model-free trivial baseline.Baselines operate on natural-language traces, matching the verifier’s intended post-hoc use; their scoring and hard-fail criteria differ.Source: §4.3 · p. 7

5 RESULTS

Across verification methods, hard-fail decisions and scalar scores show weak agreement, with Type-6 the most agreed-with method. The per-method breakdown also reveals baseline-specific failures and structural findings about the traces.

5.1 GENERAL RESULTS

  • κ ranges from −0.005 to 0.15 across nearly all method pairs, and no methods’ hard-fail decisions are interchangeable; Type-6 is the most agreed-with method.The highest hard-fail agreement is LLM-judge–Type-6 (κ = 0.15).Source: §5.1 · p. 7, Table 3 · p. 8
  • No scalar-correlation pair exceeds ρ = 0.4, and most are below 0.2; ROSCOE anti-correlates with all methods (ρ ∈ [−0.40, −0.03]).The authors interpret ROSCOE’s anti-correlations alongside its 100% hard-fail rate as indicating systematically different measurements.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 ABLATION: PER-METHOD MEASUREMENTS

  • Type-6’s most common hard-fail is derived contradiction (409 traces; mean 5.28 events per triggered trace), while only ∼3% of propositions lie on the direct implication chain.Reasoning-avoidance appears in 548 traces and unresolved doubt at the sink in 495.Source: §5.2 · p. 8
  • Baseline behavior varies sharply: ROSCOE hard-fails 100% of traces, PRMs hard-fail 68.4–100%, and LINC often emits unknown or depends on its translation model.GPT-5.6 LINC produces informative verdicts from a partial formalisation (median coverage 0.006); other listed LINC models mostly emit unknown and error otherwise.Source: §5.2 · p. 8
  • LLM judges split by model capability, and their coherence measurements on this corpus are dominated by the judge model rather than the trace.Capable judges hard-fail 61–68% of traces and agree modestly with one another; other judges hard-fail 98–100% and have pairwise κ below 0.01 against the capable judges.Source: §5.2 · p. 8

6 DISCUSSION

Verification methods disagree substantially, with performance depending on the model and data setting. The discussion frames Type-6 as a structural approach that complements, rather than replaces, other verifiers.

6 DISCUSSION

  • Across hard-fail decisions and scalar scores, verification methods disagree in ways invariant across models and data splits.The authors attribute some baseline behavior to surface features, out-of-distribution effects, and model quality.Source: §6 · pp. 8–9
  • Type-6 operates in a different regime from coherence-axis baselines, imposing few assumptions on the trace while retaining a formal framework.The discussion frames this structural focus as the reason for Type-6’s design.Source: §6 · p. 9
  • The methods are complementary: LLM judges provide rough approximate signals, PRM and LINC work within their domains, and Type-6 supports statement-level auditability.Type-6 addresses where reasoning went astray or whether it is good enough, rather than simply whether it failed.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents Type-6 as a trace-level structural verifier and reports non-interchangeable baseline decisions alongside trace pathologies its reports surface. Its current development targets single-agent contemporary CoT, with broader settings and a completeness result left open.

7 CONCLUSION

  • No method pair produces interchangeable failure decisions (κ < 0.4) or scalar scores (ρ < 0.4); Type-6 is most agreed-with, correlating with PRMs at ρ = 0.31 and LLM judges at ρ = 0.25.Those correlations exceed the corresponding correlations between those methods and each other.Source: §7 · p. 9
  • Type-6 records commitments and contradictions with statement-level provenance, and encodes meandering and loopbacks as trace structure.The verifier compares methods on 996 traces spanning multiple-choice and argumentation.Source: §7 · p. 9
  • Reasoning avoidance appears in 548 of 996 traces, while only ∼3% of propositions drive the conclusion.The conclusion also reports on-chain unresolved commitments as a surfaced structural pathology.Source: §7 · p. 9
  • Type-6 was developed for single-agent contemporary CoT; performance in multi-agent or structured reasoning and a completeness theorem remain future work.The authors say its update semantics and per-path decomposition extend without modification to multi-agent and tree-of-thought systems, but leave performance in that setup open.Source: §7 · p. 9

ETHICS STATEMENT

The ethics statement limits Type-6’s intended interpretation to coherence assessment and describes the reviewed data and annotation setting.

ETHICS STATEMENT

  • Type-6 checks internal trace coherence, not conclusion truth or author intent, so it is not a factuality checker or correctness certificate.Annotators reviewed public-benchmark and handcrafted logic, academic multiple-choice, and philosophical-argumentation traces without personal identifying data or harmful or distressing content.Source: ETHICS STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

The verifier is deterministic on a fixed annotated trace and supports statement-level auditing. The authors provide its formal materials and implementation artefacts.

REPRODUCIBILITY STATEMENT

  • A fixed annotated trace produces identical reports on every run, with statement-level behavior that is auditable.The authors provide formal rules and metatheory, plus the logic specification, parser, visualization code, annotation rubric, and annotated dataset.Source: REPRODUCIBILITY STATEMENT · pp. 9–10

AI USE STATEMENT

LLMs support trace evaluation and several baseline methods in this work. They also assisted with verifier development, writing, and proofs, which the authors reviewed or checked.

AI USE STATEMENT

  • LLMs were used as evaluators in LINC, PRM, and LLM-as-a-judge baselines, as well as in parts of the trace and verifier system.Claude Opus-4.8 aided verifier-code development and writing; GPT-5.6 Sol aided proof development, and the authors checked the proofs by hand.Source: AI USE STATEMENT · p. 10

A LIMITATIONS

The supplied material bounds the evaluation and formal guarantees, and illustrates Type-6 through worked traces, graph patterns, and rule descriptions. It also specifies the verifier’s model setup and update semantics.

A LIMITATIONS

  • Evaluation covers multiple-choice and argumentation, with minor word-problem reasoning; verification depends on LLM annotation and Type-6 lacks a completeness theorem.The authors also identify closed-model dependence for annotation as a limitation.Source: A LIMITATIONS · p. 14

B TWO WORKED EXAMPLES

  • In the Chinese Nation example, Type-6 passes at 3.5, recognizing hedged reasoning and off-chain structural issues while other methods hard-fail or fail to conclude.The example also shows an interrupted, unclosed IF antecedent; Type-6 surfaces unresolved off-chain propositions as residuals rather than hard failures.Source: B TWO WORKED EXAMPLES · pp. 14–15, Table 4 · p. 15
  • Figure 2 presents four canonical edge-semantics patterns; the pipeline is polynomial on mostly-linear traces and effectively linear under typical sparsity.The worst case is combinatorial in parallel and revisit edges, bounded by a cap and sampling fallback.Source: B TWO WORKED EXAMPLES · p. 15, Figure 2 · p. 16

C.1 MODELS, HARDWARE, AND PARAMETERS

  • GPT-5 primarily handles extraction and verification, GPT-4.1 is a fallback, and baseline token parameters were tuned on 5% of the corpus.High failure rates led to discarding some model configurations.Source: §C.1 · p. 16

D.1 JUDGEMENTS

  • The judgments describe proposition states, statement-driven configuration transitions, and constraint-forced values evaluated from Boolean conditions and target polarity.For implications, forcing fires only when the antecedent evaluates to T.Source: §D.1 · p. 17

MODAL RULES / ASSERTION AND STRUCTURAL RULES

CHAIN-ACCUMULATOR RULES / MODIFIER RULES / BACK-PROPAGATION RULES

  • If-Open terminates malformed pending chains; Then-Close builds constraints, NOT rules encode polarity, and back-propagation pins forced unknowns or logs conflict and unverifiability.Conflict and unverifiable events are emitted only when a constraint’s outcome changes.Source: Figure 5 · p. 18, Figure 6 · p. 19, Figure 7 · p. 19

D.3 NESTED OPERATOR SCOPING

  • Operator order matters: ? K π commits knowledge on π under doubt, while K ? π commits to the doubt-state without updating π’s value.Operators scope left-to-right over the stack to their right.Source: §D.3 · p. 19

D.4 METATHEORETIC PROPERTIES

This section sets out Type-6’s metatheoretic properties and connective semantics, and characterizes the verifier’s graph and path policies and computational costs under stated assumptions. It also reports variation in hard-fail rates and trace-level scores across verification methods.

D.4 METATHEORETIC PROPERTIES

  • Type-6 rules are deterministic, resolution is nondecreasing absent licensed revision, and local contradictions are logged and localized rather than propagated.Type-6 is deliberately incomplete relative to classical propositional or modal semantics; contradictions do not license arbitrary conclusions, and inference on other propositions remains unaffected.Source: §D.4 · pp. 20–21

D.5 SEMANTICS OF LOGICAL CONNECTIVES

  • Uc makes a constraint abstain computationally, while negation leaves both underdetermined values Uc and Uk unchanged.The connective tables extend strong-Kleene short-circuit behavior with Uc as abstention, and k-ary generalizations compose associatively.Source: §D.5 · p. 21

E.1 PRELIMINARIES AND NOTATION

  • The verifier constructs a labelled multidigraph with distinct logical, gap, jump, loopback, meander, and unknown edge kinds.Its path analysis is parameterized by node and edge counts, maximum out-degree and edge multiplicity, loopback budget, revisit-edge count, enumeration cap, sample count, and walk-length cap.Source: §E.1 · pp. 21–22

E.2 GENERAL WORST-CASE BOUNDS

  • Tverify = O(n^2t), Tpaths is Tenum or Tsample, and total space is O(nt + N·p̄).Here N is the number of scored paths and p̄ their mean length; the full-pipeline bound combines verification with either path enumeration or sampling.Source: §E.2 · p. 22

E.2.1 PARSING AND CONSTRUCTION

  • Graph construction takes O(nt) time and space, and is O(n) when statement length t is constant.The graph has O(nt) nodes and edges; parsing each statement takes O(t).Source: §E.2.1 · p. 22

E.2.2 PATH ENUMERATION AND SAMPLING

  • Enumeration takes O(Nemit·(|V| + L + J)·∆ + Wdead) time and O(|V| + L + J) auxiliary space.Sampling takes O(S·ℓmax^2·∆) time and O(ℓmax) auxiliary space per walk.Source: §E.2.2 · p. 22, §2 · p. 23

2. Loopback budget: at most L loopback edges per path

  • Path enumeration permits revisits only through jumps, loopbacks, or binary-connective edges, while the loopback budget caps path length and ensures termination.The budget does not reduce the leading factorial worst-case path-count bound, which can be realized without loopbacks.Source: §2 · p. 23

E.2.3 VERIFIER COST

  • Sequential and chain updates cost O(t) per statement, while back-propagation costs O(|Ki|·κmax) at step i.Across the trace, VERIFIERSTEP costs O(n^2t).Source: §E.2.3 · pp. 25–26

E.3 PROOF OF THEOREM E.1

  • In the worst case, the pipeline uses O(nt) parse/build time, O(n^2t) verification time, Tpaths for path processing, and O(N·p̄) scoring time.Total space is O(nt + N·p̄).Source: §E.3 · p. 26

E.4 THE MOSTLY-LINEAR REGIME

  • For mostly-linear graphs with t, µ, ∆, L, κmax = O(1) and B = O(log n), total time is O(n + min(K, n^c)·n) for enumeration or O(n + S·n^2) for sampling.Space is O(n + N·p̄), and c depends on backward-edge sparsity.Source: §E.4 · p. 28
  • For traces of length 20–100, graphs have |V| ≈ n, ∆ ≤ 4, µ ≤ 2, and a handful of backward edges; enumeration stays within K = 10^4 and verification runs linearly.The coherence report is produced in polynomial time in trace length.Source: §E.4 · pp. 27–28

F.1 HARD-FAIL AND TRACE-LEVEL SCORES PER MODEL

  • Hard-fail rates were 0–5% for LINC, 50–70% for capable judges, 97–100% for smaller or less-aligned judges, and 90–100% for PRMs on natural-language traces.LINC hard-fails only for syntactically ill-formed generated FOL translations, unlike methods that flag reasoning-level errors.Source: §F.1 · p. 28
  • Type-6’s overall mean trace-level score was 1.34, including 2.06 on MMLU; LLM-judges ranged from 2.0–4.2, PRMs from 0.22–4.32, and the trivial baseline was 2.60.LLM-judge rankings across generation models were inconsistent; the baseline score is fixed across splits by construction.Source: §F.1 · p. 28

F.2 PER-MODEL ANALYSIS

The per-model analysis reports hard-fail rates and coherence scores by generation model and split, while cautioning that hard-fail rates do not directly indicate accuracy. It also examines how individual methods’ signals, verdicts, and agreement vary across the corpus.

F.2 PER-MODEL ANALYSIS

  • Trivial’s four signals span the corpus, and its non-trivial correlation with judges may reflect genuinely lower coherence in shorter or more repetitive traces.The signals have a median statement count of 55.5 and a median type-token ratio of 0.376; their ranges are broad despite the baseline’s low hard-fail rate.Source: §F.2 · p. 28, Table 7 · p. 30
  • Hard-fail rates vary across LLM-judge, PRM, and Type-6 methods by generator and split, but do not directly indicate accuracy because ground truth is mostly unknown.LINC’s very low rates follow from rejecting only failed formal checks; Trivial is calibrated to reject about 0%.Source: Table 5 · p. 29

F.2.1 ROSCOE

  • ROSCOE’s hard-fail flag is degenerate because its default threshold does not transfer to natural-language reasoning, although its individual metrics show informative distributions.Chain self-consistency has median 0.056 (IQR [0.042, 0.070]); CSE-step contradiction has median 0.033 (IQR [0.018, 0.063]).Source: §F.2.1 · p. 30, Table 8 · p. 30
  • LINC has predicate grounding of 1.000 across its three models, but only GPT-5.6 produces informative entailment or non-entailment verdicts on this corpus.GPT-5.6 has the lowest coverage ratio, while Qwen-3.5-9B and GLM-4.7 emit unknown or error verdicts on nearly all traces; incomplete translation is associated with reserved verdicts.Source: §F.2.1 · p. 30, Table 9 · p. 31, Table 10 · p. 31
  • PRM hard-fail rates rise from 68% for Llama to 96% for Math-Shepherd and 100% for Qwen, suggesting sharply degraded transfer to natural-language reasoning.Math-Shepherd has a 5.3% low-confidence rate, whereas Llama and Qwen have 0%; when these PRMs lack uncertainty signals, their mislabelling is confident.Source: §F.2.1 · p. 30, Table 12 · p. 32, Table 13 · p. 32
  • LLM-judge flag rates vary substantially by judge: unresolved doubt ranges from 52–87% and modal mismatch from 29–95%, with within-method agreement below κ = 0.4.Judges differ by two to three times in how often they flag the same rubric criterion; the highest pairwise hard-fail agreement is κ = 0.382.Source: §F.2.1 · p. 30, Table 14 · p. 32, Table 15 · p. 32
  • Type-6 elevates soft contradictions and sink residuals to hard failures only when they touch the direct implication chain, limiting false positives from contextual material.The alternative graded-reduction strategies for its trace-level coherence scalar are all strongly rank-correlated.Source: §F.2.1 · p. 31, Table 17 · p. 33, Table 18 · p. 33
Loading 2609.38420v1…