Source-linked AI summary
Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue
Clayton Cohn, Joyce Fonteles, Kirk Vanacore, Gianni Mazza, Candida Crawford, Tom Hooper, Gautam Biswas, Rene Kizilcec
TL;DR
Fine-grained failure-mode diagnosis from tutoring dialogue is difficult, and whether agreement across LLM families validates these interpretations remains underexplored. This study compares three models’ classifications with an adjudicated human reference set; models agreed more with one another than with the human consensus. The findings caution that consensus can mask shared misinterpretation.
Problem
Whether cross-model agreement validates fine-grained diagnosis from authentic tutoring dialogue remains underexplored, although such diagnosis can distinguish the reasons for incorrect responses and inform different instructional responses.
Source: §1.1 · p. 2, §1.3 · p. 2Method
The study compared three LLM families’ classifications of five failure modes in tutoring dialogue with an adjudicated human reference set.
Source: §2.2 · p. 3, §2.3 · p. 4Results
LLM–LLM agreement was κ = .755–.781 and α = .769, while agreement with human consensus was κ = .524–.597.
Source: §3.2 · pp. 5–6Takeaways & Limitations
Cross-model consensus can mask shared systematic errors and should not substitute for human validation of LLM labels.
Source: §3.2 · p. 6, §4 · p. 7The exploratory evaluation used 100 deliberately sampled error moments from one K–12 mathematics tutoring platform, limiting population-level and cross-domain generalizability.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
In K-12 mathematics tutoring, student-tutor dialogue provides rich evidence of learners' problem-solving processes and sources of difficulty. Learning analytics research increasingly relies on large language models (LLMs) to extract such information from dialogue for a variety of downstream tasks, including knowledge tracing, behavioral modeling, and diagnosis of student reasoning errors. However, the validity of these model-generated interpretations remains insufficiently understood. In this exploratory study, we examine the validity of LLM classifications of five student failure modes in mathematics tutoring dialogue using an operational diagnostic codebook: uncertainty, misattribution, operator selection, conceptual gap, and procedural slip. Across models, human-LLM agreement was moderate (kappa = .524-.597), while cross-model agreement was substantially higher (kappa = .755-.781; alpha = .769). These findings show that cross-model agreement can create a misleading appearance of correctness, challenging the assumption that consensus among LLMs constitutes evidence of valid learner interpretation. For learning analytics, the implication is clear: scalable labeling is useful only if the inferred constructs are valid, and model consensus cannot substitute for independent evidence of that validity.
1. Introduction and Related Work
Student–tutor dialogue can reveal both whether learners struggle and the nature of that struggle, but fine-grained diagnosis from authentic speech remains difficult. This study tests whether agreement across LLMs supports valid interpretation of student failure modes.
1.1 Theoretical Grounding and Student Discourse
- Different failure modes can call for different instructional responses, but manual discourse labeling is resource-intensive and limits analysis at scale.Student–tutor dialogue is presented as evidence of learners’ understanding and difficulty.Source: §1.1 · pp. 1–2
1.2 LLM Labeling in Learning Analytics
- LLMs have been used for discourse coding, knowledge-tracing labels, and qualitative coding, while evidence that they can replace human coders for complex constructs remains mixed (Garg et al., 2024; Scarlatos, Baker, & Lan, 2025; X. Liu et al., 2025; Vanacore and Kizilcec, 2026).Source: §1.2 · p. 2
1.3 Detecting Student Failure Modes from Discourse
- Specific reasoning errors behind incorrect responses remain underexplored; inferring them from authentic, often disfluent dialogue is difficult and may require reasoning not directly expressed (Suraworachet et al., 2024; Booth et al., 2024; Z. Liu et al., 2026; Snyder et al., 2024; Zhang et al., 2026; G¨ures et al., 2026).The unresolved question is whether cross-family agreement reflects valid interpretation or shared error.Source: §1.3 · p. 2
1.4 Present Study and Contributions
- Across three model families, LLMs agreed strongly with one another (κ = .755–.781) but less with human consensus (κ = .524–.597), motivating validation beyond model consensus.The study examines five failure modes and identifies systematic errors, foregrounding construct validity in scalable annotation.Source: §1.4 · pp. 2–3
2. Methods
The study evaluates LLM diagnosis of student failure modes in naturally occurring K–12 math tutoring dialogue against an adjudicated human reference set, examining human alignment, cross-model consistency, and error patterns.
2. Methods
- The analysis examines alignment with human judgments, consistency across models, and error patterns; practitioner reactions serve as an informal validity check, not a formal finding.Source: §2 · p. 3, §2.2 · p. 4
2.1 Tutoring Platform and Failure-Mode Taxonomy
- The platform data comprise naturally occurring tutoring dialogue, and the study applies five operational failure-mode classes plus Not Applicable when dialogue lacks sufficient information.The study concerns classification validity, not downstream learning outcomes.Source: §2.1 · p. 3
2.2 Data Processing and Validation
- From 5,000 sampled transcripts, screening retained 3,854 localized items; 100 challenging cases formed an adjudicated human reference set after independent annotation and disagreement resolution.Codebook and prompt refinement used 330 separate items and disjoint holdouts; the 100-item set intentionally oversampled difficult classes, and the two human raters initially agreed at κ = .588. Interviews with three practitioners contextualized findings.Source: §2.2 · pp. 3–4
2.3 LLM Classification
- Three vendors’ tier-2 models classified each item using the same codebook, prompt, and few-shot examples; a Gemini 3.6 Flash maximum-reasoning run served as a robustness check.Models received the localized dialogue and task context, and each model’s item label used a three-run self-consistency majority vote.Source: §2.3 · p. 4
2.4 Evaluation Metrics
- Pairwise agreement used Cohen’s κ and three-model agreement used Krippendorff’s α; NA remained a regular label, with class metrics, McNemar’s tests, and bootstrap intervals also assessed.The κ = .70 interpretive benchmark was pre-specified, not treated as a universal validity threshold; bootstrap intervals used 1,000 item-level resamples.Source: §2.4 · pp. 4–5
3. Findings
LLMs agreed more strongly with one another than with the human reference, and majority-vote consensus did not improve agreement with that reference. Their errors shared a directional bias toward Conceptual Gap, including when the evidence did not support a diagnosis.
3.1 Human–LLM Agreement
- Human–LLM agreement ranged from κ = .524 to .597 across primary models, below the prespecified .70 benchmark; the higher-capability robustness condition remained at κ = .594.Confidence intervals substantially overlapped across models and conditions.Source: §3.1 · p. 5
3.2 LLM–LLM Agreement
- LLMs agreed more strongly with one another (κ = .755–.781; α = .769) than with humans (κ = .524–.597), while majority-vote agreement (κ = .568) did not exceed the best individual model (κ = .597).The authors caution that cross-model consensus can coexist with systematic disagreement about the inferred learner construct.Source: §3.2 · pp. 5–6
3.3 LLM Error Patterns
- For every non-CON reference class, the most common pooled tier-2 error was misclassification as Conceptual Gap, indicating shared directional bias.The row-normalized matrix also shows cross-class leakage: 10% of true NA items were labeled MIS and another 10% OP.Source: Figure 3 · p. 6, §3.3 · p. 6, Figure 2 · p. 6
- Among true NA items, 35% were classified as CON; models assigned NA to 9–10% of items versus 21% in the human reference, with significant differences (p ≤ .0098).All three models also converged on Conceptual Gap in an example where the transcript did not support a failure-mode diagnosis.Source: §3.3 · pp. 6–7
- Together, the patterns indicate that models over-attribute difficulty to conceptual deficits rather than making errors that average out as random noise.The authors note that this shared bias can systematically distort learning-analytics representations of learner difficulty.Source: §3.3 · p. 7
4. Discussion and Conclusions
Across three LLM families, models agreed more with one another than with human consensus, while sharing systematic directional errors. The discussion argues that learner diagnoses require independent validity evidence and notes limits in the data and diagnostic framework.
4. Discussion and Conclusions
- Cross-model agreement exceeded agreement with human consensus while models shared systematic directional errors, so consensus did not establish valid learner interpretation.For learning analytics, agreement can make annotation appear reliable while masking shared misinterpretation.Source: §4 · p. 7
4.1 Validity of Learner Representations in Learning Analytics
- Agreement among human or model annotators does not establish valid learner representations, and reliability should inform construct refinement rather than mechanically determine ground truth.The discussion relates this measurement problem to Thomas et al. (2026) and findings by Henkel, Vanacore, and Roberts (2025).Source: §4.1 · p. 7
4.2 Instructional Consequences of Diagnostic Misclassification
- Misclassifying a Procedural Slip as a Conceptual Gap can prompt feedback for deficient understanding, while the reverse can leave a misconception insufficiently addressed.Tutors also warned that misdirected correction may harm students’ self-perceptions or waste instructional time.Source: §4.2 · p. 7
4.3 Rethinking Traditional Discourse Annotation
- Human disagreement suggests failure modes can be difficult to infer from transcripts, and mutually exclusive labels may impose sharper boundaries than the evidence supports.Multi-label and uncertainty-aware annotation may preserve cases with several plausible interpretations.Source: §4.3 · p. 7
- Transcript-only analysis omits prosody, written work, screen state, gestures, and other interaction traces, so some disagreement may reflect missing evidence.The discussion suggests multimodal evidence may benefit valid failure-mode diagnosis.Source: §4.3 · p. 7
4.4 Limitations and Future Work
- The study’s 100 deliberately sampled error moments come from one K–12 mathematics tutoring platform, and its three tutors were not a representative stakeholder sample.The study did not collect student perspectives or evaluate downstream effects; future work should examine larger reference sets, multimodal evidence, broader taxonomies, and generalizability.Source: §4.4 · pp. 7–8
- Learning analytics should establish that LLM diagnoses validly represent learner difficulty before asking whether they improve learning.When models share biases, consensus can increase confidence without increasing correctness.Source: §4.4 · p. 8