PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability, validity, and diagnostic evidence for multi-model LLM short-answer scoring

Chunyi Zhao, Chao Li

arXiv:2609.06315v1cs.CLcs.CY

TL;DR

Short-answer scoring needs evidence beyond single-model, single-run accuracy to support assessment use. This study evaluates structured OCG-PRES guided LLM scoring, with F1-optimal thresholds of 3.0 for GPT and DeepSeek and 2.5 for Qianwen and the ensemble. It cautions that the evidence does not establish full validation or replacement of human judgment.

  • Problem

    Single-model, single-run accuracy-focused evaluations provide limited evidence for assessment use, while models may differ in scoring severity or error profile.

    Source: §1 · p. 2

  • Method

    The study evaluates LLM scoring guided by OCG-PRES, a five-dimension framework for short-answer quality.

    Source: §1 · p. 3

  • Results

    Best F1 threshold was 3.0 for GPT and DeepSeek, and 2.5 for Qianwen and the ensemble.

    Source: Threshold sensitivity summary by best… · p. 15

  • Takeaways & Limitations

    Threshold-optimized performance and fixed-threshold performance answer different evaluation questions.

    Source: Threshold sensitivity summary by best… · p. 15

    The analyses do not establish full validation or imply that automated scoring can replace human judgment.

    Source: §2.4 · p. 6

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used or proposed for educational scoring, but single-model and single-run evaluations provide limited evidence for assessment use. Short-answer scoring requires evidence about reliability, validity, severity, diagnostic value, and failure cases. This study evaluated repeated multi-model OCG-PRES guided LLM scoring for short-answer assessment. The analysis used 996 SciEntsBank responses. GPT, DeepSeek, and Qianwen each scored every response across three independent runs using five OCG-PRES dimensions: concept coverage, relation accuracy, reasoning completeness, contradiction control, and domain relevance. Scores were evaluated against official binary and five-category labels and compared with non-LLM baselines based on answer length, Jaccard keyword overlap, TF-IDF cosine similarity, and a combined traditional logistic model. Repeated-run reliability was high for all models, with ICC(3,k) = .977 for GPT, .992 for DeepSeek, and .981 for Qianwen. DeepSeek was the most stable across runs. GPT showed the strongest official-label alignment by AUC (.909), while Qianwen was stricter, with higher precision but lower recall under the fixed threshold = 3.0 rule. OCG-PRES scores followed expected diagnostic patterns across five official categories and outperformed all non-LLM baselines in AUC and F1. Repeated multi-model OCG-PRES scoring provides reliability, validity, and diagnostic evidence for LLM-assisted short-answer scoring. The findings support cautious, evidence-based use as a scoring support tool rather than a replacement for human judgement.

1. Introduction

Short-answer scoring must distinguish meaningful scientific understanding from lexical overlap, but single-model, single-run accuracy evaluations provide incomplete evidence for assessment use. This study evaluates OCG-PRES guided LLM scoring through reliability, validity, diagnostic, and baseline comparisons.

1. Introduction

  • Surface similarity can mis-score responses that share words but express incorrect relations, or that convey the intended meaning with little lexical overlap.The assessment question is not only score accuracy, but whether scores provide defensible evidence about response quality.Source: §1 · p. 2
  • Single-model, single-run, accuracy-focused evaluations leave assessment evidence about reliability, validity, consistency, severity, diagnostic value, and failure cases incomplete.Repeated runs matter because outputs may vary, and multiple models may rank responses similarly while differing in severity or error profile.Source: §1 · p. 2
  • OCG-PRES structures short-answer scoring across concept coverage, relation accuracy, reasoning completeness, contradiction control, and domain relevance.The framework makes the basis for a score more explicit by separating these forms of response evidence.Source: §1 · p. 3
  • The study evaluates GPT, DeepSeek, and Qianwen across three runs on 996 SciEntsBank responses against official labels and traditional non-LLM baselines.Its four questions address repeated-run stability, inter-model agreement and severity, label alignment, and comparison with non-LLM textual baselines.Source: §1 · p. 3

2. Literature Review / Background

Traditional text-based scoring offers transparent baselines but may miss scientific relations and reasoning; LLM assessment therefore needs evidence beyond a single performance score. OCG-PRES frames scoring as a structured, multidimensional measurement task, while the review emphasizes validation and human-judgment boundaries.

2.1 Automated Short-Answer Scoring and Traditional Textual Baselines

  • Lexical overlap, term weighting, and supervised methods offer transparent, reproducible baselines, but surface-text measures do not directly evaluate scientific relations or reasoning.Responses may overlap with a reference while reversing a causal relation, omitting a mechanism, or contradicting it; correct answers may use different wording.Source: §2.1 · pp. 3–4

2.2 LLMs in Educational Assessment

  • LLM scoring can provide structured scores or rationales, but assessment use requires attention to consistency, opacity, prompt sensitivity, model updates, bias, and over-reliance.A high performance value on one dataset does not establish reliability or validity for assessment use.Source: §2.2 · p. 4

2.3 Reliability and Validity Evidence in Automated Scoring

  • Reliability evidence addresses consistency across relevant variation, while validity evidence asks whether scores support intended interpretations and uses.Here, binary labels provide a correctness-alignment reference and five-category labels provide a diagnostic reference; these patterns are complementary, not definitive validation.Source: §2.3 · p. 4

2.4 OCG-PRES Guided Scoring

  • OCG-PRES operationalizes knowledge-system principles as an assessment-oriented framework that separates five forms of evidence in short answers.The framework is not proposed as a new reasoning machine or general theory, and the analyses do not establish full validation or replacement of human judgment.Source: §2.4 · pp. 5–6

3. Methods

The methods use official binary and five-category labels to evaluate repeated OCG-PRES scoring of SciEntsBank responses. Three LLMs score each response across runs, with text-based baselines and multiple analyses used to examine reliability, alignment, severity, and diagnostic patterns.

3.1 Dataset and Official Labels

  • The study analyzes 996 SciEntsBank responses using official binary labels and five categories: correct, partially correct/incomplete, contradictory, irrelevant, and non-domain.Binary labels support classification and ROC/AUC analyses, while five-category labels support diagnostic pattern analysis.Source: §3.1 · p. 6

3.2 OCG-PRES Guided Scoring Framework

  • OCG-PRES scores concept coverage, relation accuracy, reasoning completeness, contradiction control, and domain relevance as task-specific indicators for educational assessment.Each dimension is scored from 0 to 4; the mean is the primary response-level score, and the five-dimension sum ranges from 0 to 20.Source: §3.2 · pp. 6–7

3.3 Multi-Model Repeated Scoring Procedure

  • GPT, DeepSeek, and Qianwen each scored all 996 responses in three independent runs, yielding 8,964 scoring records.Response-level scores were summarized as each model's three-run mean, and an ensemble score averaged the nine model-run mean scores.Source: §3.3 · p. 7

3.4 Non-LLM Baseline Construction

  • Four non-LLM baselines used answer length, Jaccard overlap, TF-IDF cosine similarity, and logistic regression combining text-derived features.They were included to test whether OCG-PRES scoring provided evidence beyond answer length and surface lexical similarity.Source: §3.4 · pp. 7–8

3.5 Data Preparation and Quality Checks

  • Data checks found no duplicated or missing response-model-run records, missing labels or scores, or out-of-range OCG-PRES scores.The checks covered all expected response-model-run combinations and official labels.Source: §3.5 · p. 8

3.6 Analytical Strategy

  • Analyses assessed repeated-run ICC and agreement, inter-model agreement and severity, label alignment, category-level score patterns, and baseline comparisons.Baseline comparisons used within-analysis optimal thresholds, reported separately from fixed-threshold classification at mean_score >= 3.0.Source: §3.6 · p. 8

4. Results

The dataset passed the reported completeness and range checks, and all three models showed high repeated-run reliability. DeepSeek had the highest averaged-score ICC and the strongest item-level agreement indicators among the reported models.

4.1 Data Quality and Descriptive Overview

  • The final long-format dataset contained 8,964 scoring records from 996 distinct responses, matching the expected design of 996 responses scored by three models across three repeated runs.Checks found no duplicated response-model-run rows, missing response-model-run combinations, missing official labels or binary labels, missing OCG-PRES scores, or out-of-range OCG-PRES scores.Source: §4.1 · p. 9

4.2 Repeated-Run Stability Within Each Model

  • .722 average exact agreement and .099 mean absolute difference for DeepSeek were the strongest item-level indicators among the models.GPT's average exact agreement was .361 and mean absolute difference .242; Qianwen's were .410 and .211.Source: §4.2 · p. 9
  • ICC(3,k) was .977 for GPT, .992 for DeepSeek, and .981 for Qianwen; all three models showed high repeated-run reliability.Table 1 reports reliability results; the corresponding single-run ICC(3,1) values were .935, .977, and .945.Source: §4.2 · p. 9, Table 1

Repeated-run reliability of OCG-PRES scores

The supplied passages describe the reliability summary for averaged repeated runs and show high agreement among model-level three-run mean scores, alongside significant differences in mean scoring severity.

Repeated-run reliability of OCG-PRES scores

  • ICC(3,k) summarizes the reliability of averaged repeated runs, and the table also lists exact agreement and mean absolute difference.The supplied passages do not give the reliability values.Source: Repeated-run reliability of OCG-PRES… · p. 9

4.3 Inter-Model Agreement and Scoring Severity

  • Inter-model ICC(3,k) was .965, and pairwise Pearson correlations among model-level three-run mean scores ranged from .898 to .914.The correlations were .914 for GPT–DeepSeek, .900 for GPT–Qianwen, and .898 for DeepSeek–Qianwen.Source: §4.3 · p. 10
  • GPT scored higher than DeepSeek by .076 points, higher than Qianwen by .175 points, and DeepSeek scored higher than Qianwen by .098 points; all comparisons had p < .001.Mean scores were GPT 2.160, DeepSeek 2.084, and Qianwen 1.986.Source: §4.3 · p. 10

Inter-model agreement and scoring severity

The models’ three-run mean scores were highly correlated but differed in severity; they aligned with official binary labels, and their scores varied across the five official categories.

Official binary alignment and classification performance

  • At the fixed mean_score >= 3.0 threshold, GPT had precision and recall of .692 each, while Qianwen had precision = .740 and recall = .500.GPT’s F1 was .692; Qianwen’s was .597.Source: §4.4 · p. 11

4.4 Alignment With Official Binary Labels

  • GPT’s AUC was .909, DeepSeek’s was .900, Qianwen’s was .872, and the ensemble’s was .903 against official binary labels.The corresponding Pearson correlations were .626, .596, .556, and .612, respectively.Source: §4.4 · p. 11, Table 3

4.5 Diagnostic Patterns Across Official Five-Category Labels

  • Non-domain responses had the lowest reported means, including Qianwen = .471 and ensemble = .719.GPT’s mean was .864 and DeepSeek’s was .823.Source: §4.5 · p. 12
  • Correct responses received the highest mean scores across models, including GPT = 3.178 and ensemble = 3.078; partially correct incomplete responses scored lower.The partially correct incomplete means were GPT = 2.533 and ensemble = 2.480.Source: §4.5 · p. 11, Table 4

Mean OCG-PRES scores across official five-category labels

OCG-PRES scores differentiated official response categories and outperformed non-LLM textual baselines in AUC and F1; fixed-threshold analyses also revealed model-specific error patterns.

Mean OCG-PRES scores across official five-category labels

  • Non-domain responses had means of .864 for GPT, .823 for DeepSeek, .471 for Qianwen, and .719 for the ensemble.The five-category results report scores as model-level three-run means on the 0–4 scale.Source: Mean OCG-PRES scores across official… · p. 12

4.6 Comparison With Non-LLM Textual Baselines

  • All OCG-PRES methods exceeded the non-LLM baselines in AUC and F1; OCG-PRES GPT had AUC = .909 versus .756 for the combined traditional baseline.This comparison uses the optimal-threshold baseline comparison, not the fixed threshold = 3.0 results.Source: §4.6 · p. 12, Table 5

4.7 Disagreement and Failure Cases

  • Across models, 89 cases were three-model consistent errors and 155 showed three-model disagreement.The top 5% and top 10% model-disagreement sets contained 50 and 100 cases, respectively.Source: §4.7 · p. 13
  • Under the fixed mean_score >= 3.0 threshold, GPT produced 77 false positives and 77 false negatives; DeepSeek produced 86 false positives and 73 false negatives.These are fixed-threshold counts and should not be conflated with the optimal-threshold baseline comparison.Source: §4.7 · p. 13, Table 6

Disagreement and failure case summary

The results report model-specific error counts, describe supplementary analyses incorporated into the Results, and identify relation accuracy as the strongest dimension-level correlate of official binary labels.

Disagreement and failure case summary

  • Qianwen produced 44 false positives and 125 false negatives, whereas the ensemble produced 53 false positives and 103 false negatives.At the fixed threshold, Qianwen had substantially more false negatives than false positives.Source: Disagreement and failure case summary · p. 14

4.8 Supplementary Results Previously Planned as Appendices

  • Appendix-planned analyses were incorporated into the Results as compact tables, with remaining supplementary result files and reproducibility materials documented.The described tables cover dimension-level correlations and threshold sensitivity.Source: §4.8 · p. 14

Dimension-level correlations with official binary labels

  • Relation accuracy had the strongest Pearson association with official_binary for GPT (r = .651), DeepSeek (r = .605), and Qianwen (r = .590).Concept coverage and reasoning completeness also had moderate to strong associations, while contradiction control and domain relevance varied more across models.Source: Dimension-level correlations with… · p. 15

Threshold sensitivity summary by best F1 score

Across repeated scoring, OCG-PRES showed high reliability, distinct model severity and label-alignment patterns, and diagnostic score ordering across official categories. At optimal thresholds, its methods exceeded non-LLM baselines in AUC and F1, while the study’s implications remain bounded by its data and evaluation scope.

Threshold sensitivity summary by best F1 score

Supplementary results and reproducibility materials incorporated into the Results section

  • Supplementary files cover threshold sensitivity, total-score and dimension logistic models, non-LLM baselines, and false-positive cases; check case-level files against dataset-use permissions before verbatim use.Source: Supplementary results and… · pp. 16–17

5.1 Repeated-Run Scoring as Reliability Evidence

  • DeepSeek was the most stable model (ICC(3,k) = .992; average exact agreement = .722), although agreement and MAD results showed more item-level variation that matters for individual cases.Source: §5.1 · p. 17

5.2 Stability, Severity, and Official-Label Alignment as Distinct Properties

  • DeepSeek was most stable, GPT led official_binary correlation and AUC, and Qianwen had higher precision but lower recall at threshold 3.0.Mean scores were highest for GPT, lowest for Qianwen, and intermediate for DeepSeek, despite strong agreement in response ranking.Source: §5.2 · p. 17

5.3 Diagnostic Value of OCG-PRES

  • Across models, correct responses scored highest, followed by partially correct incomplete, irrelevant or contradictory, and non-domain responses.DeepSeek assigned especially low scores to contradictory responses, while Qianwen scored non-domain responses especially low.Source: §5.3 · p. 18

5.4 Evidence Beyond Surface Text Similarity

  • All OCG-PRES methods exceeded non-LLM baselines in AUC and F1 at optimal thresholds.The combined traditional baseline was the strongest non-LLM method.Source: §5.4 · p. 18

5.5 Implications for LLM-Assisted Educational Assessment

  • The findings support cautious LLM-assisted scoring for feedback, review, or triage rather than final high-stakes decisions or replacement of teacher judgement.High-disagreement cases, consistent model errors, and borderline classifications are identified as signals for human review.Source: §5.5 · p. 19

5.6 Limitations and Future Research

  • Evidence is limited by absent independent teacher ratings and holistic LLM comparison, one science short-answer dataset, and reference labels that may miss scoring nuance.The authors call for validation across more datasets and domains; results may also vary across subjects, languages, age groups, item formats, and assessment contexts.Source: §5.6 · p. 19

6. Conclusion

The study reports reliability, official-label alignment, diagnostic differentiation, and stronger performance than traditional textual baselines for repeated multi-model OCG-PRES scoring. The authors support further research while framing these systems as support for, not replacements for, human judgement.

6. Conclusion

  • The results provide evidence of high repeated-run reliability, inter-model agreement, systematic severity differences, and diagnostic differentiation across official five-category labels.Source: §6 · p. 19
  • DeepSeek was most stable across repeated runs, GPT showed the strongest official binary alignment, and Qianwen was the most severe.Source: §6 · p. 19
  • OCG-PRES guided scoring outperformed traditional non-LLM baselines based on answer length, Jaccard similarity, TF-IDF cosine similarity, and a combined logistic model.Source: §6 · p. 20
  • The findings support further research on LLM-assisted diagnostic scoring, with these systems supporting rather than replacing human judgement.Source: §6 · p. 20

7. Limitations and Future Research

The study identifies limits in its ratings, dataset scope, and reference labels, and recommends broader comparisons and evaluations. It also discloses AI-tool use and conditions on access to analysis materials and the original dataset.

7. Limitations and Future Research

  • The study lacked independent human teacher ratings and a direct holistic LLM baseline, used one science short-answer dataset, and relied on official labels as reference labels.Source: §7 · p. 20
  • Future work should add teacher scoring, compare OCG-PRES with direct holistic LLM scoring, evaluate additional datasets and domains, and examine human-Source: §7 · p. 20

8. AI-Use Disclosure Statement

  • AI-assisted tools supported code organisation, manuscript outlining, and language editing; the authors checked and verified analyses, numerical results, citations, and interpretations.Source: §8 · p. 20

9. Data and Code Availability Statement

  • Analysis scripts and derived result tables are available from the author upon reasonable request, while original-dataset use and redistribution are subject to provider terms.Source: §9 · p. 20
Loading 2609.06315v1…