PaperlayerView exact revision on arXiv

Source-linked AI summary

Applying Large Language Models and Chain-of-Thought for Automatic Scoring

Gyeong-Geon Lee, Ehsan Latif, Xuansheng Wu, Ninghao Liu, Xiaoming Zhai

arXiv:2312.03748v2cs.CLcs.AI

TL;DR

Automatic scoring systems still face unresolved questions about effective configurations and content-specific reasoning. This study evaluates prompt-engineered LLM scoring with Chain-of-Thought and finds improved accuracy and interpretability when reasoning is incorporated.

  • Problem

    The effective hyperparameter settings for automatic scoring remain unresolved, while systems have limited capacity to grasp content-specific knowledge and students’ rationale.

  • Method

    The study analyzes extensive student-response datasets using prompt engineering and Chain-of-Thought methods for automatic scoring.

  • Results

    Chain-of-Thought enhances the accuracy and interpretability of automatic scoring systems.

  • Takeaways & Limitations

    The findings support using domain-specific reasoning to improve automatic scoring in science education.

  • Takeaways & Limitations

    LLMs have limited capacity to grasp the depth of content-specific knowledge and the rationale behind students’ responses.

Abstract

from arXiv · show

This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1,650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics. Results indicated that few-shot (acc = .67) outperformed zero-shot learning (acc = .60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = .60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches.

2 School of Computing, University of Georgia, Athens, GA, USA

The section highlights LLMs’ potential for explainable and interpretable automatic scoring, with CoT improving accuracy and transparency when paired with item stems and scoring rubrics.

  • LLMs can facilitate explainable automatic scoring.This potential is presented alongside interpretability and transparency.
  • CoT enhances the accuracy of automatic scoring when used with item stems and scoring rubrics.The section also associates this approach with improved transparency and interpretability.
  • CoT supports transparent and interpretable scoring particularly when combined with contextual item stems and scoring rubrics.

1 Introduction

Automatic scoring is increasingly important in science education because it can provide efficient assessment and immediate feedback, yet existing systems struggle with scientific reasoning, explainability, and content-specific knowledge. This study investigates whether LLMs with chain-of-thought prompting can reduce human scoring effort while improving the accuracy and interpretability of scores under varied prompting, contextual, model, and sampling conditions.

  • Motivation: Automatic scoring can support timely assessment and immediate feedback as students apply scientific knowledge to solve problems.Immediate feedback can help students recognize and correct misunderstandings while using disciplinary core ideas and crosscutting concepts.
  • Limitations: Existing scoring systems can evaluate responses from short answers to essays but often miss scientific reasoning and students’ thinking processes.Developing these models is also described as time- and effort-consuming.
  • Research gap: Prompt-engineering approaches may reduce the need for extensive labeled training cases, but scoring accuracy remains limited by LLMs’ difficulty grasping content-specific knowledge and answer rationales.The introduction also identifies unresolved questions about which GPT models and hyperparameter settings best support automatic scoring.
  • Study purpose: The study proposes combining LLMs with chain-of-thought prompting to enhance science-scoring accuracy, capture students’ construction of scientific explanations, and align scores more closely with human judgments.CoT is characterized as intermediary natural-language reasoning steps culminating in a final output.

2 Literature Review

The literature review traces automatic scoring from labor-intensive supervised NLP and neural models toward LLM-based approaches, while emphasizing persistent accessibility, technical, and explainability challenges. It positions the study within emerging work on practical, comparative, and interpretable use of advanced LLMs for grading science responses.

  • Literature review scope: The review covers pre-LLM automatic scoring models, LLM capabilities, prompt engineering, and methods for improving LLM result accuracy.These strands frame the study’s use of GPT-family models for automatic scoring.
  • Pre-LLM scoring approaches: Earlier automatic scoring relied on supervised text-classification NLP, requiring response collection, expert scoring, algorithm development, training, testing, and iterative improvement.Token- and embedding-based representations fed neural models whose classification layers produced response labels.
  • Pre-LLM scoring approaches: Substantial datasets and trained human experts were foundational because expert assessments benchmarked performance and supplied training and testing data aligned with educational standards.The review states that robust models typically require hundreds to over a thousand diverse responses.
  • Accessibility and technical barriers: Although ensemble scoring improved results, the broader preparation, development, training, and testing process remained a formidable obstacle for researchers without programming or machine-learning expertise.The review argues that simplifying these technical complexities could make AI-enhanced scoring more accessible to educational professionals.
  • Explainability: Prior automatic-scoring research often overlooked explainability, making it difficult to understand how model factors produced predictions and creating a need for trustworthy, transparent educational use.This concern is especially important when scores affect formative assessment and teachers’ instructional decisions.

3 Methods

The study used a human-scored dataset of 1,650 middle-school science responses across six assessment tasks and evaluated six prompt-engineering strategies. Prompts were developed through an iterative review, validation, revision, and testing procedure combining contextual rubrics, examples, and Chain-of-Thought instructions.

  • Dataset and sampling: The dataset comprised 1,650 student-written responses from six tasks: three with binomial rubrics and three with trinomial rubrics.Responses had ground-truth labels assigned by human scorers, and the original items assessed Beginning, Developing, and Proficient levels.
  • Experimental design: The experiments crossed zero-shot versus few-shot learning with prompts without CoT, with CoT, or with CoT plus scoring rubric and problem context.Few-shot prompts used four human-scored examples excluded from the test dataset, whereas zero-shot prompts used no such examples.
  • Prompt-engineering procedure: The proposed WRVRT procedure iteratively writes, reviews, validates, revises, and tests prompts until they reach saturation.Experts from educational assessment, AI, large language models, machine learning, and computer science reviewed the prompts; the prompts were finalized after three revisions.

Appendix 1.

The experiment used Python 3.10 with OpenAI GPT-4 and GPT-3.5-turbo APIs to classify student-written responses. GPT classifications were evaluated against human consent labeling using standard classification metrics across six prompt configurations.

  • Implementation: The experiment was conducted in a Python 3.10 environment using OpenAI GPT-4 and GPT-3.5-turbo APIs.
  • Evaluation: GPT-family classifications of student-written responses were compared with human consent labeling.
  • Evaluation: Accuracy, precision, recall, and F1 were calculated for the GPT classifications.
  • Prompt engineering: The six prompts combined zero-shot or few-shot learning with Chain-of-Thought and, where applicable, problem context and rubrics.The prompt examples identify ZS as zero-shot, FS as few-shot, CoT as Chain-of-Thought, and CR as problem context and rubric.

4 Findings

LLMs produced explainable automatic scores, while few-shot prompting and CoT paired with problem context and rubrics generally improved scoring accuracy. GPT-4 outperformed GPT-3.5 overall, with single-call greedy sampling performing best for GPT-4.

  • Explainable responses: GPT-4 returned reasons for classifying responses as “Proficient,” “Developing,” or “Beginning,” except under FS_noCoT.Zero-shot prompts generated relatively longer explanations, whereas few-shot prompts followed the provided example structure more closely.
  • Prompting strategies: Few-shot learning achieved higher average scoring accuracy (M = .6698; SD = .1744) than zero-shot learning (M = .595; SD = .1205), a 12.6% increase.Without CoT, FS_noCoT reached M = .6604 (SD = .1342), compared with ZS_noCoT at M = .5487 (SD = .1135).
  • Category-wise performance: CoT with context and rubrics balanced accuracy across proficiency categories, including a 129.94% improvement in Quadratic Weighted Kappa for H4_2.For H4_2, Quadratic Weighted Kappa increased from .2525 (ZS_CoT) to .5806 (ZS_CoT_CR).

5 Discussion

The discussion presents prompt-based LLM scoring as accessible and more interpretable than conventional black-box approaches. It emphasizes that domain-specific CoT, especially with contextual instructions and rubrics, improves scoring transparency and that GPT-4 with single-call greedy sampling performs best overall.

  • Interpretability and accessibility: CoT produced explicit and transparent scoring outcomes that allow human users to check and correct machine scoring and identify potential ethics or bias issues.The discussion attributes this possibility to inspecting CoT steps and fixing prompts appropriately.
  • Interpretability and accessibility: Prompt-based scoring is highly accessible because it requires little human-scored data and avoids programming or strenuous model training.The approach is presented as usable by broader populations beyond professional developers and as sharply distinct from complex, unexplained classification models.
  • Prompting strategies: 7.48% higher accuracy than zero-shot learning on average was found for few-shot learning, which the authors associate with less human effort and human-like scoring structure.The discussion suggests that few-shot examples can structure GPT’s CoT reasoning and improve automatic scoring accuracy.
  • Prompting strategies: CoT was effective only when paired with problem context and scoring rubrics; general reasoning alone offered limited improvement, whereas task-specific reasoning was critical for scoring accuracy.The discussion introduces WRVRT as an iterative CoT prompting approach and reports that its overall gains arose from changes in category-wise accuracy.

6 Conclusions

The study supports LLM-based automatic scoring as feasible and interpretable, with few-shot learning and rubric-grounded CoT improving accuracy. GPT-4 with single-call greedy sampling performed best, while limitations motivate further model and prompt-engineering research.

  • Scoring strategies: Few-shot learning improved scoring accuracy by 12.6% over zero-shot learning, reducing the need for extensive human input while maintaining high accuracy.The conclusion identifies few-shot learning as a promising direction for automatic scoring tasks.
  • Scoring strategies: 13.44% increase for zero-shot and 3.7% increase for few-shot resulted when CoT was paired with contextual item stems and rubrics.Rubric-grounded CoT was identified as a significant contributor to scoring accuracy.
  • Scoring strategies: WRVRT facilitated more balanced accuracy across proficiency categories, underscoring the importance of domain-specific reasoning for LLM scoring.The approach addressed performance differences across student proficiency categories.
  • Model and sampling comparisons: 8.64% difference separated GPT-4 from GPT3.5, and GPT-4 using single-call greedy sampling outperformed ensemble voting strategies.The conclusion links this result to GPT-4’s suitability for reliable and accurate automated scoring.
  • Limitations and future research: Future research should improve model capacity, develop sophisticated prompt engineering, examine analytic rubric-component scoring, and investigate item and rubric characteristics.These directions aim to advance scoring performance and better capture nuances in students’ thinking.
  • Limitations and future research: Scoring accuracy spanned .5885-.9083, below the equivalent dataset’s .86-.94 ensemble-machine range, reflecting a trade-off between generic prompting efficiency and task-specific accuracy.The ensemble approach fine-tuned parameters for specific tasks, whereas zero-shot and few-shot approaches used pretrained LLMs more generically.

Declaration of Interest statement · Appendix 1: Comprehensive Example of Prompt Engineering Components · (Task H4_3)

The example prompt instructs an impartial science-teacher evaluation of a middle-school student's response, requiring an explanation and a rubric-based proficiency rating in a fixed format. It also specifies that the BasicRole component is concatenated only for ZS_CoT_CR and FS_CoT_CR.

  • Appendix 1: Comprehensive Example of Prompt Engineering Components: The BasicRole component is concatenated only for ZS_CoT_CR and FS_CoT_CR.
  • (Task H4_3): The evaluator is instructed to act as an impartial science teacher assessing a middle-school student's response.
  • (Task H4_3): The evaluation must begin with a short explanation and remain as objective as possible.
  • (Task H4_3): After explaining, the evaluator must classify the response as Beginning, Developing, or Proficient.
  • (Task H4_3): The rating must follow the exact format "[[rating]]," illustrated by "Rating: [[Beginning]]."
  • (Task H4_3): The evaluator must consult the provided context and rubric when assigning the rating.

ContRubTEXT · Appendix 2: Overall Model Performance Metrics

The ContRubTEXT examples show how rubric components determine Proficient, Developing, or Beginning scores, while Appendix 2 defines the overall performance metrics and GPT-4 evaluation method.

  • ContRubTEXT: The rubric defines Proficient responses as including both the gas-to-liquid state change and decreased particle motion or kinetic energy.Responses meeting both components receive Proficient; responses meeting one or neither receive Developing or Beginning, respectively.
  • ContRubTEXT: The fully correct example is scored Proficient because it identifies gas becoming liquid and decreased kinetic energy.The scoring rationale explicitly maps these statements to COMPONENT A and COMPONENT B.
  • ContRubTEXT: A response mentioning only a gas transition is scored Developing because it satisfies one component but not both.The example identifies “turning into a gas” as COMPONENT A and notes the absence of COMPONENT B.
  • ContRubTEXT: A response mentioning slower molecular motion without the state change is also scored Developing.The rationale treats “moving slower” as COMPONENT B while stating that COMPONENT A is missing.
  • ContRubTEXT: A response describing evaporation without either required component is scored Beginning.The scoring explanation classifies the response as including NONE of the criteria.
  • Appendix 2: Overall Model Performance Metrics: Appendix 2 reports Accuracy, Precision, Recall, F1, KappaQW, and proficiency-specific accuracy metrics for Proficient, Developing, and Beginning categories.The appendix abbreviates these as Acc, KappaQW, Acc_Prof, Acc_Dev, and Acc_Beg.
  • Appendix 2: Overall Model Performance Metrics: The appendix identifies ZS as Zero-Shot, FS as Few-Shot, CoT as Chain-of-Thought, and CR as Problem Context and Rubric.These abbreviations are used to label the evaluated task methods.
  • Appendix 2: Overall Model Performance Metrics: The reported GPT-4 method calls the API once with temperature = 0 and top_p = 0.01.The appendix describes this as the implementation method for the listed performance metrics.
Loading 2312.03748v2…