PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-tuning ChatGPT for Automatic Scoring

Ehsan Latif, Xiaoming Zhai

arXiv:2310.10072v3cs.CLcs.AI

TL;DR

Automatic scoring systems must handle the domain-specific language and varied responses of students, while also addressing fairness and transparency concerns. This study fine-tunes GPT-3.5 on six expert-scored science-assessment tasks and compares it with BERT. Across the tasks, fine-tuned GPT-3.5 outperformed BERT in automatic-scoring accuracy, supporting its potential for domain-specific educational assessment.

  • Problem

    Pre-trained GPT-3.5 is trained on broad online materials, whereas student responses use different language; specialized automatic scoring therefore requires domain-specific adaptation.

  • Method

    The study fine-tunes GPT-3.5 on six assessment tasks using diverse middle- and high-school responses with expert scoring, then compares it with BERT.

  • Results

    GPT-3.5 outperformed BERT in automatic-scoring accuracy across the six science-education tasks.

  • Takeaways & Limitations

    Fine-tuned GPT-3.5 shows potential for accurate automatic scoring of domain-specific student responses in education.

  • Takeaways & Limitations

    Because the study is a secondary analysis of existing responses, future research should test automatic scoring in authentic classrooms and examine student learning outcomes experimentally.

Abstract

from arXiv · show

This study highlights the potential of fine-tuned ChatGPT (GPT-3.5) for automatically scoring student written constructed responses using example assessment tasks in science education. Recent studies on OpenAI's generative model GPT-3.5 proved its superiority in predicting the natural language with high accuracy and human-like responses. GPT-3.5 has been trained over enormous online language materials such as journals and Wikipedia; therefore, more than direct usage of pre-trained GPT-3.5 is required for automatic scoring as students utilize a different language than trained material. These imply that a domain-specific model, fine-tuned over data for specific tasks, can enhance model performance. In this study, we fine-tuned GPT-3.5 on six assessment tasks with a diverse dataset of middle-school and high-school student responses and expert scoring. The six tasks comprise two multi-label and four multi-class assessment tasks. We compare the performance of fine-tuned GPT-3.5 with the fine-tuned state-of-the-art Google's generated language model, BERT. The results show that in-domain training corpora constructed from science questions and responses for BERT achieved average accuracy = 0.838, SD = 0.069. GPT-3.5 shows a remarkable average increase (9.1%) in automatic scoring accuracy (mean = 9.15, SD = 0.042) for the six tasks, p =0.001 < 0.05. Specifically, for multi-label tasks (item 1 with 5 labels; item 2 with 10 labels), GPT-3.5 achieved significantly higher scoring accuracy than BERT across all the labels, with the second item achieving a 7.1% increase. The average scoring increase for the four multi-class items for GPT-3.5 was 10.6% compared to BERT. Our study confirmed the effectiveness of fine-tuned GPT-3.5 for automatic scoring of student responses on domain-specific data in education with high accuracy. We have released fine-tuned models for public use and community engagement.

1 Introduction

Educational assessment faces complex demands, while general-purpose ChatGPT has limitations in specialized domains. This study therefore fine-tunes GPT-3.5 for science-education automatic scoring and compares it with BERT.

  • General-purpose ChatGPT has limitations in specialized educational domains, motivating fine-tuning with domain-specific knowledge and context.
  • The study fine-tunes GPT-3.5 for automatic scoring across six complex science-education assessment tasks.The tasks use student responses and expert scoring.
  • Fine-tuned GPT-3.5 achieved high automatic-scoring accuracy compared with Google’s pre-trained BERT model.The authors present this comparison as evidence of GPT-3.5’s potential for science-education assessment.
  • The evaluation includes evidence from both multi-label and multi-class assessment tasks and compares GPT-3.5-turbo with BERT.
  • The proposed fine-tuning scheme uses a simple, explainable principle to select data subsets for accurate and portable automatic scoring.

2 Automatic Scoring in Education

Automatic scoring supports large-scale educational evaluation and timely feedback, but student-response diversity, fairness, and cross-domain flexibility remain challenges. Fine-tuning language models is presented as a possible way to improve precision and contextual adaptation.

  • Automatic scoring is especially useful in large-scale classrooms where manual grading is impractical.It can provide teachers with timely information and students with customized feedback.
  • Prior automatic-scoring research has used neural networks and natural-language processing to grade lengthy responses.
  • Student-generated content varies widely, creating mismatch between scoring-system expectations and actual responses.
  • Reported challenges include varied scoring accuracy, fairness and bias concerns, and limited flexibility across disciplines and languages.
  • Fine-tuning with domain-specific expertise is proposed as a way to make automatic scoring more precise, adaptive, and context-aware.

3 Large Language Models for Automatic Scoring

Large language models are increasingly used for automatic scoring, including in specialized mathematics and science applications. Their educational use remains constrained by bias, transparency, domain-nuance, and model-architecture concerns, while fine-tuning is presented as a potential remedy.

  • LLMs attract attention for automatic scoring because of their extensive knowledge, contextual awareness, and flexibility.
  • LLMs have been applied to divergent-thinking, emotional-content, essay, mathematics, science, and translation assessment tasks.
  • Educational LLM use raises concerns about bias, fairness, domain-specific nuance, interpretability, and grading transparency.
  • BERT is primarily designed for fixed-length inputs, making sequence generation and open-ended tasks more difficult.
  • Fine-tuning ChatGPT on domain-specific corpora is presented as a way to produce more contextually relevant outputs for response evaluation and open-ended tasks.
  • The study restricts its findings to GPT-3.5 because Gemini’s public APIs did not yet provide model fine-tuning.

4 Methodology

The methodology conducts a secondary analysis of existing, expert-scored science-education response datasets. It covers two multi-label tasks from PASTA and four multi-class tasks from MTS, with middle- and high-school students.

  • The study analyzes existing datasets containing student responses paired with expert scoring.
  • The dataset includes two multi-label PASTA tasks answered by middle-school students.
  • It also includes four multi-class Mathematical Thinking in Science tasks answered by high-school students.
  • Table 1 presents the data division for each assessment item.

4.2 Assessment Task

The study evaluates NGSS-aligned science tasks spanning multi-label and multi-class assessment, using student responses and expert-developed scoring rubrics. The tasks target scientific knowledge use, mathematical reasoning, and multiple dimensions of science learning.

  • The assessment tasks align with middle-school NGSS expectations integrating disciplinary core ideas, crosscutting concepts, and science and engineering practices.
  • Multi-label assessment tasks: Two multi-label tasks assess students’ explanations of scientific phenomena across multiple knowledge dimensions.The tasks include Gas-filled balloons and Layers in test tubes.
  • Multi-label assessment tasks: Students receive simultaneous scores for understanding disciplinary core ideas, crosscutting concepts, and science and engineering practices under a validated multi-perspective rubric.The rubric contains five response perspectives and was further validated with science teachers.
  • Multi-class assessment tasks: The multi-class tasks require scientific knowledge and mathematical reasoning and target high-school NGSS ideas about energy and ecosystems.Approximately 1,400 students in grades 9–12 participated, with grading by eight teaching experts.
  • Multi-class assessment tasks: The falling-weights task asks students to explain whether water warms when a falling weight drives a paddle wheel that stirs it.A four-level rubric assesses students’ knowledge-use proficiency in their responses.

4.3 Machine Algorithms

The machine-algorithm section presents BERT as the comparison baseline and GPT-3.5-turbo as a flexible model adapted for domain-specific automatic scoring. It emphasizes fine-tuning to align scoring with educational criteria and broaden performance across task types.

  • BERT: BERT uses bidirectional self-attention to capture contextual relationships in student responses and serves as the automatic-scoring baseline.Its contextual understanding is described as useful for evaluating linked response components.
  • GPT-3.5: GPT-3.5 is a transformer-based large language model with contextual representation and few-shot capabilities across tasks.The text describes applications including translation, summarization, question answering, and programming.
  • GPT-3.5: Fine-tuning changes model weights on scoring data so GPT-3.5-turbo can capture task-specific criteria and better align with human-rater expectations.
  • GPT-3.5: GPT-3.5-turbo was selected for fine-tuning because API availability balances performance, efficiency, and affordability.
  • GPT-3.5: Text augmentation can expand and diversify scoring data, and broader response examples have been reported to improve automated evaluation of student text.

5 Experimental Setup

The experimental setup trains language models on domain-specific science assessment data containing varied student responses and human scores. It includes data processing, fine-tuning, held-out evaluation, and baseline comparison procedures.

  • The training scheme optimizes GPT-3.5-turbo on academic data to investigate how domain-specific context affects model performance.
  • The dataset should cover varied themes, question types, and complexity levels, with human-assigned scores providing supervision.
  • Data processing removes irrelevant or identifying information, tokenizes responses, and prepares JSON files for upload through the OpenAI file API.
  • Fine-tuning begins from the pretrained GPT-3.5-turbo model, whose existing language knowledge is described as speeding convergence.
  • Training choices include task-appropriate loss functions, small learning rates, multiple epochs, batch-size selection, and validation monitoring to limit overfitting.
  • Performance is assessed on a separate evaluation set using metrics such as MAE or classification accuracy and comparisons with a baseline model.

6 Results

Fine-tuned GPT-3.5-turbo significantly outperformed BERT overall, with advantages in both multi-label and multi-class science assessment tasks. Gains were especially pronounced for unbalanced multi-label labels and across the multi-class items.

  • GPT-3.5-turbo exceeded BERT by 9.1% in average scoring accuracy across the six tasks.The paired comparison found Mean Difference = 0.076, SD = 0.043; t(5) = 4.44, p = 0.007 < 0.05.
  • Multi-label Tasks: GPT-3.5-turbo matched BERT at 0.97 accuracy for Gas-Filled Balloons, while outperforming it by 0.12 for Layers in Test Tubes.No significant difference was observed for Gas-Filled Balloons; the Layers in Test Tubes difference was significant.
  • Multi-label Tasks: GPT-3.5-turbo showed higher individual-label accuracy than BERT on both multi-label tasks, including a 7.2% advantage for certain unbalanced labels.The first task had Mean Difference = 0.008, SD = 0.008; the second had Mean Difference = 0.065, SD = 0.044.
  • Multi-class Tasks: GPT-3.5-turbo outperformed BERT on Falling Weights, Gelatin, Bathtub, and Sandwater1, with accuracies ranging from 0.87 to 0.92.BERT’s corresponding accuracies ranged from 0.77 to 0.82.

7 Discussion

The discussion interprets fine-tuning as a way to adapt pre-trained models to the complexities of educational scoring. The reported results support GPT-3.5-turbo’s potential for accurate, context-sensitive assessment, including work with unbalanced categories.

  • Instruction-based fine-tuning improved automatic scoring for both multi-label and multi-class educational tasks.The discussion attributes this improvement to domain-specific adaptation that captures complexities unique to educational datasets.
  • GPT-3.5-turbo’s improved scoring of student written responses demonstrates potential for educational applications.The discussion links the performance gains to the model’s architecture and training strategy tailored to educational data.
  • Fine-tuned GPT-3.5-turbo is presented as a tool for automating educational scoring while retaining accuracy and contextual knowledge.The paper describes the fine-tuned version as created specifically for educational data.
  • GPT-3.5-turbo showed promise for scoring minority categories in unbalanced data, addressing a concern in machine training and fine-tuning.The discussion specifically identifies an advantage in minority scoring categories for the Gas-filled balloons task.
  • The findings reinforce the importance of advanced, domain-specific models for educational automatic scoring and motivate further research.The paper characterizes the results as promising while calling for continued exploration of refined educational applications.

8 Conclusions and Limitations

Fine-tuned GPT-3.5 outperformed BERT for domain-specific automatic scoring across varied science-education tasks. The study nevertheless identifies boundaries involving data setting, model coverage, ethics, teacher roles, bias, privacy, and security.

  • Conclusions: Fine-tuned GPT-3.5 outperformed BERT for automatic scoring across multi-label and multi-class science-education tasks.The approach used varied responses from middle-school and high-school students and is presented as scalable and replicable for science tasks.
  • Limitations: The secondary analysis was constrained by the existing assessment process and limited student and teacher information.Future work should test automatic scoring in authentic classrooms and examine student learning outcomes experimentally.
  • Limitations: The comparison covered only BERT and GPT-3.5 despite the rapid emergence of other large language models.The authors recommend comparing GPT-3.5 Turbo with Gemini and similar models when suitable APIs become available.
  • Limitations: Fine-tuned ChatGPT assessment raises unresolved questions about scoring fairness and transparency.The authors call for further study of these ethical issues before broader classroom implementation.
  • Limitations: AI-based scoring may shift teachers toward personalized guidance while reducing critical human engagement and personal interaction.The paper also calls for scrutiny of bias, privacy, and data security in educational use.

Declaration of Human Subject Approval

The researchers secondarily analyzed existing de-identified data that had undergone IRB review. Because the data could not be linked to individual subjects, the study was waived from further IRB review.

  • Approval: U.S. regulations define human subject research through interaction with individuals or identifiable private information.The passage states that completely de-identified, unlinkable data may fall outside this definition.
  • Approval: The study used existing de-identified data that researchers could not link to individual subjects.The data had undergone IRB review.
  • Approval: The research was not regarded as human subject research and was waived from further IRB review.
Loading 2310.10072v3…