Source-linked AI summary
Better by you, better than me, chatgpt3 as writing assistance in students essays
Zeljana Basic, Ana Banovac, Ivana Kruzic, Ivan Jerkovic
TL;DR
Whether ChatGPT-3 improves students’ essay performance as a writing assistant remained unknown. The study compared student essays written with and without ChatGPT assistance and found no evidence that the tool improved essay quality; the control group’s overall score was slightly better.
Problem
It remained unknown how ChatGPT performs as a writing assistant in students’ essay-writing environment and whether it enhances their performance.
Source: Introduction · p. 2Method
The study compared essay performance between students writing traditionally and students using ChatGPT assistance.
Source: Materials and methods · p. 3Results
The ChatGPT group did not perform better on essay-quality indicators, and the overall essay score was slightly better in the control group.
Source: Discussion · p. 7Takeaways & Limitations
The study found no evidence that ChatGPT assistance improved the quality of students’ essays in this sample.
Source: Discussion · p. 7This was students’ first interaction with ChatGPT, and the authors recommend expanding the sample size and number in future studies.
Source: Discussion · p. 8
Abstract
from arXiv · showhide
Aim: To compare students' essay writing performance with or without employing ChatGPT-3 as a writing assistant tool. Materials and methods: Eighteen students participated in the study (nine in control and nine in the experimental group that used ChatGPT-3). We scored essay elements with grades (A-D) and corresponding numerical values (4-1). We compared essay scores to students' GPTs, writing time, authenticity, and content similarity. Results: Average grade was C for both groups; for control (2.39, SD=0.71) and for experimental (2.00, SD=0.73). None of the predictors affected essay scores: group (P=0.184), writing duration (P=0.669), module (P=0.388), and GPA (P=0.532). The text unauthenticity was slightly higher in the experimental group (11.87%, SD=13.45 to 9.96%, SD=9.81%), but the similarity among essays was generally low in the overall sample (the Jaccard similarity index ranging from 0 to 0.054). In the experimental group, AI classifier recognized more potential AI-generated texts. Conclusions: This study found no evidence that using GPT as a writing tool improves essay quality since the control group outperformed the experimental group in most parameters.
Introduction
The study asks whether ChatGPT-3 can improve students’ essay performance as a writing assistant, against a backdrop of academic-use and authorship concerns.
Introduction
- ChatGPT had been described as supporting writing through feedback on style, coherence, and grammar, as well as extracting key points and providing citations.Source: Introduction · p. 2
- Prior studies reported that ChatGPT could produce essays on different topics, including high-quality short-form physics essays.Source: Introduction · p. 2
- The introduction raises concerns about academic authorship, evaluating student essays, and plagiarism, alongside proposed revisions to essay settings and guidelines.Source: Introduction · p. 2
- The study investigates whether ChatGPT improves students’ essay grades, reduces writing time, and affects text authenticity.The authors identify students’ writing environment as an unresolved test of ChatGPT as an assistant.Source: Introduction · p. 2
Materials and methods
Eighteen students were divided between traditional essay writing and ChatGPT assistance; essays were graded and compared with student and text measures.
Materials and methods
- Eighteen second-year master’s students participated, with nine in each group; groups were balanced by study module and weighted GPA.The control group wrote traditionally, while the experimental group used ChatGPT assistance.Source: Materials and methods · pp. 2–3
- Both groups wrote an 800–1000-word Croatian essay on biometric identification in forensic science, with up to four hours to finish.The control group was supervised to ensure students did not use ChatGPT.Source: Materials and methods · p. 3
- Two teachers graded essays on mechanics, style, content, and format using A–D grades converted to scores from 4 to 1.Disagreements were resolved by consensus, and each student’s overall score averaged the four grading elements.Source: Materials and methods · p. 3
- The study recorded writing time, checked document authenticity and pairwise similarity, and used an AI text classifier to assess potential AI authorship.A linear regression tested group, writing duration, module, and GPA as predictors of overall essay scores.Source: Materials and methods · p. 3
Results
The control group’s average essay score was slightly higher, and regression found no significant effects of group, writing duration, module, or GPA.
Results
- 172.22 ± 31.59 minutes for the ChatGPT-assisted group versus 179.11 ± 31.93 minutes for control; Figure 1B shows scores by duration.Source: Results · p. 4, Figure 1 · p. 5
- Mean text unauthenticity was 11.87% ±13.45 in the GPT-assisted group versus 9.96% ± 9.81% in control; overall median Jaccard similarity was 0.002.Text similarity in the overall sample was low.Source: Results · p. 4
- The AI classifier marked three ChatGPT-group texts as possibly and five as likely AI-generated; control texts received a wider range of classifications.In the control group, two texts were possibly, one likely, two unlikely, and four unclear.Source: Results · p. 4
- Essay-score associations with duration and GPA were positive, while higher-GPA control students scored higher than those in the ChatGPT group.An association between non-authentic-text proportion and essay scores was detected only in the ChatGPT group.Source: Results · p. 4
- P = 0.184 for group, P = 0.669 for writing duration, P = 0.388 for module, and P = 0.532 for GPA; none significantly affected essay scores.The regression model had R = 0.573 and P = 0.237.Source: Results · p. 4
- 2.39 ± 0.71 for control versus 2.00 ± 0.73 for ChatGPT assistance; both groups averaged grade C, with the control group slightly higher.Figure 1A presents essay scores by group.Source: Results · p. 4, Figure 1 · p. 5
Discussion
In this student sample, ChatGPT assistance did not improve essay outcomes, and the control group scored slightly better. The authors note limits to generalizing these findings and to interpreting AI-detection results for Croatian-language essays.
Discussion
- The authors describe this as the first study testing ChatGPT-3 as an essay-writing assistance tool in a student population.They contrast its findings with more promising studies based mainly on GPT and experienced-researcher interaction, which they suggest may explain the lower performance of their GPT group.Source: Discussion · p. 7
- The ChatGPT group did not perform better on the study’s indicators; overall essay scores were slightly higher in the control group.The authors suggest over-reliance on the tool or students’ unfamiliarity with it could explain the score difference.Source: Discussion · p. 7
- Students in both groups took similar time to complete the task, and longer writing time was related to better essay scores.The authors suggest that combining generated text with students’ own writing style may require extra time to finalize and assemble content.Source: Discussion · p. 7
- The experimental group had slightly more problems with plagiarism, although the authors argue GPT assistance did not necessarily increase assignment non-originality.They also note that essays in the overall sample had a higher degree of authentic text.Source: Discussion · pp. 7–8
- Pairwise essay comparisons showed no remarkable similarities, with different reasoning and styles regardless of ChatGPT use.The authors suggest the tool may produce different outcomes for the same task depending on the user’s input.Source: Discussion · p. 7
- The available ChatGPT text detector produced false positives in the control group, and most classifiers are intended for English.The authors call for improving existing algorithms for different languages or developing language-specific ones.Source: Discussion · p. 8
- The authors suggest ChatGPT’s academic-writing risks may be less concerning when students with GPT assistance do not outperform controls, but it could distract students and contribute to underperformance.They compare using ChatGPT to web-based search, where what users find depends on what they already know.Source: Discussion · p. 8
- The limited sample size prevents generalizing the findings or using a more comprehensive statistical approach.The study also involved Croatian-language essays, students’ first interaction with ChatGPT, and the possibility that English-language use could have generated higher-quality information.Source: Discussion · p. 8