Source-linked AI summary
Neural Automated Writing Evaluation with Corrective Feedback
Izia Xiaoxiao Wang, Xihan Wu, Edith Coates, Min Zeng, Jiexin Kuang, Siliang Liu, Mengyang Qiu, Jungyeul Park
TL;DR
Because essay feedback and revision are laborious, the paper integrates neural AWE with GEC for a simulated examination workflow. Its conclusion presents the system as a promising resource for L2 learning, with scoring and grammatical corrections returned together.
Problem
AWE and GEC address scoring and grammatical correction separately, while providing and revising essay feedback is laborious for instructors and learners.
Source: §1 · p. 1Method
The paper integrates neural AWE and GEC so learners can submit writing for assessment and corrective feedback in a simulated examination workflow.
Source: §1 · pp. 1–2, §4 · p. 5Results
The conclusion presents the integrated system as a promising language resource for improving the L2 learning experience.
Source: §5 · p. 6Takeaways & Limitations
Instructors can intervene with the AWE and GEC process to adjust rubrics and conventions before results reach students.
Source: §4 · p. 5Prompting GPT for GEC did not obtain state-of-the-art results.
Source: §3 · p. 4
Abstract
from arXiv · showhide
The utilization of technology in second language learning and teaching has become ubiquitous. For the assessment of writing specifically, automated writing evaluation (AWE) and grammatical error correction (GEC) have become immensely popular and effective methods for enhancing writing proficiency and delivering instant and individualized feedback to learners. By leveraging the power of natural language processing (NLP) and machine learning algorithms, AWE and GEC systems have been developed separately to provide language learners with automated corrective feedback and more accurate and unbiased scoring that would otherwise be subject to examiners. In this paper, we propose an integrated system for automated writing evaluation with corrective feedback as a means of bridging the gap between AWE and GEC results for second language learners. This system enables language learners to simulate the essay writing tests: a student writes and submits an essay, and the system returns the assessment of the writing along with suggested grammatical error corrections. Given that automated scoring and grammatical correction are more efficient and cost-effective than human grading, this integrated system would also alleviate the burden of manually correcting innumerable essays.
1 Introduction
The introduction motivates combining automated scoring and grammatical correction to ease the work of essay feedback while giving L2 learners prompt, individualized support. It presents an integrated system for simulated examination practice.
1 Introduction
- Providing and revising essay feedback is laborious, while learners and instructors expect sufficient feedback on submissions.AWE and GEC are presented as ways to alleviate instructors’ workload and learners’ effort in understanding feedback.Source: §1 · p. 1
- AWE evaluates and scores written prose computationally, whereas GEC detects and corrects grammatical errors.The introduction describes AWE as predicting holistic or rubric scores and GEC commonly as translating incorrect sentences into correct ones.Source: §1 · p. 1
- The paper proposes integrating AWE and GEC to support language learning through simulated examination situations.The system is intended to provide learners with immediate scoring and corrective feedback.Source: §1 · pp. 1–2
2 Previous Work
Previous work describes neural approaches to AWE and GEC, while earlier AWE–GEC integrations focused on learners’ perceptions. The paper identifies neural integration and its relation to writing performance as an area of interest.
AWE
- AWE research includes RNN-based essay scoring and transformer-based models, with multi-loss BERT fine-tuning reported to produce state-of-the-art results.Examples include RNN scoring architectures (Alikaniotis et al., 2016), transformer variants, and multi-loss fine-tuning.Source: §2 · p. 2
GEC
- Recent state-of-the-art GEC systems comprise sequence-to-sequence NMT-based and sequence-tagging edit-based approaches; edit-based models predict editing operations rather than full sentences, dramatically improving inference speed while maintaining high performance.Source: §2 · p. 2
AWE–GEC integration
- Prior AWE–GEC integration studies examined learner perceptions, while the influence of feedback on writing performance remained unknown.The paper introduces neural AWE and GEC integration as a first step toward results close to state of the art.Source: §2 · p. 2
3 System Description
The system combines AWE scores with inline GEC corrections in a learner-facing examination workflow. Its description covers the scoring and correction models, datasets, preprocessing, and displayed results.
System workflow
- Learners submit writing to receive inline GEC corrections alongside one overall AWE score and eight rubric scores.The workflow combines the two outputs for the learner and is described as simulating examination situations.Source: §3 · pp. 2–3, Figure 1 · p. 3
Interface
- The interface marks deleted tokens in faded red and inserted grammatical corrections in green, with AWE scores scaled from 0 to 100.AWE scores can be min-max normalized to learner proficiency when required.Source: Figure 2 · p. 3, §3 · p. 3
AWE
- The AWE setup uses ASAP and ASAP++ essays, which cover eight prompts and include human-rated overall and rubric scores.ASAP++ supplements ASAP with additional rubric scores for prompts that originally had only overall scores.Source: §3 · p. 3
- The AWE pipeline denoises ASAP text with two GPT-3.5-turbo-instruct prompts, restores original wording with ERRANT annotations, and trains RoBERTa-base for linear regression.The reported evaluation compares cleaned text with prompt-by-prompt denoising results and calculates QWK after score normalization.Source: §3 · pp. 3–4
GEC
- 65.29 F0.5 is the best BERT model’s score on the BEA 2019 test set.The GEC models use four BEA 2019 Shared Task datasets and an augmented total of 1,718,693 sentence pairs.Source: §3 · p. 4
- GPT prompting for GEC did not obtain state-of-the-art results, so it is not the reported best-performing GEC approach.The text says this limitation also appears in recent prior work (Loem et al., 2023).Source: §3 · p. 4
4 Discussion and Future Perspectives
The discussion outlines plans to connect GEC results to AWE accuracy features, extend both systems toward multilingual use, and support instructors as well as learners. It also identifies learner and instructor perceptions as questions for planned experiments.
Features and metrics
- GEC results are planned as accuracy features for AWE, whose complexity and fluency features are already represented in the integrated system.The AWE and GEC models are currently trained independently.Source: §4 · p. 4
Multilingual aspects for GEC
- The current English GEC system is being extended toward Chinese and Korean using multilingual Transformers such as mBART.The described training data include Chinese and Korean L2 writing samples.Source: §4 · pp. 4–5
Multilingual aspects for AWE and universal rubrics
- The English AWE system is designed for multilingual input through shared scoring schemes and rubrics based on communicative effectiveness rather than language-specific traits.The authors describe developing multilingual rubric scoring using English, Chinese, and Korean evaluation rubrics.Source: §4 · p. 5
Teacher’s view
- Instructors can adjust the AWE and GEC process to their rubrics before students receive results, extending the workflow toward actual examinations.The authors say this could reduce instructors’ manual essay-correction burden and let them focus on teaching.Source: §4 · p. 5, Figure 3 · p. 5
Perception problem
- Planned experiments will examine learner and instructor perceptions because the effect of automated feedback across proficiency levels remains unclear.The learner arm concerns system results without instructor intervention; the instructor arm concerns evaluating results conveyed to learners.Source: §4 · p. 6, Table 2 · p. 6
5 Conclusion
The conclusion presents a public-facing integration of neural AWE and GEC for language education. It describes hands-on writing practice and examination simulation as intended uses.
5 Conclusion
- The conclusion presents the proposed system as a seamless integration of neural AWE and GEC for language education.Source: §5 · p. 6
- The system is intended for public access as a language resource for learners seeking to improve writing skills and language proficiency through hands-on mistakes.Source: §5 · p. 6
- The system encourages learners to “write it right” and provides features that extend from simulated examinations toward actual examination situations.Source: §5 · p. 6
Ethics Statement
The authors obtained a TCPS 2: CORE-2022 ethics-course certificate to confirm Behavioural Research Ethics at the University of British Columbia.
Ethics Statement
- The authors obtained an ethics-course certificate to confirm Behavioural Research Ethics at the University of British Columbia.The passage does not state that the certificate constitutes ethics approval.Source: Ethics Statement · p. 6
- The certificate is for the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2) course, CORE-2022.Source: Ethics Statement · p. 6
- The stated ethics confirmation concerns research involving humans.Source: Ethics Statement · p. 6