Source-linked AI summary
Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays
Rahul Kumar, Sandeep Mathias, Sriparna Saha, Pushpak Bhattacharyya
TL;DR
Automatic essay grading has emphasized holistic scores, leaving limited use of trait information for holistic assessment. This paper jointly scores holistic essays and traits with multi-task learning, finding that the BiLSTM-based MTL system performs best holistically while remaining competitive on traits and speeding processing.
Problem
Most automatic essay-grading research focuses on holistic scores, with limited use of essay-trait information for holistic assessment.
Method
The paper uses multi-task learning with holistic essay scoring as the primary task and trait scoring as auxiliary tasks, comparing LSTM and BiLSTM models with single-task systems.
Results
The MTL BiLSTM system outperforms STL-based systems on holistic scoring, performs close to STL systems on traits, and achieves a 2.30 to 3.70 speed-up.
Takeaways & Limitations
Trait scoring can support holistic essay grading while also identifying important traits for different essay sets through ablation tests.
Takeaways & Limitations
The dataset lacks narrativity scores for narrative essays, and cross-domain essay grading remains future work.
Abstract
from arXiv · showhide
Most research in the area of automatic essay grading (AEG) is geared towards scoring the essay holistically while there has also been some work done on scoring individual essay traits. In this paper, we describe a way to score essays holistically using a multi-task learning (MTL) approach, where scoring the essay holistically is the primary task, and scoring the essay traits is the auxiliary task. We compare our results with a single-task learning (STL) approach, using both LSTMs and BiLSTMs. We also compare our results of the auxiliary task with such tasks done in other AEG systems. To find out which traits work best for different types of essays, we conduct ablation tests for each of the essay traits. We also report the runtime and number of training parameters for each system. We find that MTL-based BiLSTM system gives the best results for scoring the essay holistically, as well as performing well on scoring the essay traits.
1 Introduction
Automatic essay grading has focused mainly on holistic scores, while this paper asks whether essay-trait information can improve holistic scoring and support simultaneous trait assessment.
- Essay traits describe aspects such as content, organization, style, and prompt adherence that can help explain an essay’s overall score.
- Automatic essay grading has largely emphasized holistic scoring rather than the role of individual essay traits.
- The paper asks whether information learned from scoring essay traits can be used to score essays holistically.
- Its multi-task framework scores essay traits as auxiliary tasks while scoring the essay holistically as the primary task.
- The paper evaluates the approach across essay types and traits and provides code and data for reproducibility and further research.
2 Motivation
Trait-specific scores may provide useful information for both explaining holistic essay grades and improving automatic scoring because trait scores correlate strongly with overall scores.
- Trait-specific scores can tell writers which essay aspects need improvement, whereas a holistic score alone provides less specific feedback.
- Trait scores correlate strongly with overall essay scores, with Pearson correlations above 0.7 across all essay sets in the dataset.
- The authors therefore argue that trait scores may provide more relevant information for holistic automatic essay grading.
3 Related Work
Prior work separately developed holistic and trait-specific essay grading, while this paper combines trait scores with holistic grading through multi-task learning.
- 3.1 Holistic Essay Grading: Related work includes commercial and research systems for assigning holistic essay scores, using machine-learning and deep-learning approaches such as LSTMs and CNNs.
- 3.2 Trait-specific Essay Grading: Researchers have scored traits including fluency, organization, thesis clarity, coherence, prompt adherence, argument strength, stance, style, and narrative quality, but not used trait information to score essays holistically.
- 3.2 Trait-specific Essay Grading: Earlier multi-trait systems established neural approaches for trait scoring, whereas this work combines trait scores with holistic essay grading.
- 3.2 Trait-specific Essay Grading: The proposed multi-task approach jointly scores essays and traits, aiming to speed training without substantial loss in trait-scoring performance.
- 3.3 Multi-task Learning: Other multi-task systems used auxiliary tasks such as sentence classification, paragraph classification, sentence reordering, noise identification, or domain-adversarial training.
4 System Architecture
The system builds essay representations from token and sentence processing, then uses separate STL stacks or a shared MTL architecture that combines predicted trait scores with holistic representations.
- 4.1 STL Essay Grading Stack: Each essay-grading stack processes tokenized sentences and returns either a holistic essay score or an individual trait score.
- 4.1 STL Essay Grading Stack: The stack uses word embeddings, a one-dimensional CNN for local information, attention pooling for sentence representations, and recurrent processing with LSTM or BiLSTM layers.
- 4.1 STL Essay Grading Stack: Attention-pooled essay representations feed a sigmoid-activated dense layer, with scores normalized to [0, 1] before prediction and rescaled afterward.
- 4.1 STL Essay Grading Stack: Figure 1 presents the essay-stack architecture used by the single-task learning systems.
- 4.2 MTL Model: In MTL, the word-embedding layer is shared, while trait-specific and overall essay representations are learned before predicted trait scores are concatenated with the holistic representation.
5 Dataset Used
The experiments use the ASAP AEG dataset, comprising eight essay sets and nearly 13,000 essays, with prompt-dependent trait annotations and a limitation for narrativity scores.
- The ASAP AEG dataset contains eight essay sets and nearly 13,000 essays written by students in classes 7 to 10.
- Table 1 summarizes each essay set’s scoring range, trait scoring, average word count, number of traits, essay count, and essay type.
- Overall scores come from the original ASAP dataset, while trait-specific scores are supplemented with annotations from Mathias and Bhattacharyya (2018a).
- Essay sets differ by prompt type and include distinct trait inventories, such as content, organization, word choice, conventions, language, narrativity, and prompt adherence.Argumentative or persuasive, source-dependent, and narrative or descriptive essays use different trait combinations.
- Narrativity is unavailable for the narrative essays because neither the original ASAP dataset nor Mathias and Bhattacharyya (2018a) scored it.
6 Experiments
The experiments compare single-task and multi-task LSTM variants under quadratic weighted kappa and five-fold validation, using fixed network settings across configurations.
- 6.1 Evaluation Metric: Quadratic weighted kappa is used because scores are discrete and ordered, and it distinguishes mismatches more sharply than linear weighted kappa.
- 6.2 Evaluation Method: Five-fold cross-validation uses 60% training, 20% validation, and 20% testing data, with validation-selected models evaluated on each test fold.The experiments use the same data splits as Taghipour and Ng (2016).
- 6.3 Network Hyperparameters: The systems use shared hyperparameters across STL and MTL and across LSTM and BiLSTM configurations, including GloVe embeddings, 100 recurrent hidden units, 100 epochs, and dropout of 0.5.The configuration also uses a word-level CNN window of 5 with 100 filters, batch size 100, RMSProp, learning rate 0.001, and momentum 0.9.
- 6.3 Network Hyperparameters: Table 2 records which traits are available for each essay set, using scores from the original ASAP dataset and ASAP++ Mathias and Bhattacharyya (2018a).
- 6.4 Experimental Configurations: The study compares STL-LSTM, STL-BiLSTM, MTL-LSTM, MTL-BiLSTM, and a state-of-the-art string-kernel system for holistic essay scoring.STL predicts one score at a time, whereas MTL predicts the essay score and all traits simultaneously; BiLSTM uses both forward and reverse directions.
- 6.4 Experimental Configurations: The experiments use the same training, validation, and test splits for the external string-kernel comparison because its authors did not provide their folds.
7 Results and Analysis
Across holistic scoring, trait scoring, ablations, error analysis, and resource use, the experiments favor MTL-BiLSTM for holistic performance while retaining strong trait scores and reducing training time.
- 7.1 Performance on Holistic Essay Scoring: MTL-BiLSTM performs best among the evaluated systems for holistic essay scoring and matches Tay et al.’s reported results.Table 4 compares STL, MTL, string-kernel, and Tay et al.’s systems across Prompts 1–8.
- 7.1 Performance on Holistic Essay Scoring: MTL systems are tested against STL-LSTM, while statistically significant MTL and BiLSTM improvements are marked in Table 4.The comparisons use paired t-tests with p < 0.05.
- 7.2 Performance on Scoring Essay Traits: MTL-BiLSTM scores essay traits at about 97% of STL-LSTM performance for most traits, although STL-LSTM generally performs better on the individual trait task.The systems optimize different objectives: STL-LSTM targets the trait alone, whereas MTL jointly optimizes holistic scoring.
- 7.3 Ablation Tests: Content is the most important trait for three essay sets, while Prompt Adherence and Word Choice are most important for two essay sets each.Importance is determined by the holistic-score drop after ablating one trait at a time.
- 7.4 Error Analysis: MTL generally helps holistic scoring when no well-defined rule maps trait scores to the holistic score.For Prompts 7 and 8, pipelined trait prediction produced QWK values of 0.796 vs. 0.795 and 0.684 vs. 0.699, respectively, relative to the MTL-based comparison.
- 7.5 Runtime Analysis: MTL models provide a 2.30 to 3.70 speed-up over STL models for training holistic and trait scorers.Training time covers all prompts, traits, folds, and 100 epochs.
- 7.5 Runtime Analysis: MTL models use about 1.38 million parameters for Prompts 3–7 and over 1.85 million for Prompt 8, which has six traits.The number of MTL parameters varies with the number of essay traits, whereas STL parameter counts are constant across essay sets.
8 Conclusion and Future Work
The paper uses multi-task learning to score essays and their traits by concatenating essay representations with auxiliary trait predictions. The MTL BiLSTM outperforms STL systems, approaches state-of-the-art holistic scoring, performs comparably on traits, and trains 2.30–3.70 times faster than STL models.
- 8 Conclusion and Future Work: The approach concatenates an essay representation with trait scores predicted as an auxiliary task.
- 8 Conclusion and Future Work: The MTL BiLSTM outperforms STL-based systems on holistic essay scoring and has results comparable with Tay et al.’s state-of-the-art system.
- 8 Conclusion and Future Work: MTL trait-scoring performance is close to that of STL systems, while ablations identify important traits for corresponding essay sets.
- 8 Conclusion and Future Work: MTL training is 2.30 to 3.70 times faster than STL training.
- 8 Conclusion and Future Work: Future work will use trait scoring for text-level writer feedback and investigate cross-domain essay grading across prompts.
Ethics Statement
The essay data protect writer privacy through anonymization, and the trait-specific annotation data were collected ethically with compensation for annotators.
- Ethics Statement: Nearly 13,000 essays were anonymized by removing names, dates, numbers, and other identifying information.
- Ethics Statement: Trait-specific annotators were adequately compensated at INR 5 per essay.