Source-linked AI summary
On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation
Yongjie Wang, Chuan Wang, Ruobing Li, Hui Lin
TL;DR
AES aims to automate essay assessment, but BERT-based systems had not consistently surpassed other deep learning models. The paper jointly learns multi-scale essay representations with BERT and adds multiple losses plus out-of-domain transfer learning; it reports almost state-of-the-art deep-learning results on ASAP and strong benefits for long texts.
Problem
BERT-based AES systems had not properly outperformed LSTM and other deep learning models, partly because essay representation and training objectives were limited.
Method
The paper jointly learns multi-scale essay representations with BERT and combines multiple losses with transfer learning from out-of-domain essays.
Results
The approach obtains almost state-of-the-art results among deep learning models on ASAP and shows significant advantages for long-text tasks.
Takeaways & Limitations
Multi-scale essay representation is presented as an effective choice for long-text tasks and generalizes to the CommonLit Readability Prize dataset.
Takeaways & Limitations
Future work could explore soft multi-scale representation and more linguistically reasonable segmentation scales.
Abstract
from arXiv · showhide
In recent years, pre-trained models have become dominant in most natural language processing (NLP) tasks. However, in the area of Automated Essay Scoring (AES), pre-trained models such as BERT have not been properly used to outperform other deep learning models such as LSTM. In this paper, we introduce a novel multi-scale essay representation for BERT that can be jointly learned. We also employ multiple losses and transfer learning from out-of-domain essays to further improve the performance. Experiment results show that our approach derives much benefit from joint learning of multi-scale essay representation and obtains almost the state-of-the-art result among all deep learning models in the ASAP task. Our multi-scale essay representation also generalizes well to CommonLit Readability Prize data set, which suggests that the novel text representation proposed in this paper may be a new and effective choice for long-text tasks.
1 Introduction
Automated Essay Scoring supports automated assessment but remains challenging because BERT-based systems have not consistently surpassed other deep learning approaches. The paper addresses this gap with jointly learned multi-scale representations, multiple losses, and transfer learning.
- Motivation: Teacher scoring draws on signals at token, sentence, paragraph, and other granularity levels, motivating multi-scale essay features.These features include vocabulary, syntax, word counts, and essay structure.
- Motivation: BERT-based AES systems have struggled to outperform LSTM and other deep neural network methods because essay-level representation learning is limited.The paper identifies sentence-level pre-training, limited AES data, and reliance on MSE as key limitations.
- Method: The proposed BERT approach jointly learns multi-scale essay representations and outperforms state-of-the-art LSTM-based deep learning models.It explicitly extracts multi-scale features while leveraging knowledge from sentence data.
- Results: The method obtains almost state-of-the-art results among deep learning models on ASAP, with reported advantages for long-text tasks.The contribution statement frames multi-scale representation as particularly beneficial for long essays.
- Method: Multiple losses inspired by teacher rating and transfer learning from out-of-domain essays with R-Drop further improve essay-scoring performance.The method combines additional losses with MSE to incorporate scoring-distribution diversity.
2 Related Work
AES research spans traditional feature-based systems, deep neural networks, and pre-trained language models. Related work emphasizes the portability of learned features while noting that BERT-based AES had not yet shown a clear advantage over other neural approaches.
- Taxonomy: AES approaches are grouped into traditional systems, deep neural networks, and pre-training methods.This categorization organizes the field by feature construction and representation-learning strategy.
- Traditional AES: Traditional AES uses handcrafted linguistic features with regression or ranking, allowing good performance with small datasets.Manual feature design relies on prior knowledge from linguists but is complicated and less portable.
- Deep Neural Networks AES: Deep neural AES models such as LSTM and CNN automatically learn complex essay features, making scoring an end-to-end task.These methods achieved results comparable with traditional AES in recent work.
- Pre-training AES: Pre-training AES fine-tunes language models for essay scoring, but reported BERT-based systems failed to outperform other deep learning methods.The related work contrasts pre-training's broad NLP success with its weaker demonstrated advantage in AES.
3 Approach
The approach jointly learns essay representations at token, segment, and document scales with BERT, then combines their predictions. It also trains with regression, similarity, and ranking losses designed to reflect essay-scoring judgments.
- 3.2 Multi-scale Essay Representation: Joint multi-scale BERT representations combine token-, segment-, and document-level essay information for scoring.Token-scale representations use max pooling, segment-scale representations use BERT, LSTM, and attention pooling, and document-scale representations use BERT’s [CLS] output.
- 3.2 Multi-scale Essay Representation: BERT processes token sequences with [CLS], [SEP], padding, and position, segmentation, and token embeddings, while long sequences are divided into segment inputs.The segment-scale procedure truncates or pads essays, divides them into segments, obtains [CLS] vectors, and processes those vectors with an LSTM followed by attention pooling.
- 3.3 Model Architecture: The final essay score sums predictions from all segment scales with the shared document- and token-scale prediction.The document- and token-scale representations are concatenated before regression, while each segment scale has its own prediction.
- 3.4 Loss Function: Similarity loss models batch-level score-distribution agreement, while Margin Ranking loss penalizes incorrect pairwise essay ordering.These objectives reflect teachers’ use of overall score distributions and the importance of sorting essays by score.
- 3.4 Loss Function: The training objective combines MSE, Margin Ranking, and Similarity losses with tuned weights.MSE measures squared prediction error, Similarity compares predicted and label vectors, and Margin Ranking penalizes incorrect essay ordering.
4 Experiment
The experiments evaluate multi-scale BERT representations, transfer learning, multiple losses, and R-Drop for automated essay scoring across ASAP and CRP settings. Results consistently favor joint multi-scale representations, with further gains from transfer learning and auxiliary losses.
- 4 Experiment: The experiments compare document-, token-, and segment-scale BERT representations, Longformer variants, transfer-learning models, and multiple-loss configurations using ASAP QWK and CRP RMSE.Transfer learning pre-trains on out-of-domain essays before fine-tuning on in-domain essays; the multi-scale model combines document, token, and segment representations.
- Results: 0.761 to 0.772: the proposed model improves performance on ASAP prompts 1, 2, and 8 whose WordPiece lengths exceed 510, nearly matching the published neural state of the art.Longformer performs poorly in this comparison, while the proposed approach outperforms a neural-feature-only system reported at QWK 0.730.
- Results: 0.764 to 0.782: multi-scale BERT representations outperform LSTM-based models under the same regression loss.The comparison isolates the representation change and is reported as evidence for stronger long-text encoding.
- Multi-scale Representation: BERT-DOC-TOK outperforms single-scale BERT-DOC and BERT-TOK, while adding segment-scale features improves performance further on ASAP and CRP.The feature-scale comparisons report significant improvements for BERT-DOC-TOK-SEG in most ASAP cases and on CRP.
- Reasons for Effectiveness of Multi-scale Representation: Longformer-DOC-TOK-SEG significantly outperforms Longformer-DOC, indicating that multi-scale encoding contributes beyond long-sequence processing alone.The comparison also reports that Longformer-DOC performs poorly despite supporting long-sequence encoding.
- Transfer Learning with Multiple Losses and R-Drop: 0.782 to 0.788: transfer learning improves BERT-DOC-TOK-SEG on ASAP, while multiple losses raise performance to 0.790 and R-Drop adds a slight QWK gain.The authors attribute the multiple-loss improvement to ranking and score-distribution information, and describe R-Drop as regularization.
5 Conclusion and Future Work
The paper proposes multi-scale essay representation with multiple losses and transfer learning for automated essay scoring, nearly matching state-of-the-art deep-learning results. It also identifies soft multi-scale representation as a future direction.
- The approach nearly obtains state-of-the-art results among deep-learning models while using multi-scale representation, multiple losses, and transfer learning for AES.
- Multi-scale representation shows a significant advantage for handling long texts.
- Exploring soft multi-scale representation and linguistically informed segmentation could bring further improvement.
A Appendix
The appendix describes a greedy search for selecting segment scales because the number of possible scale combinations grows exponentially. The procedure evaluates candidate scales and retains the best-performing combination on the development set.
- The greedy search selects segment-scale combinations using development-set performance because all combinations grow exponentially with the number of scales.
- The search initializes the scale set with document-scale and token-scale representations.
- Candidate segment scales are evaluated with the token- and document-scale representations, then ranked by average development-set QWK.
- The procedure tests progressively larger prefixes of the ranked candidate list and retains the combination with the best development-set performance.