Source-linked AI summary
Language models and Automated Essay Scoring
Pedro Uria Rodriguez, Amir Jafari, Christopher M. Ormerod
TL;DR
Essay scoring is challenging because essays can be long, language quality varies, and hand-scored training data is often limited. The paper compares BERT and XLNet on Kaggle AES against traditional methods and human agreement; neural-network ensembles outperform BOW and exceed human agreement on average in some configurations.
Problem
AES poses modeling challenges because essays can be long, language quality varies, and training samples are often limited by the expense of hand-scoring.
Source: §2 · p. 3Method
The paper compares BERT and XLNet on Kaggle AES with traditional methods, including BOW and LSTM, and describes their transformer architectures.
Source: §6 · p. 15Results
Neural networks averaged 6% higher QWK than BOW, while LSTM and BERT + XLNet ensembles exceeded human agreement by 0.79% and 0.47%, respectively.
Source: §5 · p. 15Takeaways & Limitations
The paper presents BERT and XLNet as essay-scoring approaches and explains their architectures with notation intended to support further improvements.
Source: §6 · p. 15For LSTMs, unrolling and rolling the network approximates rather than computes the true gradient with respect to weights.
Source: §3.2 · p. 7
Abstract
from arXiv · showhide
In this paper, we present a new comparative study on automatic essay scoring (AES). The current state-of-the-art natural language processing (NLP) neural network architectures are used in this work to achieve above human-level accuracy on the publicly available Kaggle AES dataset. We compare two powerful language models, BERT and XLNet, and describe all the layers and network architectures in these models. We elucidate the network architectures of BERT and XLNet using clear notation and diagrams and explain the advantages of transformer architectures over traditional recurrent neural network architectures. Linear algebra notation is used to clarify the functions of transformers and attention mechanisms. We compare the results with more traditional methods, such as bag of words (BOW) and long short term memory (LSTM) networks.
1. Introduction
Automated essay scoring uses statistical models to assign essay grades, and neural networks underpin many current state-of-the-art results. This study compares transformer language models on a Kaggle AES benchmark against earlier approaches.
1. Introduction
- AES is a statistical-model task that assigns grades to essays in educational settings.Its initial use was to reduce scoring costs.Source: §1 · p. 1
- Neural networks are associated with almost all current state-of-the-art AES results, which the paper frames as a classification task.Feedforward networks can model complex relationships, while embeddings represent words or subwords as vectors that preserve semantic information.Source: §1 · p. 1
- RNNs model and classify sequential data, and researchers have applied them with embeddings to NLP tasks including AES.The paper situates essay scoring among applications such as sentiment analysis, topic labeling, language detection, and machine translation [12].Source: §1 · p. 1
- The paper motivates AES by noting that scoring large numbers of student essays can be expensive and time-consuming.AES engines were initially used to reduce scoring costs.Source: §1 · p. 1
- The paper compares BERT and XLNet, transformer-based language models, with earlier AES approaches using the Kaggle dataset as a benchmark.The dataset and prior results provide a way to compare current neural architectures against previous results.Source: §1 · p. 2
2. Automated Essay Scoring
AES models infer essay scores from text, but long essays, limited hand-scored data, and language quality complicate neural-network modeling. The section describes data preparation, score-agreement measures, and evaluation on the ASAP dataset.
2. Automated Essay Scoring
- AES uses essay text as input and human-assigned scores as targets, with training, test, and validation sets used to infer grades.Essays are scored by expert raters using a holistic rubric; the ASAP training material was publicly available.Source: §2 · p. 2
- On ASAP, two raters provide scores, and evaluation uses exact agreement and quadratic weighted kappa (QWK) to measure agreement.QWK captures agreement beyond chance while weighting the extent of disagreement, and uses the full confusion matrix rather than only exact matches.Source: §2 · pp. 2–3
- Long essays, language and spelling quality, and limited training samples make AES challenging for neural networks.Longer sequences can make it difficult to retain information from an essay’s beginning, contributing to convergence issues or low performance.Source: §2 · p. 3
- Earlier AES approaches include BOW and classifiers, while neural-network approaches use embeddings with sequence models such as LSTMs or GRUs.Gating mechanisms mitigate long-term dependency issues to some degree, while attention mechanisms have been shown to accommodate them more effectively.Source: §2 · p. 3
3. Models
This section describes how BOW and recurrent neural networks represent text, then details the attention-based architectures and language-model training of BERT and XLNet. It also explains XLNet’s paired content and query streams for predicting target tokens without [MASK] replacement.
3.1. BOW.
- BOW represents each essay as a word-frequency vector weighted with TF-IDF, but it ignores word order and can be passed to a traditional classifier.The method builds a vocabulary, counts its words, removes very frequent terms, and normalizes the TF-IDF vectors.Source: §3.1 · p. 4
3.2. LSTM.
- LSTMs address vanishing-gradient and instability problems in RNNs using a Constant Error Carousel and input, forget, and output gates to preserve selected information over long delays.RNNs process ordered sequences using prior inputs, outputs, or state; deep-learning frameworks unroll delayed networks for static backpropagation, which approximates the true weight gradients.Source: §3.2 · pp. 4–7
3.3. BERT.
- BERT uses multi-head self-attention to combine information across sequence tokens, then passes token representations through 12 encoder layers and pools the [CLS] output for downstream tasks.Its attention uses learned query, key, and value projections; the base model has 12 heads and 768-dimensional embeddings. Word, positional, and segment embeddings are combined at the input, and an attention mask suppresses padded tokens.Source: §3.3 · pp. 7–9
- BERT’s pretraining masks 15% of tokens for prediction and also trains on next-sentence prediction, combining the two task losses [15].The next-sentence task uses a true following sentence half the time and a randomly sampled sentence the other half; masked-token loss applies only to masked words.Source: §3.3 · pp. 8–9
3.4. XLNET.
- XLNet combines Transformer-XL sequence memory and relative positional attention with permutation language modeling to use bidirectional context while predicting autoregressively.Its multi-head attention maps inputs directly into separate head subspaces, producing richer word representations; pretraining predicts only the last 16.67% of tokens in each factorization.Source: §3.4 · pp. 10–12
- XLNet’s query stream uses the target position but not its content, attending alongside the content stream, which can see the target token.The streams run layer by layer; after 12 layers, XLNet projects the query output to target-token vocabulary scores and minimizes cross-entropy against the real tokens. This pretraining method avoids replacing targets with [MASK], which is absent during fine-tuning.Source: §3.4 · p. 12
4. Fine Tuning and Experiments
The paper fine-tunes BERT and XLNet for essay-score classification, testing transfer-learning and regularization strategies and adapting inputs for essays longer than the models’ sequence limit.
4. Fine Tuning and Experiments
- Pre-training output layers are replaced by a score-classification layer, and the whole model is fine-tuned using the essay representation from the special [CLS] token.BERT uses the first token; XLNet places [CLS] at the end of the essay.Source: §4 · pp. 12–13
- To address catastrophic forgetting and overfitting, experiments use gradual unfreezing, layer-specific learning rates, hidden states from different layers, preprocessing, dropout, and model ensembles.Gradual unfreezing trains the last layer first, then unfreezes one additional layer per epoch; discriminative fine-tuning assigns different learning rates across layers.Source: §4 · p. 13
- The experiments use Adam and base models; uncased BERT works slightly better overall, while large models perform similarly to base models despite higher fine-tuning cost.The selected learning rates were e−5 or 5e−6, and the largest batch sizes fitting a 12GB GPU were 9 for BERT and 8 for XLNet.Source: §4 · p. 13
- Essays exceeding 510 tokens are split into overlapping sequences, and the final score is the rounded average of predictions across splits.Using only the first 510 tokens or the first 128 and last 382 tokens made no significant difference; the authors argue truncating essay content could be unethical.Source: §4 · p. 13
5. Results
On the test set, neural-network ensembles outperform BOW and some exceed inter-human agreement, while individual BERT, XLNet, and LSTM models have similar average QWK; development experiments favor simpler BERT changes overall.
5. Results
- Development-set experiments favored removing stop-words and using three BERT layers, whereas XLNet gained from stop-word removal on only two items and not from using three layers.Methods to avoid catastrophic forgetting generally performed poorly, and increased dropout was not a good overall choice.Source: §5 · pp. 13–14
- Neural networks average 6% higher QWK than BOW; LSTM and BERT + XLNet ensembles exceed inter-human agreement by 0.79% and 0.47%, respectively, and score higher than humans on 5 of 8 items.The final per-item QWK comparisons are presented in Table 2.Source: Table 2 · p. 14, §5 · pp. 14–15
- BERT, XLNet, and LSTM have similar average QWK; ensembles improve over individual models by 0.51% for BERT, 0.53% for XLNet, and 1.45% for LSTM.The 12-model BERT + XLNet ensemble improves by 1.01% over BERT and 1.42% over XLNet; the authors suggest model count may explain the larger LSTM ensemble gain.Source: §5 · pp. 14–15
6. Conclusions
The paper compares BERT and XLNet with traditional methods and human standards for Kaggle essay scoring, reporting better results than human and rule-based techniques and describing the models’ architectures.
6. Conclusions
- The paper reports better results than human and rule-based techniques and presents BERT and XLNet architectures and their essay-scoring implementation.It describes the contribution as explaining and generalizing the architectures with simple notation, then implementing a classification technique for essay scoring.Source: §6 · p. 15