Source-linked AI summary
Automatic Text Scoring Using Neural Networks
Dimitrios Alikaniotis, Helen Yannakoudakis, Marek Rei
TL;DR
Automated Text Scoring can reduce costly and inconsistent manual marking, but effective systems have relied on manually engineered features tuned to particular domains. This paper learns score-specific word representations and uses recurrent networks to represent essays, achieving results that surpass similar and manually engineered systems while enabling visualization of discriminative text regions.
Problem
Existing ATS performance depends substantially on manually selected and tuned linguistic features for different domains, genres, prompts, and datasets.
Method
The model learns score-specific word embeddings and uses recurrent neural networks, including LSTMs, to form representations of extended-response texts.
Results
The architecture surpasses similar state-of-the-art and manually feature-engineered systems, while its visualization method can provide useful feedback to authors.
Takeaways & Limitations
A fully automated ATS framework can learn task-specific representations without prior grammar or domain knowledge and expose information underlying its scoring criteria.
Takeaways & Limitations
For inconsistent scoring across timesteps, the authors identify smaller sentence or phrase inputs as an alternative to computationally expensive gold-score feedback at every timestep.
Abstract
from arXiv · showhide
Automated Text Scoring (ATS) provides a cost-effective and consistent alternative to human marking. However, in order to achieve good performance, the predictive features of the system need to be manually engineered by human experts. We introduce a model that forms word representations by learning the extent to which specific words contribute to the text's score. Using Long-Short Term Memory networks to represent the meaning of texts, we demonstrate that a fully automated framework is able to achieve excellent results over similar approaches. In an attempt to make our results more interpretable, and inspired by recent advances in visualizing neural networks, we introduce a novel method for identifying the regions of the text that the model has found more discriminative.
1 Introduction
Automated Text Scoring offers efficient, consistent grading, but strong performance has traditionally required manually engineered, domain-specific features. The paper proposes recurrent neural networks that learn representations automatically for holistic scoring of extended responses and supports visualization of discriminative text regions.
- ATS automatically scores texts on a marking scale, offering cost-effective large-scale grading and consistent application of marking criteria.
- Existing ATS systems combine features covering grammar, vocabulary, style, topic relevance, and discourse coherence and cohesion.
- Although prior regression and ranking approaches can match human examiners, their hand-selected features require substantial tuning across domains, genres, and prompts.
- The paper proposes recurrent neural networks to learn dataset-specific feature representations automatically while predicting holistic scores for extended-response writing items.
- The authors also describe a preliminary visualization method for identifying text information that the network exploits when assigning a score.
2 Related Work
Related work has approached automated text scoring with diverse statistical, linguistic, semantic, and machine-learning methods. These systems include early regression and semantic-similarity models, operational feature-based scoring, ranking approaches, and newer datasets and feature combinations.
- Early systems included PEG’s linear regression over textual-feature vectors and Intelligent Essay Assessor’s latent-semantic similarity scoring.
- Other approaches used parsers, Naive Bayes classification, regression-fitted grammar, vocabulary, and style features, and discriminative learning-to-rank models.
- Later work combined hierarchical classification or regression with linguistic, semantic, rhetorical, opinion-expression, and topical features.
- Additional feature-based studies incorporated word-association statistics, lexical chains, and interactions with discourse elements for essay evaluation.
- The ASAP contest released around twenty thousand middle-school English-speaking student texts, 60% of which were marked, for automated scoring research.
3 Models
The models learn contextual and score-specific word representations, encode essays with LSTMs, and compare these approaches with paragraph-vector and regression baselines.
- 3.1 C&W Embeddings: C&W embeddings learn word vectors by distinguishing real word sequences from sequences with randomly substituted words.The model maps vocabulary words to columns of an embedding matrix and optimizes a hinge loss separating original and noisy sequences.
- 3.2 Augmented C&W model: Score-specific word embeddings add essay-score prediction to contextual learning, weighting score-specific and contextual losses with α.Lower α emphasizes score information, while higher α favors contextual information.
- 3.2 Augmented C&W model: SSWEs separate correctly spelled words from misspellings while retaining their contextual relatedness.The model uses the tendency of misspellings to occur in lower-scoring essays to make this distinction.
- 3.3 Long-Short Term Memory Network: LSTMs process word embeddings sequentially and use the final hidden representation to predict essay scores with linear regression.The experiments include unidirectional, bidirectional, and deeper LSTM representations, with bidirectional passes concatenated before prediction.
- 3.4 Other Baselines: PV-DM directly learns essay embeddings by assigning each paragraph a memory vector, then feeds paragraph vectors to linear regression for scoring.The paragraph vector retains information from contexts appearing throughout the paragraph.
4 Dataset
The Kaggle dataset contains 12,976 student essays from eight prompt-specific sets, with scores from two raters and fixed training, validation, and test splits.
- Dataset: 12,976 essays of 150–550 words were written by Grade 7–10 students across eight prompts and marked by two raters.The raters had Cohen’s κ = 0.86, and the experiments use a resolved combined score.
- Dataset: 64% of the dataset was used for training, 16% for validation, and 20% for testing.The authors release validation and test essay identifiers, source code, and hyperparameter values.
5 Experiments
Experiments compare neural, embedding-based, paragraph-vector, and SVM models on the Kaggle dataset using validation-set optimization and test-set evaluation. The strongest results come from SSWE-based LSTM models, while performance varies across correlation and RMSE metrics.
- Experimental setup: Models were evaluated on the Kaggle dataset after Bayesian Optimization selected hyperparameters using a separate validation set.The test set reports Spearman correlation, Pearson correlation, RMSE, and Cohen’s κ with quadratic weights.
- Baseline comparisons: SVMs achieved competitive correlations of ρ = 0.78 and r = 0.77 but had among the weakest RMSE values at 8.85.They outperformed the listed doc2vec, LSTM, BLSTM, and deeper counterparts in correlation.
- Neural model comparisons: Word2vec-trained two-layer BLSTMs were among the most competitive correlation models, outperforming all models except those using pre-trained embeddings and SSWEs.Adding depth or bidirectionality did not always help, but improved performance here relative to unidirectional counterparts.
- Embedding comparisons: The pre-trained word2vec plus two-layer BLSTM configuration reached ρ = 0.79, r = 0.91, and RMSE = 3.2.Its comparison with the authors’ models was not entirely fair because the pre-trained embeddings used a much larger corpus.
- Best results: The best SSWE plus two-layer BLSTM model reached ρ = 0.91, r = 0.96, and RMSE = 2.4, improving correlation by around 10%.The authors report that this approach also outperformed the other models in Cohen’s κ.
- Discussion: The SSWE plus LSTM approach required no prior grammar or domain knowledge and used no text preprocessing beyond simple tokenization.The authors describe its scoring as very human-like and report that it outperformed other state-of-the-art systems.
6 Visualizing the black box
The paper introduces gradient-based visualizations that identify words contributing to an LSTM’s score predictions, making the model’s internal scoring criteria more interpretable. The method also exposes concrete strengths and limitations, including delayed gradients and difficulty distinguishing repeated words used differently.
- Method: The visualization method evaluates word-vector quality by measuring how problematic a word in context is for the network’s score prediction.An essay is passed left to right, and the LSTM predicts using a pseudo-score without updating network weights.
- Visualization: Table 2 encodes word-vector quality with colors ranging from dark red for low quality to dark green for high quality.The full essay text appears in black, while the word-level quality values are overlaid in color.
- Method: Essay-dependent quality scores combine error-gradient magnitudes from maximum and minimum pseudo-scores.The combined score is computed as L(ỹmax, f(x)) − L(ỹmin, f(x)), with the essay-set maximum used for ỹmax.
- Findings: The examples show that the model favors correctly placed punctuation and long-distance dependencies, while handling proper names poorly and some POS mistakes successfully.The visualization also contains a false negative for “satisfied,” demonstrating that the model is not perfect.
- Limitations: Because gradients are calculated only after the essay ends, repeated words used correctly and incorrectly may not be distinguished.The authors note that providing gold scores at each timestep would be computationally expensive; smaller sentence or phrase inputs are an alternative.
7 Conclusion
The paper concludes that its deep neural architecture learns score-specific word representations and uses an LSTM to form essay representations. It surpasses comparable state-of-the-art and manually engineered systems, while its visualization method supports interpretable feedback.
- Contribution: The model learns score-specific word embeddings that capture local contextual and usage information for essay scoring.These embeddings are subsequently used by a recurrent neural network to represent complete essays.
- Results: The architecture surpasses similar state-of-the-art systems and manually feature-engineered systems near the upper bound of prior work.
- Interpretability: The visualization method explores the basis of the network’s internal scoring criteria and can provide useful feedback to authors.