Source-linked AI summary
Linguistic Features for Readability Assessment
Tovly Deutsch, Masoud Jasbi, Stuart Shieber
TL;DR
Readability research has not established whether linguistic features can improve deep learning models, despite their use in traditional approaches. This paper combines linguistic features with deep models and finds they do not generally improve deep-learning performance; the authors suggest the models may already capture useful features.
Problem
Method
The paper combines linguistic features with outputs from Transformer and hierarchical attention models and evaluates them on WeeBit and Newsela with different training-set sizes.
Source: §1 · p. 1Results
Linguistic features occasionally improved performance, particularly with small training sets, but the resulting models did not achieve state-of-the-art performance.
Source: §5 · p. 9Takeaways & Limitations
The findings provide preliminary evidence that state-of-the-art deep learning models may implicitly capture linguistic features useful for readability assessment.
Source: §5 · p. 9Ambiguity in corpus construction limits assessment of label consistency and validity, so state-of-the-art models may already reach the maximum possible performance on available corpora.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Readability assessment aims to automatically classify text by the level appropriate for learning readers. Traditional approaches to this task utilize a variety of linguistically motivated features paired with simple machine learning models. More recent methods have improved performance by discarding these features and utilizing deep learning models. However, it is unknown whether augmenting deep learning models with linguistically motivated features would improve performance further. This paper combines these two approaches with the goal of improving overall model performance and addressing this question. Evaluating on two large readability corpora, we find that, given sufficient training data, augmenting deep learning models with linguistically motivated features does not improve state-of-the-art performance. Our results provide preliminary evidence for the hypothesis that the state-of-the-art deep learning models represent linguistic features of the text related to readability. Future research on the nature of representations formed in these models can shed light on the learned features and their relations to linguistically motivated ones hypothesized in traditional approaches.
1 Introduction
Readability assessment classifies text by appropriate reading level, but it is unknown whether linguistic features can improve deep learning models. The paper combines linguistic features with deep models and finds no substantial benefit given sufficient training data.
1 Introduction
- Traditional systems feed linguistically motivated features into simple models, while newer readability systems use deep neural networks and attention mechanisms.The paper frames combining these approaches as a possible way to improve performance, particularly with limited data.Source: §1 · p. 1
- Given sufficient training data, combining linguistic features with deep learning does not substantially improve performance over deep learning methods.The evaluation uses Transformers and hierarchical attention networks on WeeBit and Newsela, with different training-set sizes.Source: §1 · p. 1
- The paper evaluates the joint use of linguistic features and deep learning models to test whether their combination improves readability assessment.The introduction describes the rest of the paper as covering related research, model construction, evaluations on two corpora, and implications.Source: §1 · pp. 1–2
2 Related Work
Readability research has developed across corpora, features, and models, from small datasets and simple features to deep learning. This paper tests whether linguistic features could complement deep models, especially where data are limited.
2 Related Work
- Readability corpora grew from WeeklyReader to WeeBit, while Newsela represents articles at multiple reading levels to reduce topic-readability correlation.WeeBit contains roughly 6,000 documents; Newsela was constructed so each article appears at multiple reading levels.Source: §2 · p. 2
- Feature research expanded from simple character counts to psycholinguistic, lexical, and syntactic measures, while modeling progressed from linear models and SVMs to deep neural networks.Vajjala and Meurers (2012) assembled a broad feature set drawing on linguistic databases and measures associated with second-language acquisition.Source: §2 · p. 2
- Deep learning models have achieved state-of-the-art readability performance while generally excluding linguistic features and relying on models to learn relevant features from raw word embeddings.The paper motivates testing whether precomputed features may help in data-poor settings where acquiring data is expensive and error-prone.Source: §2 · p. 2
3 Methodology
The methodology evaluates readability models on WeeBit and Newsela using alternative label treatments. WeeBit is downsampled for one setting, while the labeling approaches vary in how they represent readability levels.
3.1.1 WeeBit
- WeeBit classes were undersampled to 625 documents each to form the downsampled corpus, followed by preprocessing to remove extraneous material.The corpus combines WeeklyReader and BBC Bitesize documents spanning intended reader ages 7 to 16.Source: §3.1.1 · p. 2
3.1.2 Newsela
- Newsela contains 1,911 articles rewritten up to four times for different reading levels, with grade labels from grade 2 to grade 12.The shared topics across reading levels make Newsela more challenging to label than WeeBit, according to the passage.Source: §3.1.2 · pp. 2–3
3.1.3 Labeling Approaches
- The study compares classification, age regression, and ordered class regression to test ways of representing ordinal readability labels.Age regression uses mean reader ages; ordered class regression assigns equidistant integers, preserving order but not relative difficulty differences. Final inferences use classification for comparison with prior work.Source: §3.1.3 · p. 3
3.2 Features
The feature set combines established readability measures with new syntactic features focused on ambiguity and diversity. The novel measures include parse-probability variation and differences between sentence-level and document-level part-of-speech distributions.
3.2 Features
- The study adds features focused on syntactic ambiguity and syntactic diversity, aspects it describes as underused in readability research.These new features are used alongside existing linguistic features; future work could test their individual effects.Source: §3.2 · p. 3, §3.2.2 · p. 4
3.2.1 Existing Features
- Existing features span traditional word-length formulas, normalized part-of-speech and constituent counts, syntactic measures, and word characteristics from linguistic databases.The set also includes features such as constituent lengths and syntactic-tree heights, plus lexical properties and age-of-acquisition ratings.Source: §3.2.1 · pp. 3–4
3.2.2 Novel Syntactic Features
- Parse deviation uses the standard deviation of the x most probable parse log probabilities, using all valid parses when fewer than x exist.The raw number of parses is avoided because it is highly sensitive to sentence length; large x can also create low-probability tails that increase standard deviation.Source: §3.2.2 · p. 4
- PDMx measures the largest parse log probability minus the mean for the x most probable parses, using all valid parses when fewer than x exist.The paper selected PDM10, PD10, and PD2 to balance parse investigation against noise from implausible parses.Source: §3.2.2 · p. 4
- POSDdev measures grammatical heterogeneity as the standard deviation of a document’s part-of-speech count distribution.Syntactic uniqueness is measured using Kullback-Leibler divergence between sentence and document POS count distributions.Source: §3.2.2 · pp. 4–5
3.3 Models
The paper evaluates models ranging from simple classifiers to neural architectures, including transformers and hierarchical attention networks. The neural models provide the outputs later used alongside linguistic features.
SVMs, Linear Models, and Logistic Regression
- The experiments also evaluated SVMs, a linear support vector classifier, and logistic regression, with little optimization for the latter two classifiers.The SVM used Scikit-Learn, with hyperparameter optimization following Hsu et al. (2003); logistic regression used the SAG optimizer.Source: §3.3 · p. 5
CNN
- A CNN based on Kim (2014) was included because of its demonstrated performance on sentence classification.The implementation used Keras, TensorFlow, and Magpie.Source: §3.3 · p. 5
Transformer
- The transformer uses attention to focus on input parts and adds a linear output layer for the desired task; this paper uses BERT.BERT is pretrained on BooksCorpus and English Wikipedia, then fine-tuned on a readability corpus.Source: §3.3 · p. 5
HAN
- The hierarchical attention network uses bidirectional RNNs and separate attention mechanisms for words within sentences and sentences within documents.Its hierarchy is intended to reflect document structure; the implementation follows the original architecture.Source: §3.3 · p. 5
3.4 Incorporating Linguistic Features with Neural Models
The authors ask whether linguistic features can improve neural readability models that may fail to generalize across corpora. They combine neural outputs with linguistic features and use a simpler classifier for the final prediction.
Motivation
- Neural models may fail to generalize across readability corpora, motivating the question of whether linguistic features can improve their performance.Prior BERT results cited in the paper differ between WeeBit and Newsela; the authors hypothesize that linguistic features may supply useful inductive bias.Source: §3.4 · pp. 5–6
Combining Features
- The authors use a neural model’s single numerical output as a feature, join it with linguistic features, and feed the combined features to a simpler model such as an SVM.SVMs were chosen for their simplicity and frequent use in integrating numerical features.Source: §3.4 · p. 6
- For CNNs, the authors tested label approaches including grade classes and age regression, but used simple classification for other model types.They restricted intermediary results for those other models because the approaches produced insubstantial differences in final inferences.Source: §3.4 · p. 6
3.5 Training and Evaluation Details
Experiments used five-fold cross-validation and reported weighted or macro F1. The two result tables cover Newsela and downsampled WeeBit, while the paper also evaluates the WeeBit corpus.
Training
- All experiments used 5-fold cross-validation, and neural models were trained with Adam using model-specific learning rates and epoch counts.CNN, HAN, and transformer learning rates were 10^-3, 10^-4, and 2^-5; they were trained for 30, 20, and 3 epochs, respectively.Source: §3.5 · p. 6
Evaluation
- Weighted F1 weights each class by its sample count, whereas macro F1 gives each class equal weight.Weighted F1 therefore prioritizes common classes; macro F1 prioritizes smaller classes relative to their sample counts.Source: §3.5 · p. 6
Results Tables
- The supplied caption identifies Table 1 as presenting the top 10 performing model results for transformer and CNN models on the Newsela corpus.Source: Table 1 · p. 6
4 Results
Across two readability corpora, linguistic features generally add little to the strongest deep learning models when training data is sufficient, with a limited benefit on very small training subsets. Results vary by corpus and model, and the authors suggest that deep models may already represent relevant linguistic features.
4 Results
- Given sufficient data, linguistic features provide little to no benefit over independent deep learning models across the two corpora.The authors describe these results as covering a portion of the approaches tested; full results are in appendix B.Source: §4 · p. 6
4.1 Newsela Experiments
- On Newsela, the top-performing models did not use linguistic features, although feature-augmented transformers surpassed the HAN; the transformer’s weighted F1 improved by 0.22.The authors suggest the features may capture information transformers miss or lack data to learn, including non-semantic qualities such as syntax.Source: §4.1 · pp. 6–7
4.2 WeeBit Experiments
- Downsampled and full WeeBit results were largely comparable in raw scores and model rankings.The authors analyzed the downsampled and full datasets separately.Source: §4.2 · p. 7
4.2.1 Downsampled WeeBit Experiments
- On downsampled WeeBit, the best-performing model was a transformer without linguistic features, while word-type models also performed near the top.The authors note that word-type models have no access to word order and interpret their performance as evidence that word-type features provide substantial readability information; Table 2 presents selected top results.Source: §4.2.1 · p. 7, Table 2 · p. 7
4.2.2 Full WeeBit Experiments
- On full imbalanced WeeBit, model rankings changed little, and the feature-augmented SVM’s advantage over the transformer was under 0.005.The authors say this difference is extremely small and not likely to be statistically significant; gains may not hold when documents do not match the dataset distribution.Source: §4.2.2 · p. 7
4.3 Effects of Training Set Size
- Linguistic features helped CNNs only on training subsets smaller than 200 documents; above that size, performance changes were insubstantial.The trials held the test set constant while varying training-set size; Figure 1 shows performance differences across training sizes.Source: §4.3 · p. 8, Figure 1 · p. 8
4.4 Effects of Linguistic Features
- The authors conclude that, within the available corpora, model complexity, and model structures, linguistic features add no information beyond what state-of-the-art models have learned.They suggest the models may already represent those features and propose probing model representations; more diverse or consistently labeled corpora could change the value of linguistic features.Source: §4.4 · p. 8
5 Conclusion
The paper evaluates linguistic features alongside deep learning models on two readability corpora, finding that feature augmentation does not generally improve deep-learning performance. The authors qualify this conclusion by noting limitations in corpus construction and propose further study of model representations.
5 Conclusion
- The study examines whether linguistic features improve state-of-the-art models, using features focused on syntactic properties ignored by existing features.The features were incorporated into several model types and evaluated on two distinct corpora.Source: §5 · p. 8
- Linguistic features occasionally improved performance, particularly with smaller training sets, but the models did not achieve state-of-the-art performance.The evaluation covered two distinct corpora with different readability-assessment challenges.Source: §5 · pp. 8–9
- Because linguistic features generally did not improve deep models, the authors suggest those models may already capture features useful for readability assessment.They propose probing model representations to investigate this possibility.Source: §5 · p. 9
- Ambiguity in corpus construction limits assessment of label consistency and validity, so state-of-the-art models may already be near the maximum possible performance.The authors call for rigorously and consistently constructed readability corpora; averaging multiple labelers is one suggested way accuracy could improve.Source: §5 · p. 9
A Feature Definitions
The appendix organizes feature definitions into novel syntactic, existing syntactic-parse-based, POS-tag-based, psycholinguistic, and traditional categories. It also specifies conventions for ratios and several terms used in the feature definitions.
A Feature Definitions
- Undefined ratios are treated as zero; complex nominals follow Vajjala and Meurers (2012), polysyllabic means more than two syllables, and long words have at least seven characters.Age-of-acquisition rating norms are described in Kuperman et al. (2012).Source: A Feature Definitions · p. 11
- The appendix lists novel syntactic features alongside existing syntactic-parse-based and POS-tag-based feature definitions.The supplied table titles identify these categories but do not provide their individual feature contents.Source: Table 4 · p. 11, Table 5 · p. 12, Table 6 · p. 13
- Separate tables cover psycholinguistic and traditional feature definitions.The supplied titles identify the categories without detailing their entries.Source: Table 7 · p. 13, Table 8 · p. 14, Table 9 · p. 14
B Full Model Results
The full-model results section includes tables for downsampled WeeBit, WeeBit, and Newsela, with the listed model results sorted by weighted F1. The supplied passages do not establish comparative winners across these tables.
B Full Model Results
- The WeeBit downsampled model-results table is sorted by weighted F1.The supplied passages do not support identifying a winning model or assigning the listed values to particular metrics.Source: Table 10 · p. 15
- The WeeBit model-results table is sorted by weighted F1.The supplied passages do not support identifying a winning model or assigning the listed values to particular metrics.Source: Table 11 · p. 16
- The Newsela model-results table is sorted by weighted F1.The supplied passages do not support identifying a winning model or assigning the listed values to particular metrics.Source: Table 12 · p. 17