Source-linked AI summary
Supervised and Unsupervised Neural Approaches to Text Readability
Matej Martinc, Senja Pollak, Marko Robnik-Šikonja
TL;DR
Readability prediction needs methods that reduce reliance on scarce labels and manually engineered features while handling variation across languages and datasets. The paper develops unsupervised neural-language-model and supervised neural-classification approaches, finding that neural methods are viable but have dataset-dependent strengths and important trade-offs.
Problem
Existing readability models rely on scarce labelled data or extensive feature engineering, and their features may transfer poorly across corpora and languages.
Method
The paper combines an unsupervised neural language-model measure with supervised recurrent, hierarchical-attention, and transfer-learning classifiers evaluated on English and Slovenian corpora.
Results
Neural classifiers can match or outperform state-of-the-art feature-engineered approaches, while unsupervised neural measures capture high-level properties but cannot compete with shallow lexical indicators alone.
Takeaways & Limitations
Neural approaches are viable for readability prediction, with architecture suitability depending on dataset specifics and RSRS enabling localization by training-set choice.
Takeaways & Limitations
BERT truncates documents longer than 512 byte-pair tokens, causing substantial information loss in the longer OneStopEnglish and Newsela datasets.
Abstract
from arXiv · showhide
We present a set of novel neural supervised and unsupervised approaches for determining the readability of documents. In the unsupervised setting, we leverage neural language models, whereas in the supervised setting, three different neural classification architectures are tested. We show that the proposed neural unsupervised approach is robust, transferable across languages and allows adaptation to a specific readability task and data set. By systematic comparison of several neural architectures on a number of benchmark and new labelled readability datasets in two languages, this study also offers a comprehensive analysis of different neural approaches to readability classification. We expose their strengths and weaknesses, compare their performance to current state-of-the-art classification approaches to readability, which in most cases still rely on extensive feature engineering, and propose possibilities for improvements.
1. Introduction
Readability research moved from simple lexical and syntactic formulas toward machine-learning and neural approaches, motivated by scarce labelled data and poor cross-corpus transfer. This study proposes an unsupervised neural language-model measure and evaluates three supervised neural architecture families across languages and datasets.
- Traditional readability formulas use statistical lexical and syntactic features, while newer approaches model classification, regression, or ranking with manually engineered attributes.
- Machine-learning readability models generally improve on traditional and cohesion-based methods but depend on scarce labelled datasets and features that transfer poorly across corpora and languages.
- Neural readability research remains limited, leaving uncertainty about which architectures suit distinct readability tasks and datasets.
- The study introduces an unsupervised neural-language-model approach requiring only domain texts, with a trainable and cross-language readability measure called RSRS.
- It also tests recurrent, hierarchical-attention, and transfer-learning architectures against state-of-the-art classifiers on four labelled readability corpora.
2. Related work
Related work spans traditional, cohesion, lexico-semantic, syntactic, and language-model features for automated readability measurement. Existing evidence finds that simple traditional indicators often remain competitive, while transfer across genres and languages is limited.
- Readability systems use traditional, discourse-cohesion, lexico-semantic, syntactic, and language-model feature groups.
- Traditional features: Traditional formulas estimate readability from word length, sentence length, word difficulty, education years, syllables, or polysyllabic-word counts.
- Traditional features: Higher GFI and FKGL values indicate greater educational requirements, whereas higher FRE values indicate more readable text.
- Traditional readability measures were developed mainly for English, and evidence indicates that genre-specific vocabulary can undermine cross-genre prediction.
- Discourse cohesion features: Cohesion features include anaphoric chains, entity density, lexical cohesion, and part-of-speech-based measures.
- Discourse cohesion features: A review of 65 discourse features found that they generally add little predictive power compared with traditional formulas or simple statistics such as sentence length.
- Lexico-semantic features: Lexico-semantic features measure vocabulary difficulty and variation through measures such as TTR, corrected TTR, n-grams, and word-list features.
- Syntactic features: Syntactic features represent grammatical complexity using parse-tree and grammatical-relation measurements.
3. Datasets
The study evaluates readability methods on four varied corpora spanning English and Slovenian, different genres, audiences, document lengths, and aligned versus unaligned classes. This diversity supports robustness analysis but limits broad conclusions across datasets, languages, and genres.
- The experiments use four labelled readability corpora: WeeBit, OneStopEnglish, Newsela, and Slovenian school books.
- Test corpora: WeeBit assigns articles to five age-targeted classes spanning readers aged 7–8 through 14–16.
- Test corpora: OneStopEnglish contains aligned beginner, intermediate, and advanced texts written for English-as-a-Second-Language learners.
- Test corpora: Newsela contains original English news articles with up to four manually rewritten simplified versions for each original.
- Test corpora: The Slovenian school-book corpus contains 3,639,665 words from 125 books spanning primary and secondary grades and sixteen subjects.
- Dataset differences: The datasets differ in language, semantic alignment, document length, genre, and target audience, including English news and educational texts alongside Slovenian school books.
- Dataset differences: This dataset diversity limits general conclusions across genres, languages, and datasets while enabling analysis of how dataset specifics affect predictor robustness.
- Dataset differences: Cross-dataset supervised experiments were omitted because similar readability scores do not guarantee comparable difficulty across corpora.
4. Unsupervised neural approach
The unsupervised approach uses neural language-model statistics to estimate readability, with RSRS weighting less readable words more heavily. Results show that perplexity alone is weak and dataset- or language-model-dependent, while RSRS combined with shallow indicators is more useful across datasets.
- Neural language model architectures: Neural language models provide richer contextual information than n-gram models, motivating their use for readability prediction.Their broader historical context is attributed to improved perplexity, and the paper reports no prior use of deep neural language models for readability determination.
- Ranked sentence readability score: RSRS ranks words by word negative log-likelihood and weights their contributions so that a few difficult words can reduce sentence readability more strongly.Out-of-vocabulary words receive doubled square-root rank weights, and document readability is computed by averaging sentence scores.
- Ranked sentence readability score: RSRS aims to capture high-level structural and semantic information through context-sensitive neural language-model statistics rather than relying only on traditional lexical and syntactic features.The method assumes that complex grammatical and lexical structures reduce language-model performance, increasing WNLL for words in such contexts.
- Experimental results: Perplexity measures generally performed worst, whereas TCN and RLM RSRS showed useful information when combined with shallow lexical sophistication indicators.The paper therefore distinguishes language-model statistics used alone from their combination with other readability predictors.
- Experimental results: BERT perplexity correlated negatively with readability on Newsela and Slovenian school books, with ρ of −0.673 and −0.563, but correlations were weak on WeeBit and OneStopEnglish.The authors relate the stronger correlations partly to BERT being trained mainly on adult-oriented corpora.
- Experimental results: A single general-corpus language model was not a viable unsupervised readability predictor for younger audiences or English learners according to the reported results.The authors state that further experiments and data are needed to identify the causes of discrepancies, including weak WeeBit correlations and language-resource differences.
- Experimental results: Training-corpus readability affected results: Wiki-simple and Wiki-balanced measures often outperformed Wiki-normal measures, but several expected corpus-specific patterns did not hold.On Newsela, RLM perplexity-balanced exceeded RLM perplexity-simple by 0.042, while TCN perplexity-simple exceeded TCN perplexity-balanced by 0.029.
- Experimental results: Correlation rankings varied substantially across genres and languages, with average sentence length ranking first on Newsela but twelfth on OneStopEnglish and seventh on Slovenian school books.GFI ranked second on the Slovenian school books corpus but much lower on the English corpora.
5. Supervised neural approach
The supervised approach evaluates RNN, HAN, and transfer-learning classifiers for readability across multiple corpora using common metrics and no corpus-specific parameter tuning. Performance varies by dataset: neural models can surpass feature-engineered baselines, but architecture choice depends on document properties, especially length and hierarchical structure.
- The study tests recurrent neural networks, hierarchical attention networks, and transfer-learning classifiers for supervised readability prediction.
- All corpora use the same classifier configuration without corpus-specific tuning, evaluated by accuracy, weighted precision, weighted recall, weighted F1-score, and QWK.
- WeeBit: 85.73% accuracy: BERT is the strongest WeeBit classifier, exceeding BiLSTM by about 8 percentage points and reaching 95.27% QWK.
- Newsela: 81.01% F1-score and 81.38% accuracy: HAN achieves the best Newsela result, while BERT is about 6 percentage points worse by accuracy and F1 criteria.
- OneStopEnglish: 78.72% accuracy: HAN is best on OneStopEnglish, while BiLSTM and BERT perform about 10 percentage points worse across accuracy, F1-score, and QWK.
- Slovenian school books: 52.19% F1-score: BiLSTM is best on the Slovenian school-books corpus, whereas BERT reaches about 45.45% F1-score and 41.57% accuracy.
- Overall findings: Neural approaches outperform feature-engineered state-of-the-art classifiers where comparisons are available, but gains are not substantial and the best architecture depends on the dataset.
6. Conclusion
The study finds that neural methods support both unsupervised and supervised readability prediction, while performance depends on data characteristics, training-set choice, and the trade-off between accuracy, interpretability, and computational cost.
- Neural language models can support unsupervised readability prediction, but combining neural and traditional indicators improves RSRS correlation with human scores.The unsupervised approach captures semantic information, yet shallow lexical indicators remain competitive.
- RSRS is adaptable across languages, and its behavior can be customized through the language-model training set.A 2.4 million-word Slovenian training corpus was sufficient to outperform traditional readability measures.
- The study’s Slovenian results vary by corpus and genre, so further datasets are needed to confirm how topic and genre affect readability-measure rankings.Average sentence length performed competitively on English but ranked poorly on the Slovenian school-book corpus.
- Neural classifiers can match or outperform state-of-the-art systems that rely on extensive feature engineering, but they reduce interpretability and explainability.This limitation is especially relevant in educational applications requiring reasons for readability judgments.
- Classifier performance varies across corpora, with document length identified as a major factor and computational gains requiring comparison against simpler baselines.On OneStopEnglish, HAN reached 78.72% accuracy versus 78.13% for a 155-feature classifier and 77.25% for generic n-grams.
- Future work targets cross-corpus, cross-genre, and cross-language transfer, neural ensembles, explainable predictions, and architectures that process longer documents.Longformer and Linformer are proposed for documents containing thousands of tokens.