PaperlayerView exact revision on arXiv

Source-linked AI summary

Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features

Bruce W. Lee, Yoo Sung Jang, Jason Hyung-Jong Lee

arXiv:2109.12258v2cs.CLcs.AI

TL;DR

Readability assessment has relied heavily on handcrafted linguistic features, while transformers had shown limited advantage on smaller datasets. This paper develops advanced-semantic features and hybrid models combining handcrafted features, traditional ML, and transformers; its RoBERTa-RF-T1 model achieves 99% accuracy on OneStopEnglish, 20.3% above the previous SOTA.

  • Problem

    Handcrafted features lack coverage, and transformers had shown little advantage over traditional ML on smaller readability datasets, amid a shortage of large public datasets.

    Source: §1 · p. 1, §4.1 · p. 5

  • Method

    The authors search across transformers and traditional ML, develop advanced-semantic features, and combine handcrafted features with transformers in hybrid models.

    Source: §1 · pp. 1–2

  • Results

    Hybrid models achieved SOTA on WeeBit and OneStopEnglish; RoBERTa-RF-T1 reached 99% accuracy on OneStopEnglish, 20.3% above previous SOTA.

    Source: §1 · p. 2, §4.4 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that handcrafted features help model performance on smaller datasets and add orthogonal information to the model.

    Source: §1 · p. 2, §6 · p. 9

    Readability assessment often relies on one dataset for train/test/validation, making domain overfitting a common problem; broader general-use models remain challenging without a way to connect datasets or a new large public dataset.

    Source: §5 · p. 9

Abstract

from arXiv · show

We report two essential improvements in readability assessment: 1. three novel features in advanced semantics and 2. the timely evidence that traditional ML models (e.g. Random Forest, using handcrafted features) can combine with transformers (e.g. RoBERTa) to augment model performance. First, we explore suitable transformers and traditional ML models. Then, we extract 255 handcrafted linguistic features using self-developed extraction software. Finally, we assemble those to create several hybrid models, achieving state-of-the-art (SOTA) accuracy on popular datasets in readability assessment. The use of handcrafted features help model performance on smaller datasets. Notably, our RoBERTA-RF-T1 hybrid achieves the near-perfect classification accuracy of 99%, a 20.3% increase from the previous SOTA.

1 Introduction

The introduction frames readability assessment around handcrafted linguistic features, then presents advanced-semantic features and transformer–traditional-ML hybrids as contributions aimed at gaps in feature coverage and performance on smaller datasets.

1 Introduction

  • Readability assessment has largely relied on handcrafted linguistic features, from simple syllable counts to semantic complexity.The introduction describes traditional ML systems as explainable and competitive with modern neural models.Source: §1 · p. 1
  • Large public RA datasets are scarce, while transformers had shown limited advantage over traditional ML on smaller datasets such as OneStopEnglish.WeeBit is identified as a rare large public dataset, although it must be obtained directly from its authors.Source: §1 · p. 1
  • The authors target a scarcity of advanced-semantics research by representing texts as latent topics and calculating their probability distribution.They frame this as a way to investigate how deeper semantic structures relate to readability.Source: §1 · p. 2
  • 99% accuracy on OneStopEnglish from RoBERTa-RF-T1 is 20.3% higher than the previous SOTA, according to the introduction.The paper also reports SOTA results on WeeBit and OneStopEnglish.Source: §1 · p. 2
  • The study combines a broad search over transformers and traditional ML with 255 handcrafted features and hybrid models.The authors introduce LingFeat, an openly available Python toolkit for feature extraction.Source: §1 · pp. 1–2

2 Advanced Semantics

The section addresses the limited study of advanced semantics in readability assessment by developing passage-level LDA topic-distribution features. It evaluates semantic richness, clarity, and noise as measures of deeper meaning structure.

2.1 Overview

  • Advanced semantics had received little study beyond word-level complexity, motivating passage-level features that add orthogonal information to readability models.The authors note that human-labeled evaluation may favor recognizable surface features; they also caution that world knowledge may not directly indicate difficulty and word-level substitutes may overlap.Source: §2.1 · p. 2

2.2 Approach

  • LDA topic-probability distributions are used to represent deeper text meaning, with topic models trainable on different data to account for domain or world knowledge.The method sorts topic probabilities and measures their distribution rather than the identities of the topics.Source: §2.2 · p. 3

2.3 Semantic Richness

  • Semantic richness measures how broadly populated the passage’s topics are, combining topic count and probabilities while weighting higher topic indices.The proposed passage-level measure adapts a co-occurrence concept to LDA and increases faster as the number of topics grows.Source: §2.3 · p. 3

2.4 Semantic Clarity

  • Semantic clarity measures how concentrated the topic-probability distribution is around its largest topic, indicating how identifiable the main topic is.The authors use an alternative to standard statistical skewness because it did not capture the anticipated trends.Source: §2.4 · p. 3

2.5 Semantic Noise

  • Semantic noise measures the tail weight of sorted topic probabilities, with more leptokurtic distributions indicating higher noise than comparable platykurtic ones.The measure uses the kurtosis equation under Fisher’s definition.Source: §2.5 · p. 3

3 Covered Features

The paper covers 255 linguistic features grouped into 14 subgroups, spanning advanced semantics, discourse, syntax, lexico-semantics, and shallow traditional measures.

3 Covered Features

  • The feature inventory contains 255 linguistic features across 14 subgroups, with added variations for existing features and dependencies documented separately.The full feature list, codes, and definitions are provided in appendix B; external dependencies are reported in appendix D.Source: §3 · p. 4

3.1 Advanced Semantic Features (AdSem)

  • Advanced semantic features use LDA models trained on Wikipedia, WeeBit, or OneStopEnglish data, with four topic counts and four measures extracted per model.The topic counts are 50, 100, 150, and 200; the measures are richness, clarity, noise, and total discovered topics.Source: §3.1 · p. 4

3.2 Discourse-Based Features (Disco)

  • Discourse features include entity counts and entity-grid transition ratios, alongside local coherence scores.The entity-grid features use 16 transition patterns; the local coherence implementation follows cited prior work.Source: §3.2 · p. 4

3.3 Syntactic Features (Synta)

  • Syntactic features cover phrase counts, parse-tree structure, and Universal POS-tag counts, alongside the rationale that syntactic complexity relates to processing time and text complexity.The tree features use constituency-parser output, with NLTK used for final calculations.Source: §3.3 · p. 4

3.4 Lexico-Semantic Features (LxSem)

  • Lexico-semantic features measure category variation, lexical richness, age of acquisition, and word familiarity using established feature variants and resources.The measures include five TTR variations, MTLD with default TTR 0.72, age-of-acquisition features, and SubtlexUS frequency.Source: §3.4 · p. 4

3.5 Shallow Traditional Features (ShaTr)

  • Shallow traditional features capture surface difficulty through token and syllable counts and established readability formulas.The formulas include Flesch-Kincaid Grade Level, Automated Readability, Gunning Fog, Smog Index, and Linsear Write.Source: §3.5 · p. 4

4 Hybrid Model

The paper combines transformer predictions with handcrafted linguistic features in a secondary traditional ML model, searching across model and feature-set combinations. The resulting hybrids set state-of-the-art results on the reported datasets, with gains varying by dataset and training-set size.

4.1 Overview

  • The hybrid appends a neural model’s soft-label predictions to handcrafted features, then trains a non-neural secondary predictor; the authors brute-force search four neural models, four traditional models, and 14 feature sets.The neural model generates softmax probabilities for readability classes, and reprediction aligns training and test data dimensions.Source: §4.1 · p. 5, §4.3 · p. 7

4.2.1 Datasets and Evaluation Setups

  • The evaluation covers WeeBit, OneStopEnglish, and Cambridge using accuracy, weighted F1, precision, recall, and QWK, averaged over stratified five-fold splits.WeeBit classes were downsampled to 625 passages each, Cambridge to 60 items per class, and OneStopEnglish required no downsampling.Source: §4.2.1 · p. 5

4.2.2 Search on Neural Model

  • RoBERTa and BART outperform BERT and XLNet on most metrics, including on OneStopEnglish, while the authors suggest pretraining input size may help explain differences.They searched hyperparameters on WeeBit and reused that configuration on other corpora without corpus-specific tweaking.Source: §4.2.2 · p. 6

4.2.3 Search on Non-Neural Model

  • Optimized non-neural models generally have lower accuracy than transformers, while advanced-semantic features perform well across measures and WorF and PsyF rank in the top seven across corpora.The feature search also finds that advanced semantics and discourse add distinct information, as indicated by performance changes.Source: §4.2.3 · p. 6, §4.2.4 · pp. 6–7

4.4 Hybrid Model Results

  • Hybrid models achieve SOTA on WeeBit (BART-RF-T1) and OneStopEnglish (RoBERTa-RF-T1); RoBERTa-RF-T1 reaches near-perfect accuracy of 99% on OneStopEnglish, a 20.3% increase from previous SOTA by HAN (Meng et al., 2020).On Cambridge, accuracy is 76.3% on a small dataset of only 60 items/class; Xia et al. (2016) uses extra data.Source: §4.4 · p. 7
  • Hybrid gains vary by dataset: Cambridge improves by about 10% in accuracy on average, WeeBit gains little over transformer-only results, and BART benefits little on WeeBit and OneStopEnglish.The authors say choosing an effective combination requires testing several combinations, and that hybrids tend to improve already-good models rather than fair-performing ones.Source: §4.4 · p. 8

4.6 Why Was Our BERT Better?

  • The authors report that their BERT reaches 89.3% accuracy on WeeBit versus 83.9% for Deutsch et al. (2020) and 85.7% for Martinc et al. (2021), attributing differences to hyperparameters.They also omit ReadNet from Table 5 because its Cambridge score and WeeBit result are not directly comparable under the reported dataset conditions.Source: §4.6 · p. 8, §4.5 · p. 8

4.7 Data Size Effect

  • 85.8% accuracy on WeeBit with 750 passages lets BERT-GB-T1 match the prior 85.7% SOTA using 30% of the original training data.Across sampled training sizes, the hybrid outperforms BERT by 0.01–0.05, with the gap shrinking as training data increases.Source: Figure 3 · p. 8, §4.7 · p. 9

5 Cross Domain Evaluation

The authors did not conduct cross-domain evaluation, judging it out of scope for this study. They identify dataset-specific labeling and domain overfitting as challenges to general-use readability assessment.

5 Cross Domain Evaluation

  • Cross-domain evaluation was left for future work because a thorough study across at least two datasets was out of scope.The authors say they had not yet reached the point of seriously addressing cross-domain evaluation because in-domain accuracies still had room to improve.Source: §5 · p. 9
  • Dataset class labels vary among expert-labeled datasets, so aligning them for cross-domain evaluation requires substantial work.The authors mention class mapping as one such effort.Source: §5 · p. 9
  • Domain overfitting is common when readability assessment uses one dataset for training, testing, and validation, making general-use models challenging to develop.The authors say this remains challenging without a methodology to connect datasets or a new large public dataset.Source: §5 · p. 9

6 Conclusion

The conclusion reports a 99% OneStopEnglish result for RoBERTA-RF-T1 and a WeeBit SOTA result for BERT-GB-T1 using only 30% of the original training data. It also states that the new advanced semantic features add orthogonal information.

6 Conclusion

  • 99% accuracy on OneStopEnglish was achieved by RoBERTA-RF-T1, while BERT-GB-T1 beat the previous WeeBit SOTA using only 30% of the original train data.The authors also report that the new advanced semantic features add orthogonal information to the model.Source: §6 · p. 9
  • The authors report that their new advanced semantic features add orthogonal information to the model.The conclusion presents this alongside the hybrid models' SOTA results.Source: §6 · p. 9
  • The conclusion states that the created hybrid models achieved SOTA results.The reported examples are RoBERTA-RF-T1 on OneStopEnglish and BERT-GB-T1 on WeeBit.Source: §6 · p. 9

A Trend, Advanced Semantic Features

The section illustrates how semantic richness, clarity, and noise vary across example probability lists. It also explains how those lists are identified and how Table 7 presents the trends.

A Trend, Advanced Semantic Features

  • Table 7 presents trends for richness, clarity, and noise, with values scaled by 10 and L-M and H-M meaning Low-Mid and High-Mid.The table's lists are numbered 1 through 5 from top to bottom; “out” denotes raw output from equations.Source: Table 7 · p. 13, A Trend, Advanced Semantic Features · p. 13
  • The sorted probability lists are shown in Figure 1, while Table 7 numbers each list from top to bottom.The text identifies “out” as raw output from equations.Source: A Trend, Advanced Semantic Features · p. 13
  • Semantic richness increases with longer lists of meaningful topics; for equal-length lists, the one with more meaningful topics has greater richness.The examples also show a shorter list with more meaningful topics scoring higher than a longer list with fewer meaningful topics.Source: A Trend, Advanced Semantic Features · p. 13
  • Semantic clarity rewards short probability lists where the maximum probability stands out from the other elements.The examples contrast lists with the same maximum and show that repetition of the top probability makes the main topic harder to identify.Source: A Trend, Advanced Semantic Features · p. 14
  • Semantic noise increases for longer lists containing more extraneous, low-probability topics.In the examples, list 5 has the least semantic noise.Source: A Trend, Advanced Semantic Features · p. 14

B Features, Codes, and Definitions

The appendix catalogs knowledge and linguistic feature tables, sets conventions for feature codes, and lists software and models used for linguistic analysis.

B Features, Codes, and Definitions

  • The feature tables include Wikipedia, WeeBit, and OneStopEnglish knowledge features, alongside entity-density and entity-grid feature groups.Additional tables cover phrasal, tree-structural, part-of-speech, variation-ratio, type-token-ratio, psycholinguistic, word-familiarity, and shallow features.Source: Table 8 · p. 14, Table 9 · p. 14, Table 10 · p. 14

C Rules Behind Feature Codes

  • Feature codes distinguish count-based features from score-based features, with count codes ending in C and using prefixes for total, sentence average, token average, or ratio.Count-based features use the format xx_xxxxx_C; score-based features use xxxxxxx_S.Source: C Rules Behind Feature Codes · p. 16
  • Feature-code explanations use capital letters to mark new words, followed by lowercase letters from the same word.The stated examples abbreviate Coleman Liau as ColeLia and Age of Acquisition Kuperman of words as AAKuW.Source: C Rules Behind Feature Codes · p. 16

D Scientific Artifacts

  • The authors use Online LDA implemented with Gensim v4.0 and spaCy v3.08 with the en_core_web_sm pretrained model for general NLP tasks.Constituency parsing uses the CRF parser in SuPar v1.0.Source: D Scientific Artifacts · pp. 16–17

D.1 Transformers

The study evaluates transformer and non-neural classifiers, preprocesses text for handcrafted-feature extraction, and tunes both model families. Transformer training uses a shared setup with model-specific learning rates.

D.1 Transformers

  • BERT, RoBERTa, BART, and XLNet are the four transformers used in the study.They are listed as HuggingFace transformer models.Source: §D.1 · p. 17

D.2 Non-Neural Models

  • The non-neural models include support vector, random forest, logistic regression, and gradient-boosting classifiers.The study identifies the first three as SciKit-Learn models and gradient boosting as an XGBoost model.Source: §D.2 · p. 17

E Preprocessing

  • Handcrafted-feature extraction preprocesses text by removing special characters and short words, lowercasing, tokenizing, and removing default NLTK stopwords.The authors say these steps are used only during handcrafted-feature extraction.Source: E Preprocessing · p. 17

F.1 Machine Learning Models

  • Non-neural models undergo randomized hyperparameter search followed by grid search, with model-specific parameters tuned and other parameters left at defaults.The described parameters include regularization, kernels, tree settings, learning rate, and optimization choices.Source: §F.1 · p. 17, Table 23 · p. 17

F.2 Transformers

  • All tested transformers use AdamW, a linear scheduler, 10% warmup, batch size 8, and 3 epochs.BERT uses a 2e-5 learning rate; the other three transformers use 3e-5.Source: §F.2 · p. 17

G Feature Combinations

The feature sets group linguistic properties by type, and the reported comparisons associate advanced semantic and discourse features with distinctive information and better performance when more linguistic branches are included.

G Feature Combinations

  • Performance decreases from T1 to T2 and T3, which the authors cite as evidence that advanced semantic and discourse features add distinctive information.F1, precision, recall, and QWK follow the same trend.Source: G Feature Combinations · p. 18
  • More linguistic branches generally indicate better performance.The passage describes this as a general trend rather than a universal result.Source: G Feature Combinations · p. 18
  • The feature sets are defined by aims: T captures linguistic properties, H high-level properties, L surface-level properties, E external-data features, and P performance-based features.The passage notes that the reported accuracy is on WeeBit.Source: G Feature Combinations · pp. 17–18, Table 24 · p. 17

H Transformers Training Time

The paper reports average transformer training times per fold under a specified dataset and hardware setup, and presents LingFeat as a Python toolkit for extracting handcrafted linguistic features.

H Transformers Training Time

  • Transformer training times are averages per fold, using 80% of the full dataset for training on an NVIDIA Tesla V100 GPU.The reported order is BERT, RoBERTa, XLNet, and BART; this passage provides no time values.Source: H Transformers Training Time · p. 18

I LingFeat

  • LingFeat is a Python NLP feature-extraction package that extracts 255 handcrafted linguistic features from English string input.The package is available on PyPI and GitHub.Source: I LingFeat · p. 18
  • LingFeat features are accessed through subgroups rather than individually, and the described code applies to version 1.0.The authors cite the scarcity of large-scale handcrafted-feature extraction toolkits in readability assessment as a reason for presenting LingFeat as a contribution.Source: I LingFeat · p. 18
Loading 2109.12258v2…