Source-linked AI summary
Text Readability Assessment for Second Language Learners
Menglin Xia, Ekaterina Kochmar, Ted Briscoe
TL;DR
The paper tackles L2 readability assessment despite limited level-annotated learner data. It collects CEFR-graded learner texts and adapts native-corpus models through generalization, domain adaptation, and self-training. The best learner-text model achieves ACC=0.797 and PCC=0.938.
Problem
L2 readability assessment lacks significantly sized, properly annotated learner data, while native and L2 readers can perceive text difficulty differently.
Method
The paper collects a CEFR-graded L2 dataset and adapts models trained on native corpora using generalization, domain adaptation, and self-training.
Results
ACC=0.797 and PCC=0.938 are achieved by the best learner-text model using native data as unlabelled training data for classification.
Takeaways & Limitations
Learner-tailored data and methods for exploiting native corpora support readability estimation when L2-labelled data are limited.
Takeaways & Limitations
Future work must improve the L2 readability framework and identify feature sets that generalize well to unseen text.
Abstract
from arXiv · showhide
This paper addresses the task of readability assessment for the texts aimed at second language (L2) learners. One of the major challenges in this task is the lack of significantly sized level-annotated data. For the present work, we collected a dataset of CEFR-graded texts tailored for learners of English as an L2 and investigated text readability assessment for both native and L2 learners. We applied a generalization method to adapt models trained on larger native corpora to estimate text readability for learners, and explored domain adaptation and self-learning techniques to make use of the native data to improve system performance on the limited L2 data. In our experiments, the best performing model for readability on learner texts achieves an accuracy of 0.797 and PCC of $0.938$.
1 Introduction
L2 readability assessment is needed because texts can be difficult for learners in ways that differ from native readers, while suitable annotated data remain scarce. The paper addresses these challenges by collecting learner-tailored texts and adapting native-data methods.
- Automated readability analysis can support selecting appropriate reading materials and assessing reading ability in second-language education.
- Readability depends on textual, reader, and contextual variables, including lexical and syntactic complexity, organization, familiarity, and interest.
- L2 learners may perceive text comprehensibility differently from native speakers, with grammatical aspects often contributing more to their understanding.
- L2 readability assessment lacks significantly sized, properly annotated datasets, despite the availability of larger native-speaker corpora.
- The paper compiles a level-graded dataset tailored to L2 learners and investigates methods for using existing native corpora.
- 0.797 accuracy and 0.938 PCC were achieved by the best-performing readability model in the experiments.
2 Related Work
Earlier readability research mainly targets native readers and uses statistical, discourse, feature-based, and machine-learning approaches. L2-focused work is emerging, but progress remains constrained by limited well-labelled learner data.
- Language modelling, naïve Bayes, discourse features, classification, and regression have all been used to estimate text readability.
- Changing feature sets has produced larger improvements than changing the machine-learning framework in prior work.
- Most previous readability studies predict difficulty for native readers rather than L2 learners.
- L2 readability studies have examined grammatical features and learner-oriented measures, with grammatical features reported as more important for L2 than L1 prediction.
- L2 readability research is also emerging for French, Portuguese, and Swedish, often using textbook levels assigned by publishers or instructors.
- 0.497 salience is not a paper result; the supplied passage states that L2 readability research remains in an early stage because well-labelled learner data are scarce.
3 Data
The paper combines a cleaned native-reader corpus with a Cambridge English Exams dataset whose A2–C2 levels are assigned for L2 learners. It notes that exam-task effects on difficulty are not addressed.
- The WeeBit corpus was cleaned by re-extracting text from raw HTML and discarding documents without proper reading passages.
- Native age-specific corpora use annotations based on criteria that differ from those relevant to L2 readers.
- The Cambridge dataset contains reading passages from KET, PET, FCE, CAE, and CPE exams.
- A2–C2 CEFR levels assigned to Cambridge reading papers are treated as L2 reading-difficulty levels.
- The Cambridge Exams data provide language-testing annotations considered more appropriate for L2 readability assessment than native-reader levels.
- The CEFR defines six increasing proficiency levels from A1 through C2, while this dataset covers A2–C2.
- The dataset ignores possible effects of task type on reading difficulty.
4 Readability Measures
The paper evaluates readability using diverse linguistic feature groups and compares classification with pairwise ranking on the WeeBit corpus.
- Feature groups: Feature engineering combines traditional surface metrics, syntactic complexity, language modeling, discourse, and lexico-semantic measures.The feature set includes sentence and word statistics, parser-derived complexity, n-grams, cohesion indicators, and vocabulary-based measures.
- Lexico-semantic features: EVP features estimate lexical complexity from the proportion of words assigned to CEFR levels for L2 learners.The English Vocabulary Profile records learner-acquired words and phrases by CEFR level and is incorporated into the feature set.
- Discourse features: Entity-based discourse features represent cohesion through entity densities and grammatical-role distributions across sentences.The entity grid encodes whether each entity is a subject, object, neither, or absent in each sentence.
- Results: 0.803 ACC and 0.900 PCC are achieved on WeeBit with all features and a classification model.Ablation tests found that all feature sets contributed to overall model performance.
- Model comparison: Classification and ranking performance varies across feature sets, with neither model consistently outperforming the other.When all features are combined, classification outperforms ranking on the same dataset.
5 Readability Assessment on L2 Data
Because native-reader and L2 texts differ in perceived difficulty, native-trained models generalize unevenly to learner data. Ranking, score-to-CEFR mapping, EasyAdapt, and self-training were evaluated to improve readability estimation with limited L2 data.
- 5.1 Generalization Experiment: Native-reader texts can mismatch L2 difficulty levels, limiting direct transfer to learner readability assessment.The WeeBit corpus targets native-speaking children, whereas Cambridge Exams texts are annotated for L2 learners.
- 5.1 Generalization Experiment: Classification accuracy dropped greatly on L2 data, while correlation remained relatively high because many texts were assigned to higher native-corpus levels.The mismatch reflects Cambridge Exams texts being more difficult on average than WeeBit texts.
- 5.1 Generalization Experiment: Ranking preserved relatively high pairwise accuracy and correlation when transferred to L2 data, unlike classification.Ranking captures relative document difficulty rather than assigning documents directly to differently annotated levels.
- 5.2 Mapping Ranking Scores to CEFR Levels: Ranking was more accurate than classification for unseen learner texts, motivating a function that maps ranking scores into CEFR levels.The mapping was evaluated with five-fold cross-validation and included regression, separation-boundary, and classification approaches.
- 5.2 Mapping Ranking Scores to CEFR Levels: Classification-based mapping achieved the best accuracy among the three mapping groups, while polynomial mapping also produced strong PCC.All three mapping approaches significantly outperformed naive native-to-L2 classification generalization at p<0.05.
- 5.3 Domain Adaptation from Native to L2 Data: EasyAdapt ranking significantly outperformed naive generalization but did not surpass a model trained directly on L2 data.EasyAdapt augments features to capture general and domain-specific information while restricting ranking pairs to the same domain.
- 5.4 Using Self-training to Enhance the Classification Model: Self-training significantly improved L2 readability estimation by adding selected native data, although the covered difficulty range may affect performance.The best result used K=10 for 9 iterations, adding 270 texts.
6 Conclusions and Future Work
The study evaluates readability assessment for native and L2 learners, adapting native-data models and testing domain adaptation and self-training. Self-training produces the strongest reported L2 result, while future work targets better generalization to unseen text.
- Results: The ranking model reaches ACC=0.622 and PCC=0.864 when mapping native-trained predictions to CEFR levels on unseen text.It outperforms naive classification generalization, which reaches ACC=0.233 and PCC=0.730.
- Results: On the original datasets, the linear SVM reaches ACC=0.803 and PCC=0.900 for native data and ACC=0.785 and PCC=0.924 for L2 data.
- Results: ACC=0.797 and PCC=0.938 are achieved by self-training a classification model with native data as unlabeled training data.This is the best reported result for the native-to-L2 setting.
- Results: ACC=0.707 and PCC=0.899 are achieved by applying EasyAdapt to ranking with native data as the source domain and L2 data as the target domain.
- Future Work: Future work will improve the L2 readability-assessment framework and identify a feature set that generalizes well to unseen text.