ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment

• Figure 1 presents language distributions by domain and example sentences with human readability levels from 1, easiest, to 6, hardest.Source: Figure 1 · p. 1
• Existing readability datasets are concentrated in Wikipedia or news and lack the domain and language-family diversity required to evaluate model generalization.Source: §1 · pp. 1–2
• README++ contains 9,757 human-annotated sentences from 112 data sources in Arabic, English, French, Hindi, and Russian, rated on the CEFR scale.The dataset targets readability assessment for second-language learners.Source: §1 · p. 2
• The benchmark evaluates monolingual and multilingual LMs in supervised, unsupervised, and few-shot prompting settings.Its diversity supports analysis of few-shot prompting and shortcomings in unsupervised methods.Source: §1 · p. 2
• LMs fine-tuned on README++ perform better on unseen domains and transfer better from English to Arabic, French, Hindi, Russian, Italian, and German than models trained on previous datasets.Source: §1 · p. 2
• README++ covers sentence lengths and readability levels more broadly than CEFR-SP in its English portion.Source: Figure 2 · p. 2

• Many prior datasets provide document-level labels, which are suboptimal for sentence-level readability compared with individual-sentence ground-truth labels.Source: §2 · p. 2
• Existing sentence-level datasets are mostly limited to high-resource English and European Latin-script languages and use few data sources.Source: §2 · p. 3
• README++ covers 112 data sources and five languages with sentence-level CEFR annotations, increasing domain and language diversity relative to prior resources.Source: §2 · p. 3, Table 2 · p. 3
• Multilingual readability studies commonly use transformer fine-tuning, but training data is often unavailable beyond a few high-resource languages.Source: §2 · p. 3
• README++ benchmarks supervised, unsupervised, and few-shot methods across Arabic, Latin, Brahmic, and Cyrillic scripts, with English-trained models transferring better to six target languages than models trained on previous datasets.Source: §2 · p. 3

3 Constructing README++ Corpus

• README++ samples 9,757 sentences from 112 sources across five languages to support multilingual, cross-lingual, and cross-domain experiments.The language counts are 1,945 Arabic, 1,669 French, 2,861 English, 1,524 Hindi, and 1,758 Russian sentences.Source: §3 · p. 3

3.1 Data Collection

• Data collection combines website extraction, PDF extraction, existing-dataset sampling, and manual sentence collection, followed by quality filtering.Sources are sampled by domain, and sentences containing low-quality, toxic, hateful, or offensive language are removed.Source: §3.1 · p. 4
• Up to three preceding sentences are collected as optional context because sampled sentences may occur within paragraph bodies.The authors also evaluate providing context to language models during fine-tuning.Source: §3.1 · p. 4

3.2 Readability Annotation

• CEFR annotation maps readability to six learner-proficiency levels from A1 through C2, using can-do descriptors to guide annotators.Source: §3.2 · p. 4
• Rank-and-rate annotation first orders batches of five sentences by readability, then rates each sentence individually on the six-point CEFR scale.The comparative procedure is intended to reduce indistinguishable ratings and subjectivity; pilot studies found higher agreement than rating alone.Source: §3.2 · p. 4
• Three annotators produce the final rating by majority voting, with high agreement reported using Krippendorff’s alpha and Pearson correlation.Source: Table 3 · p. 4, §3.2 · p. 5

4 Benchmarking Experiments

• The benchmark uses a consistent 60/10/30% per-domain split and evaluates fine-tuned multilingual and monolingual models alongside five-shot prompting models.Source: §4 · p. 5, §4.1 · p. 5
• The corpus spans diverse domains, readability levels, and sentence lengths, creating a testbed for evaluating robustness across textual variation.Source: §4 · p. 5, Figure 3 · p. 5

4.1.1 Results

• Fine-tuned models reached Pearson correlations of 0.7-0.9, with mT5L among the strongest across languages, while five-shot prompted causal models performed worse in every language.Source: §4.1.1 · p. 5, Figure 4 · p. 6
• Increasing the number of domains represented in Llama2-7b demonstrations improved correlation to slightly above 0.7 and outweighed gains from adding more shots.Source: §4.1.1 · p. 5, Figure 5 · p. 6

4.2 Unsupervised Methods

• Traditional comparisons include sentence length, ARI, FKGL, and OSMAN alongside LM-based unsupervised scores.Source: §4.2 · p. 6
• RSRS combines language-model word losses with sentence length, assigning higher weights to words with larger losses and to out-of-vocabulary tokens.Source: §4.2 · p. 6

4.2.1 Results

• RSRS correlated better than traditional metrics in English but was model-dependent elsewhere, with multilingual models outperforming monolingual ones for Arabic, Hindi, and Russian.Source: §4.2.1 · p. 6, Figure 6 · p. 7
• Increasing the transliteration penalty improved correlation by up to 6-7% for monolingual models and 1-3% for multilingual models, indicating greater multilingual robustness.Source: §4.2.1 · p. 7, Figure 7 · p. 7

5 Cross-Domain Cross-Lingual Analyses

• README++-trained models outperform models trained on previous datasets in both unseen-domain generalization and cross-lingual transfer.The cross-domain comparison uses CEFR-SP and ALC Corpus baselines, while cross-lingual transfer compares against CEFR-SP and CompDS.Source: §5.1 · p. 8, §5.2 · p. 8, Table 4 · p. 8

5.1 Performance on Unseen Domains

• Models fine-tuned on README++ achieve better performance on unseen domains than equivalent models trained on CEFR-SP for English and ALC for Arabic.The experiments remove randomly sampled domains for testing and train on the remaining README++ data; the Arabic evaluation converts labels to a shared five-level CEFR scale.Source: §5.1 · p. 8, Table 4 · p. 8

5.2 Performance on Cross-lingual Transfer

• README++ English fine-tuning yields stronger zero-shot transfer than CEFR-SP or CompDS across Arabic, Hindi, French, Russian, Italian, and German.The XLMRL results reach correlation values of 0.7 in most languages, with performance increases of 50% in several cases.Source: §5.2 · p. 8, Table 5 · p. 8
• The cross-lingual evaluation compares models trained on matched sample sizes from README++, CEFR-SP, and full English CompDS datasets.Evaluation covers README++ test sets for Arabic, Hindi, French, and Russian, plus Italian CompDS and German TextComplexityDE, with levels 6 and 7 merged for the latter datasets.Source: §5.2 · p. 8

6 Conclusion

• README++ contains 9757 human-annotated sentences in five languages from 112 data sources, with readability labels based on the CEFR scale.The languages are Arabic, English, French, Hindi, and Russian.Source: §6 · p. 8
• README++-trained language models achieve strong performance across textual domains and outperform prior-dataset models in transfer from English to six target languages.The target languages are Arabic, French, Hindi, Russian, Italian, and German.Source: §6 · p. 9

Limitations

• The medical domain is available only in English, although medical-related texts in other languages appear within Research and Wikipedia domains.Most other domains include texts in all considered languages, with exceptions where openly accessible data was unavailable.Source: Limitations · p. 9
• README++ does not cover low-resource languages, limiting evaluation of cross-lingual transfer in those scenarios.The authors propose extending the dataset to additional languages using a forthcoming annotation interface.Source: Limitations · p. 9
• Transliterations can negatively affect the LM-based RSRS unsupervised metric because of its handling of rare words.The authors note that rare jargon and complex terminology may themselves increase sentence difficulty, motivating further study.Source: Limitations · p. 9

Ethical Considerations

• The corpus primarily uses public-domain, Creative Commons, or otherwise permitted sources, with source licenses and terms documented in Appendix G.The authors sampled a small number of sentences for academic readability labeling.Source: Ethical Considerations · p. 9
• Sentences and annotations from the i2b2/VA and Hindi Product Reviews datasets will be withheld unless original-author access permission is obtained.Source: Ethical Considerations · p. 9
• Annotators were informed about dataset use, and sampled social-media or forum sentences containing offensive speech, stereotypes, or private information were excluded.Source: Ethical Considerations · p. 9

A.1 Textual Domains

• README++ collects multilingual sentences from diverse domains and data sources, with examples spanning Wikipedia, news, research, literature, legal texts, dialogue, reviews, and forums.Sources include websites, PDFs, existing datasets, and manually collected sentences.Source: §A.1 · pp. 15–16
• The dataset includes culture-specific materials, including original-language poetry, quotations, literature, and language-specific constitutional or governmental texts.Examples are provided for English, Arabic, Hindi, French, and Russian.Source: §A.1 · pp. 15, 17
• Medical texts are available only in English, while several other domains also lack comparable resources in particular languages.The corpus is therefore not uniformly balanced across languages and domains.Source: §A.1 · pp. 15, 17

A.2 Domain Distribution

• Basic levels mainly contain straightforward domains, intermediate levels factual domains, and proficient levels scientific, technical, legal, medical, or highly literary text.This distribution is reported across languages and domains in the corpus.Source: §A.2 · p. 17, Table 6 · p. 18

B CEFR Levels Descriptors

• Annotators were familiarized with CEFR level descriptors that define learner capabilities before assigning readability labels.The scale provides the annotation grounding for sentence readability.Source: B CEFR Levels Descriptors · p. 17, §A.2 · p. 17

C Traditional Metrics

• ARI estimates the grade level needed to understand a text using statistical features based on words and characters.The metric is computed from a formula described in the paper.Source: C Traditional Metrics · p. 18
• FKGL estimates grade level like ARI but additionally incorporates the total number of syllables.Both metrics are statistical formulas based on word, character, and syllable counts.Source: C Traditional Metrics · p. 18
• OSMAN computes Arabic narrative readability from word, sentence, long-word, syllable, and Faseeh-word counts.The formula defines A, B, C, D, G, and H as these corresponding counts.Source: C Traditional Metrics · p. 18

D.1 Language Models

• The study evaluates multilingual, monolingual, and encoder-decoder language models for readability assessment, including mBERT, XLM-R, language-specific BERT variants, and mT5.The model inventory spans Arabic, English, French, Hindi, and Russian settings.Source: §D.1 · p. 18, Table 8 · p. 19

D.2 Corpus Split

• README++ uses train, validation, and test splits with a 60%/10%/30% ratio per domain, ensuring every domain appears in each split.The same split is used across experiments except for the domain-generalization study.Source: §D.2 · p. 19, Table 9 · p. 19

D.3 Few-shot Prompt

• Few-shot prompts contain a task description, readability definition, CEFR examples, scored example sentences, and a target sentence.The prompt scaffold stays fixed while the sampling of demonstrations is varied.Source: §D.3 · p. 19, Table 10 · p. 20
• Readability levels vary substantially across domains and languages, motivating evaluation procedures that account for data diversity.The paper presents this variation as a central property of README++.Source: Figure 8 · p. 21

E.2 Domain Correlation

• In 18 of 21 domains, XLM-R trained on README++ outperforms the CEFR-SP-trained model in correlation with human readability labels.The comparison uses README++ data from 112 sources versus CEFR-SP data from one source.Source: §E.2 · p. 21, Figure 11 · p. 22

E.3 Zero-shot Cross Lingual Transfer

• README++ fine-tuning significantly improves zero-shot transfer to six target languages compared with earlier datasets, consistently across models.Per-domain results also show superiority for Arabic, French, Hindi, and Russian.Source: §E.3 · p. 21, Table 11 · p. 23, Table 12 · p. 24

E.4 Effect of Context

• Prepending up to three preceding sentences during supervised fine-tuning decreases performance in most cases or has little to no effect.Context is separated from the target sentence with a [SEP] token when available.Source: §E.4 · p. 21, Figure 10 · p. 22, Figure 12 · p. 23

F Annotation Interface

• The annotation interface uses rank-and-rate labeling for batches of five sentences, with optional context display and transliteration markers for Arabic and Hindi.Annotators first rank sentences, then assign CEFR-aligned scores from 1 to 6.Source: F Annotation Interface · p. 22, Figure 16 · p. 33, Figure 17 · p. 33

G License and Use Terms

• License tables record source-specific licenses, public-domain status, permissions, or author-request requirements for corpus use.Some sources require access permission from their original authors.Source: G License and Use Terms · pp. 22–23, Table 18 · p. 35