PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Traditional and Neural Approaches to Multilingual Readability Assessment

Joshua Wong, Chris Tanner

arXiv:2609.10792v1cs.CL

TL;DR

Readability models achieve strong performance, but it remains unclear whether transformers learn the linguistic features that make text difficult, especially under subjective and rater-dependent labels. The paper uses SHAP-selected features as TCAV concepts to compare traditional classifiers with multilingual and language-specific encoders across five languages. Transformers recover several classical signals and CEFR ordering, but alignment varies and separability does not always indicate directional influence.

  • Problem

    It remains unclear whether transformer readability encoders use the same linguistic features as traditional classifiers, despite subjective and rater-dependent readability labels.

  • Method

    The study trains traditional classifiers, selects their predictors with SHAP, and uses them as TCAV concept sets to probe XLM-R and language-specific encoders across five languages.

  • Results

    Transformers recover surface-length, syntactic, and lexical-diversity signals and reflect CEFR ordering, with alignment varying by language, layer, and model family.

  • Takeaways & Limitations

    Language-specific encoders generally track traditional readability models more clearly than XLM-R, while linear separability alone is insufficient evidence of functional concept use.

  • Takeaways & Limitations

    The study uses one sentence-level dataset, does not average transformer results across random seeds, and compares only one multilingual and one language-specific encoder per language.

Abstract

from arXiv · show

Transformer-based models excel at Automatic Readability Assessment (ARA), yet feature-based models remain in active use because their predictions tie back to linguistic properties. This matters because readability labels are subjective and rater-dependent, so high accuracy on noisy ground truth may reflect surface patterns rather than the linguistic structure that defines difficulty. We test whether transformers internalize the same features as traditional models across Arabic, English, French, Hindi, and Russian using the ReadMe++ dataset. Shapley Additive Explanations (SHAP) identify the features driving traditional classifiers, which we then use as TCAV concept sets to probe multilingual XLM-R and language-specific encoders. Transformers recover surface-length, syntactic, and lexical-diversity signals, and reflect the ordinal CEFR structure of the traditional models. Alignment varies by model family, language, and layer, with language-specific encoders tracking traditional models more clearly than XLM-R. High linear separability does not always imply directional influence, limiting linear probing for count-based readability features.

1 Introduction

The paper asks whether transformer encoders learn the same interpretable linguistic signals used by traditional readability classifiers across five languages. It proposes a SHAP–TCAV comparison and finds partial alignment that varies by language, layer, and model family.

  • Motivation: High accuracy can be insufficient evidence of linguistic understanding because readability labels are subjective and rater-dependent.The paper motivates interpretability analysis because noisy ground truth may permit surface-pattern matching without recovering the structure of difficulty.
  • Research question and approach: The study tests whether transformers recover the linguistic features that traditional readability models use across Arabic, English, French, Hindi, and Russian.The comparison uses the README++ dataset and addresses whether predictive accuracy reflects linguistic structure rather than only noisy labels.
  • Research question and approach: SHAP identifies features driving traditional classifiers, and those features become TCAV concept sets for probing XLM-R and language-specific encoders.This directly compares feature importance in traditional models with directional influence in neural representations.
  • Findings: Transformers recover surface length, syntactic complexity, and lexical diversity, while some also reflect the ordinal structure of CEFR levels.Alignment varies across languages, layers, and model families, with language-specific encoders generally showing clearer patterns than XLM-R.
  • Contributions: The contribution is a multilingual comparison of feature-based and transformer classifiers plus a SHAP–TCAV framework for testing interpretable feature alignment.The framework characterizes variation by language, model family, and encoder layer.

2 Related Work

Prior readability research developed from formula-based and handcrafted linguistic features toward multilingual and transformer models. Existing work evaluates predictive performance, but whether neural readability encoders use the same features as traditional classifiers remains untested.

  • Feature-based and multilingual ARA: Traditional readability systems progressed from surface formulas to handcrafted lexical, syntactic, part-of-speech, and discourse features.These features remain useful because they connect predictions to linguistic properties.
  • Feature-based and multilingual ARA: Multilingual research uses language-independent features, language-specific models, cross-lingual transfer, and multilingual encoders across CEFR resources.UniversalCEFR benchmarks feature-based models, fine-tuned encoders, and instruction-tuned LLMs across languages and label granularities.
  • Research gap and interpretability: Prior studies measure predictive performance but do not test whether neural models encode the linguistic features used by feature-based readability systems.This untested alignment question motivates the paper’s comparison.
  • Transformer-based and hybrid ARA: Transformers often outperform traditional models, while hybrid systems combining neural representations or predictions with handcrafted features perform strongly in low-data or cross-domain settings.This creates a tension over whether explicit linguistic features provide information transformers do not fully capture.
  • Research gap and interpretability: SHAP identifies important input features, whereas probing tests recoverability; TCAV addresses the gap between linear separability and functional influence.A feature can be linearly separable without influencing predictions, motivating directional concept-based analysis.

3 Methodology

The study compares traditional feature-based readability classifiers with multilingual and language-specific transformers across five languages, using SHAP-selected linguistic features as TCAV concepts. It combines controlled data preparation, feature selection, classifier training, and layer-wise probing to assess whether neural representations encode interpretable readability signals.

  • Method overview: The pipeline trains traditional classifiers on handcrafted linguistic features, uses SHAP to rank predictors, and probes transformer representations with those features as TCAV concepts.This compares feature importance in traditional models with directional influence in multilingual and language-specific encoders.
  • Data: README++ supplies 9,685 sentence-level examples across Arabic, English, French, Hindi, and Russian, split stratified by language into 60% training, 10% validation, and 30% test data.The corpus uses one annotation protocol and CEFR labels, while class distributions vary across languages.
  • Features: LFTK extracts interpretable surface, density, lexical, and syntactic features, while readability formula features are excluded; correlation clustering reduces the feature sets before modeling.The reduction yields 59 English features and 32–35 selected features for each non-English language.
  • Models: Logistic Regression, Linear SVM, and Random Forest classifiers are trained per language with balanced class weights, while XLM-R and language-specific encoders are fine-tuned for six-way CEFR classification.Transformer models use a fresh classification head, 12 encoder layers, and the learning rate selected by validation QWK.
  • Concept construction: SHAP importance is aggregated across classifier families with Borda ranks, and the 10 lowest- and 10 highest-ranked features per language become TCAV concepts.Each positive concept set contains 100 sentences with the highest value of its selected feature.
  • TCAV probing: TCAV separates concept recoverability from predictive influence by measuring both CAV classifier accuracy and the fraction of examples whose concept direction raises a CEFR-class probability.Activations and gradients are extracted layer by layer from held-out test examples, with full 512-token input sequences.

4 Results

Transformers outperform traditional classifiers on readability prediction, while SHAP-selected linguistic features are recoverable in transformer representations with language-, model-, and layer-dependent alignment. Linear separability and directional influence diverge most clearly in Arabic.

  • 4.1 Model Performance: Transformers outperform traditional classifiers across languages on QWK and accuracy, with QWK rising from 0.65–0.78 to 0.83–0.86 and accuracy from 0.44–0.51 to 0.53–0.64.Language-specific encoders also exceed traditional classifiers on macro F1 in English, Arabic, and Russian; transformer families are broadly similar on QWK but diverge more on accuracy and macro F1.
  • 4.2 What Feature-Based Models Use: Surface-length features rank among the strongest predictors in every language, while type-token ratio appears in the top seven across all five languages.Total characters appears in every language’s top three, and average characters per word appears in the top four for every language.
  • 4.2 What Feature-Based Models Use: SHAP contributions invert between A1 and C2 for surface-length features, yielding an ordinally coherent pattern despite flat cross-entropy and no ordinal constraint.High length-feature values produce negative contributions for A1 and positive contributions for C2, with intermediate classes transitioning between these poles.
  • 4.2 What Feature-Based Models Use: Feature directions follow linguistic intuition: longer sentences and words push toward harder CEFR levels, whereas raw type-token ratio pushes toward easier text.Length-normalized lexical diversity reverses this pattern in Arabic, Hindi, and Russian, and English pronoun density trends toward easier text.
  • 4.3 CAV Separability: CAVs classify top-10 SHAP concepts more accurately than bottom-10 concepts across all five languages, with separability varying across layers and model families.Language-specific encoders often show clearer layerwise patterns, while XLM-R can be noisier or non-monotonic.
  • 4.4 TCAV Directionality: TCAV reproduces SHAP-derived directions in four languages, but Arabic shows high separability without consistent directional influence.English BERT is the clearest example across layers 0–5; Hindi reverses the usual model-family pattern, with XLM-R providing clearer directionality than MuRIL.

5 Discussion

Feature-based classifiers and transformers converge on readability signals and directions, but the clarity of this agreement depends on language and model family. The ordinal CEFR pattern emerges despite non-ordinal training, while Arabic and Hindi expose important limits and hypotheses for further testing.

  • Transformers rediscover the same readability signals and directions identified by feature-based classifiers, despite never receiving feature values as inputs.Agreement in both separability and polarity supports convergence on the same underlying structure.
  • Agreement is clearer in some languages and model families: language-specific encoders show stable layer-wise trajectories, whereas XLM-R is noisier in middle layers.Hindi reverses this pattern, while Arabic departs from it differently.
  • The Hindi reversal is attributed to data conditions rather than language, especially its small, balanced subset and potentially unstable CAV training.MuRIL’s limited and partly transliterated pretraining data may also contribute; varying data size and class balance would test the hypothesis.
  • Arabic shows separable concepts without consistent directional influence, suggesting that morphology may weaken the linear link between surface counts and difficulty.Morphology-aware concepts and nonlinear probes are proposed as the next test.
  • The shared A1-to-C2 inversion indicates ordinal CEFR structure emerges from the task despite flat cross-entropy training.Neither analysis was designed to produce ordinal structure, and testing whether ordinal losses add less requires further work.

6 Conclusion

The paper argues that transformer readability gains rest on linguistic features already trusted by traditional ARA, and introduces SHAP–TCAV as complementary evidence for comparing them. Agreement narrows explanations, while disagreement identifies where the interpretability comparison reaches its limits.

  • Transformer ARA performance gains rest on linguistic features already trusted by traditional models, giving the shift toward encoders firmer interpretive footing.
  • SHAP–TCAV combines explicit feature usage with neural representation geometry, so agreement narrows explanations and disagreement locates limits such as Arabic.The procedure could transfer to settings where interpretable baselines and neural models coexist, though broader validation remains future work.

Limitations

The study’s evidence is bounded by one sentence-level dataset, asymmetric feature coverage, prespecified TCAV directions, and a limited model comparison. These constraints restrict generalization and leave concept discovery and causal influence for future work.

  • The experiments use only README++ at sentence level on the CEFR scale, without averaging transformer results across random seeds.
  • Cross-lingual feature coverage is asymmetric because general-purpose LFTK features are narrower for non-English languages, especially entity density and lexical familiarity.
  • TCAV tests only prespecified linear directions and is sensitive to random-set choice and concept entanglement, so it cannot reveal unspecified concepts.
  • The comparison includes one multilingual and one language-specific encoder per language, limiting generalization to larger or differently structured models.

Ethics Statement

The work uses public readability corpora and released checkpoints without new human-subject participation or direct misuse risks. It nevertheless flags possible deployment bias from limited-domain data and uneven multilingual feature availability.

  • The study uses publicly available corpora and existing annotations, with fine-tuned models derived from publicly released checkpoints under permissive licenses.
  • Readability models trained on limited-domain data may produce biased judgments in educational settings.
  • Uneven feature availability across languages creates an equity concern for multilingual ARA.

A Additional SHAP Results

Additional SHAP analyses show that the ordinal polarity between the easiest and hardest CEFR classes holds across all four languages examined.

  • The A1-to-C2 ordinal polarity reported in the main text holds across Arabic, English, French, and Russian.Figures 6–9 show one SHAP beeswarm panel per CEFR class, with feature values encoded by color and SHAP contribution by horizontal position.

B Additional TCAV Results

Additional TCAV results show coherent directional patterns for top SHAP features in French and Russian across mid-to-late layers.

  • French and Russian show coherent directional TCAV patterns for the top 10 SHAP features across mid-to-late layer groups.XLM-R and language-specific encoders are shown separately, with C2 in the upper panel and A1 in the lower panel; asterisks mark non-significant results.

C Additional CAV Accuracy Results

Additional CAV analyses extend the classifier-accuracy comparison across French, Hindi, and Russian, while the supplied implementation passages add no empirical findings.

  • Across French, Hindi, and Russian, mean CAV classifier accuracy preserves the gap between top and bottom 10 SHAP features across layers.Figure 12 reports mean accuracy across 12 layers and complements the Arabic and English plots in the main text.
Loading 2609.10792v1…