Semi-Supervised Joint Estimation of Word and Document Readability

Motivation

• Existing readability methods often require annotated lexical or textual resources, while such resources remain scarce outside major European languages and costly to construct.Resources have been created for languages including English, French, and Swedish, but manually building them demands linguistic expertise and effort.Source: §1 · p. 1
• Word and document difficulty are recursively related: document difficulty correlates with its hardest word, while word difficulty correlates with the easiest document containing it.This bidirectional relationship motivates estimating both levels together rather than treating them independently.Source: §1 · p. 1

Proposed approach

• The proposed method jointly estimates word and document readability semi-supervisedly by modeling words and documents as graph nodes and recursively inferring their embeddings with GCN layers.The approach uses both labeled supervision and the recursive word-document relationship.Source: §1 · pp. 1–2

Contributions

• GCNs effectively exploit unlabeled data and remain effective when fewer labeled examples are available.The paper reframes readability estimation as a semi-supervised joint problem and identifies GCNs as a solution for this setting.Source: §1 · p. 2

Task formulation

• Joint readability estimation maps words and documents to difficulty labels, supporting either classification or regression formulations.The classification setting uses six CEFR levels from A1 to C2, while regression produces a real-valued readability estimate.Source: §2 · p. 2

Semi-supervised setting

• Words and documents are partitioned into labeled and unlabeled subsets, with inference using labeled supervision and potentially word-document relationships among unlabeled instances.The labeled subsets are W_L and D_L, while W_U and D_U provide additional structure.Source: §2 · p. 2

3.1 Recursive Relationship of Word and Document Difficulty

• Word difficulty correlates with the minimum difficulty of documents containing the word, while document difficulty correlates with the maximum difficulty of its words.This recursive relationship motivates joint estimation rather than treating word and document readability independently.Source: §3.1 · p. 2, Figure 2 · p. 2

3.2 Graph Convolutional Networks on Word-Document Graph

• The GCN treats words and documents as graph nodes whose hidden layers recursively connect them to capture potentially nonlinear readability relationships.Unlabeled evaluation nodes are included because GCN inference is transductive.Source: §3.2 · pp. 2–3
• The graph uses TF-IDF document-word edges, normalized PMI word-word edges, and self-loops to encode heterogeneous relationships.The adjacency matrix is symmetrically normalized using node degrees before convolution.Source: §3.2 · pp. 2–3
• Separate final linear layers predict word and document difficulty, while a weighted combination of their losses trains the joint model.The framework supports regression with mean squared error and classification with softmax cross-entropy.Source: §3.2 · p. 3
• A simplified linear GCN performed worse than the ReLU hidden-layer model, and a shared final linear layer also underperformed.These comparisons support using nonlinear hidden layers and separate output layers for the two node types.Source: §3.2 · p. 3

4.1 Results

• GCNs improve both document and word accuracy over the baselines in the reported difficulty-estimation results.Table 2 reports accuracy and correlation for logistic regression and GCN classification and regression variants.Source: §4.1 · p. 4, Table 2 · p. 4
• GCN propagates an A1 document label to its word node, correctly identifying “bicycle” where logistic regression predicts A2 from input features alone.Source: §4.1 · p. 4

4.2 Ablation Study on Features

• Without traditional or embedding features, GCN achieves results comparable to some baselines, showing that word-document graph structure supplies complementary readability signals.The ablation’s “None” setting uses one-hot node features and relies solely on graph structure.Source: §4.2 · p. 4, Table 3 · p. 4
• Removing BERT embeddings decreases document accuracy by 0.112 on the Cambridge Readability Dataset.The result supports BERT embeddings as a powerful document-readability feature in this setting.Source: §4.2 · p. 4

4.3 Training on Less Labeled Data

• GCN performance remains robust with smaller training sets, especially for word accuracy, by exploiting recursive word-document relationships and graph context.The word-accuracy advantage over the baseline becomes larger than the document-accuracy gap when training data is small.Source: §4.3 · p. 4

• The GCN jointly estimates word and document readability and achieves higher accuracy while using smaller amounts of labeled data.Its advantage is attributed to capturing the recursive difficulty relationship between words and documents.Source: §5 · p. 4, Figure 3 · p. 5
• Figure 3 compares word and document accuracy across different amounts of training data.Source: Figure 3 · p. 5
• The framework could be extended to include subwords, paragraphs, and grammatical concepts, which the paper leaves for future work.Source: §5 · p. 5

Search procedure

• The hyperparameter search samples 200 combinations across α, learning rate, dropout, epochs, hidden units, layers, and PMI window width.Source: A Hyperparameter Details · p. 6

Classification setting

• Classification uses separate GCN configurations for document and word difficulty estimation.Document estimation uses dropout 0.5, 500 epochs, 512 hidden units, 2 layers, and PMI window width 5; word estimation uses dropout 0.2, 250 epochs, 64 hidden units, 1 layer, and disabled PMI.Source: A Hyperparameter Details · p. 6

Regression setting

• Regression also uses separate GCN configurations for document and word difficulty estimation.Document estimation uses dropout 0.3, 1500 epochs, 128 hidden units, 2 layers, and PMI window width 5; word estimation uses dropout 0.1, 500 epochs, 512 hidden units, 2 layers, and disabled PMI.Source: A Hyperparameter Details · p. 6