SemEval-2021 Task 1: Lexical Complexity Prediction

1 Introduction

• Identifying potentially difficult words can support interventions such as lexical simplification or contextual explanations.LCP is also relevant to readability assessment and ranking possible replacement words.Source: §1 · pp. 1–2
• LCP assigns continuous complexity values rather than binary labels, aggregating multiple annotators’ Likert judgments to represent where a word falls on the scale.This approach aims to capture differing judgments more fully than a complex-or-not decision.Source: §1 · p. 1
• In context, annotators can give the same word differing complexity ratings, which may yield a medium value rather than a forced binary decision.The example describes “observation” as potentially difficult for lay readers despite being clear to someone with scientific training.Source: §1 · p. 2
• The shared task provided 10,800 instances across three genres, with unigrams and bigrams, and separate subtasks for single words and multi-word expressions.Trial, training, and test splits were released for the competition.Source: §1 · p. 2

2 Related Tasks

• The 2016 CWI task used binary labels, with a 2,237-instance training set annotated by 20 people per instance and an 88,221-instance test set annotated by one person per instance (Paetzold and Specia, 2016a).The winning system used threshold-based methods and Simple Wikipedia features (Paetzold and Specia, 2016b).Source: §2 · p. 2
• The 2018 CWI task included binary classification and a regression or probabilistic-classification objective estimating whether language learners considered target words complex (Yimam et al., 2018).Source: §2 · p. 2
• The 2018 task supplied datasets across genres and languages, including English, German, Spanish, and French (Yimam et al., 2018).Source: §2 · p. 3
• Prior systems used lexical, frequency, n-gram, embedding, and psycholinguistic features with varied classifiers; the 2018 winner combined Adaboost with WordNet, POS, dependency, and psycholinguistic features.The 2016 post-competition analysis found that most systems performed poorly, citing annotation design and small training data as main factors.Source: §2 · pp. 2–3

3 Data

• Because CompLex 1.0 annotations were judged poor and averaged seven annotators per instance, the authors added ten Mechanical Turk annotations per instance for the shared task.The earlier dataset retained only instances with at least four annotations and was less reliable than expected.Source: §3 · p. 3
• CompLex 2.0 aggregates retained CompLex 1.0 annotations with new data across three genres, covering single and multi-word instances.CompLex 1.0 drew from Europarl, the World English Bible, and biomedical literature, and focused on nouns and selected noun phrases.Source: §3 · p. 3
• Each instance receives the mean of multiple ratings on a five-point Likert scale mapped from 0 for Very Easy to 1 for Very Difficult.Averaging assumes annotation noise evens out and that annotation distributions are unimodal.Source: §3 · pp. 3–4
• Instances include their enclosing sentence, and repeated tokens can appear in up to five contexts per genre with separate complexity ratings.The repeated contexts are intended to penalize systems that ignore context; explicit word-sense distinctions are left to participants.Source: §3 · p. 4
• The corpus includes 1,800 MWEs, selected as noun-noun or adjective-noun patterns followed by a non-noun, but they are not guaranteed to be semantic MWEs.Source: §3 · p. 4
• The selected genres are intended to encourage systems to learn complexity annotations that generalize across text types.The corpus uses specific texts rather than only informal or news-style material.Source: §3 · p. 4
• The corpus contains 10,800 contexts with 5,617 unique words; multi-word expressions average 0.419 complexity versus 0.302 for single words.Each genre has 3,600 contexts, and each token has about two contexts on average.Source: §3 · p. 4

4 Data Splits

• The split procedure stratified by genre, complexity, and token, keeping instances of the same token in only one split to prevent token-specific overfitting.Source: §4 · p. 4
• For each genre, groups of 20 were allocated as one trial, two test, and 17 training groups, producing an approximate 5-85-10 trial-training-test split.The test instances were randomized before release to prevent label guessing from ordering.Source: §4 · pp. 4–5
• The reported split statistics show roughly even representation of genres across trial, training, and test sets.Table 2 presents the statistics for the three shared-task splits.Source: §4 · p. 5, Table 2 · p. 5

• Participants were ranked primarily by Pearson’s correlation, with Spearman’s rank, MAE, and MSE also used for secondary rankings.Pearson’s correlation measures how variables change together and is robust to scale changes; Spearman’s rank considers ordering, while MAE reflects prediction distance from gold labels.Source: §5 · p. 5
• Sub-task 2 participants submitted predictions for both single words and MWEs, but MWE-only submissions were not ranked because the test set contained 184 MWEs.Source: §5 · p. 5
• Winning systems reached Pearson’s Correlation of 0.7886 on Sub-task 1 and 0.8612 on Sub-task 2, with R2 scores of 0.6210 and 0.7389, respectively.Source: §5 · p. 6
• The majority of systems beat the log-frequency Google Web1T baseline, and rankings varied only slightly across Spearman’s rank, MAE, and MSE.Source: §5 · p. 6
• Scores were generally higher on Sub-task 2, which included single words and MWEs, than on Sub-task 1, which covered single words only.The authors suggest the difference may relate to MWEs being more complex and potentially making between-group complexity variance easier to model.Source: §5 · pp. 5–6

6 Participating Systems

• The section analyzes 32 teams’ system-description papers, summarizing top systems and comparing their features and approaches.Source: §6 · p. 6

6.1 System Summaries

• JUST BLUE attained the highest Pearson’s Correlation for Sub-task 1 and did not participate in Sub-task 2; DeepBlueAI attained the highest Pearson’s Correlation on Sub-task 2.Table 3 lists the top 10 systems for each task by Pearson’s correlation and includes R2 scores to help interpret the correlation.Source: §6.1 · p. 6, Table 3 · p. 6
• CS-UM6P ranked fifth on Sub-task 1 and seventh on Sub-task 2 by Pearson’s correlation, but had much lower R2 scores on both tasks.The authors suggest that significant outliers in the system’s output may account for the difference.Source: §6.1 · p. 7

6.2 Approaches

• Deep-learning systems occupied the top two places on both sub-tasks, feature-based systems reached third and fourth on Sub-task 1 with similar scores, and hybrid systems generally performed worse than either approach.Source: §6.2 · pp. 7–8
• Feature-based systems drew on lexical frequency and word length, psycholinguistic and semantic cues, sentence context, vocabulary lists, and static or contextual embeddings.Gradient Boosted Regression and Random Forest Regression were popular approaches for using these features.Source: §6.2 · p. 7
• Deep-learning systems typically fine-tuned pretrained language models on concatenated target-token and context inputs, with the ⟨CLS⟩ embedding used for complexity prediction.Reported adaptations included adversarial training, multitask learning, dummy annotation generation, and capsule networks.Source: §6.2 · pp. 7–8

6.3 MWEs

• For MWEs, feature-based systems averaged token features or scores, or doubled the feature space and trained a model on both words.The task asked participants to predict both single words and MWEs in Sub-task 2.Source: §6.3 · p. 8

7 Discussion

• Continuous annotations extend binary complex-word identification by representing degrees of difficulty and supporting rankings across words.The task provides in-context annotations across three genres for single tokens and multi-word expressions.Source: §7 · p. 8
• Complexity scores can support lexical simplification and readability assessment; thresholds can convert continuous scores to binary decisions for audience-specific use.The threshold depends on the target audience, and the annotation categories should guide its selection.Source: §7 · p. 8
• The corpus is not representative of English word frequencies: harder words were preselected, and annotations skew toward the easier Likert categories.The authors say a representative corpus would require annotating words without filtering.Source: §7 · p. 8
• Averaged complexity labels remain subjective: the authors retained instances with wide annotator disagreement, although fixed values may not reflect a particular reader or group.Individual annotations were not released, with the stated aim of focusing systems on predicting complexity.Source: §7 · pp. 8–9
• Pearson’s Correlation reached 0.7886 on Task 1 and 0.8612 on Task 2, with R2 of 0.6172 and 0.7389, and systems strongly outperformed the word-frequency baseline.The authors suggest MWEs may have been easier to predict because they are typically more complex and may exhibit lower variance.Source: §7 · p. 9
• Only 0.0182 Pearson’s Correlation points separated ranks 1 and 10 on Task 1, and 0.021 on Task 2; another random split could change the ordering.The authors note that the small differences make rankings sensitive to the dataset split.Source: §7 · p. 9
• Deep learning and feature-based systems showed little performance variation, though deep learning systems attained the highest scores; systems without context appeared to rank worse.Using previous CWI datasets may also have helped performance.Source: §7 · p. 9
• The task covers English only; the authors hope to extend it to other languages.Source: §7 · p. 9

8 Conclusion

• The paper presents SemEval-2021 Task 1 on Lexical Complexity Prediction.Source: §8 · p. 9
• The task dataset provides continuous complexity annotations in context across three genres.Source: §8 · p. 9
• The paper analyzes participating systems and categorizes their findings to help future researchers understand approaches suitable for LCP.Source: §8 · p. 9

A Full Results

• Table 4 reports Sub-task 1 results and ranks for Pearson, Spearman, MAE, MSE, and R2.Source: Table 4 · p. 13
• Table 5 reports Sub-task 2 results and ranks for Pearson, Spearman, MAE, MSE, and R2.Source: Table 5 · p. 14

B System Features

• Table 6 lists participating systems alongside their features and classification approaches.Source: Table 6 · p. 15