Source-linked AI summary
A Report on the Complex Word Identification Shared Task 2018
Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo H. Paetzold, Lucia Specia, Sanja Štajner, Anaïs Tack, Marcos Zampieri
TL;DR
CWI research needs evidence on which words challenge readers across languages, genres, and task formulations. This paper reports the second CWI shared task using multilingual, multi-genre datasets and four tracks with binary and probabilistic classification. Traditional feature-engineering systems generally outperformed neural and embedding-based approaches, while cross-lingual results were promising, including for French without French training data.
Problem
The shared task addresses the need to identify words that challenge target readers across multiple languages, genres, and formulations of complexity.
Method
The paper reports a shared task using multilingual and multi-genre datasets, four language tracks, and binary and probabilistic classification tasks.
Results
Traditional feature engineering-based approaches generally outperformed neural network and word embedding-based approaches, while cross-lingual results were promising and French scores were superior or equivalent to German and Spanish scores without French training data.
Takeaways & Limitations
The findings indicate improving CWI results and promising cross-lingual performance, although the traditional-feature advantage was less pronounced for probabilistic classification.
Takeaways & Limitations
The French datasets likely produced less varied complex-word annotations because they had fewer similarly participating annotators than English or Spanish.
Abstract
from arXiv · showhide
We report the findings of the second Complex Word Identification (CWI) shared task organized as part of the BEA workshop co-located with NAACL-HLT'2018. The second CWI shared task featured multilingual and multi-genre datasets divided into four tracks: English monolingual, German monolingual, Spanish monolingual, and a multilingual track with a French test set, and two tasks: binary classification and probabilistic classification. A total of 12 teams submitted their results in different task/track combinations and 11 of them wrote system description papers that are referred to in this report and appear in the BEA workshop proceedings.
1 Introduction
Complex Word Identification (CWI) identifies words considered complex by a target population and is commonly the first step in lexical simplification. The 2018 shared task expanded evaluation across languages, tracks, and binary or probabilistic formulations.
- CWI identifies words considered complex by a target population and commonly begins lexical simplification pipelines.
- The 2018 shared task evaluated systems on English, German, Spanish, and multilingual CWI with a French test set.
- Participants received labeled training data, then submitted predictions for an unlabeled test set evaluated by the organizers.
- The task included binary labeling of words as complex or simple and probabilistic assignment of their likelihood of being complex.
- Participants could choose which task and track combinations to enter.
2 Related Work
The related work describes the earlier SemEval 2016 CWI dataset and directs readers to prior systems presented for that shared task.
- The 2016 shared task introduced a manually annotated CWI dataset containing 9,200 sentences annotated by 400 non-native English speakers.The training set contained 200 sentences with 20 annotations per target, while 9,000 test sentences had one annotation per target.
- The 2016 dataset used 200 training sentences and 9,000 test sentences with different annotation densities.
- Prior approaches from SemEval 2016 Task 11 are presented in the paper’s Table 1.
3 Datasets
The 2018 CWI shared task used multilingual datasets, including English texts from multiple genres and a French test set, with annotations collected from native and non-native speakers. Dataset analyses reveal substantial cross-language differences in agreement, complex-instance rates, and multiword-expression annotations.
- Dataset coverage: The shared-task datasets covered English, German, and Spanish, while French data were collected separately to support the cross-lingual experiment.English included News, WikiNews, and Wikipedia genres; French data consisted of Wikipedia texts from a comparable simplified corpus.
- Annotation process: Annotations were collected through MTurk by presenting annotators with 5–10-sentence paragraphs and asking them to highlight difficult words or phrases for a specified target reader.The instructions targeted readers such as children, language learners, or people with reading impairments.
- Agreement: Around 90% of English complex phrases were selected by at least two annotators, but agreement fell to 83% in the lowest language-specific case.Native annotators agreed within their group at 84% or above, non-native annotators at 83% or above, while absolute native–non-native agreement was 70%.
- Agreement: German annotators showed higher agreement among non-native speakers than native speakers: 70.66% versus 58.5%.On average, 6.1 non-native and 3.9 native speakers participated in a German HIT.
- Cross-language differences: Spanish annotations had lower agreement than English and German annotations, while Spanish annotators highlighted more multiword expressions.The Spanish annotations came almost exclusively from native annotators.
- Cross-language differences: French had a lower relative count of complex instances but a higher relative proportion of multiword-expression annotations than the other datasets.The authors attribute this disparity partly to the ten-annotation HIT limit, which could encourage grouping adjacent complex words, and partly to fewer French annotators.
4 System Descriptions and Results
The shared-task systems combined engineered, lexical, contextual, morphological, semantic, and neural features across monolingual and multilingual tracks. Traditional feature-engineering systems generally performed best, while selected systems achieved strong cross-lingual and task-specific results.
- Evaluation setup: The evaluation covered monolingual English and multilingual German, Spanish, and French tracks, with binary and probabilistic classification results reported separately.The tables organize results by language track and task type.
- Evaluation setup: Baseline systems used frequency and length features with Nearest Centroid for binary classification and Linear Regression for probabilistic classification.Accuracy and macro-averaged F1 evaluated binary classification, while MAE evaluated probabilistic classification.
- Cross-lingual results: Character n-gram features maximized English performance but generally degraded performance for cross-language complex word identification.The English setup combined NEWS, WIKINEWS, and WIKIPEDIA training data with 3-, 4-, and 5-gram features.
- System results: XGBoost gave the best binary results for one feature-engineering system, while its LSTMs gave the best probabilistic results.The submitted systems also included average embeddings, transfer-learning LSTMs, and a voting combination.
- System results: Feature ablation found length, frequency, and corpus-probability features important for both tasks, with learner-corpus features more influential than general-domain features.The finding was reported for random-forest classifiers and regressors.
- System results: Most submitted systems ranked first for the English monolingual binary and probabilistic classification tasks.This pattern was reported across the results in Tables 6 through 9.
- Cross-lingual results: The strongest cross-lingual systems performed best on the French dataset in both binary and probabilistic classification.Their results were described as a promising direction for creating CWI datasets for new languages.
5 Conclusions
The second CWI shared task expanded evaluation across languages and showed improving results, with traditional feature engineering generally outperforming neural and embedding-based approaches. Cross-lingual experiments were especially promising, including French evaluation without French training data.
- Participation: The shared task received submissions from 12 teams, and 11 teams subsequently contributed system description papers reviewed in the report.Thirty teams enrolled in the competition.
- Overall findings: Traditional feature-engineering approaches generally outperformed neural-network and word-embedding approaches, while CWI results improved compared with SemEval 2016.The performance difference was less pronounced for probabilistic classification.
- Cross-lingual findings: Cross-lingual experimental results were described as very promising for CWI research.The conclusion presents this as one of the report’s most important findings.
- Cross-lingual findings: French results achieved superior or equivalent scores to German and Spanish datasets when trained on one or more other languages, although the scores were not directly comparable.No French training dataset was provided.