Source-linked AI summary
SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, Lucia Specia
TL;DR
The paper addresses evaluation of semantic similarity across multilingual, cross-lingual, and MT quality-estimation settings. It presents the 2017 shared-task evaluation and introduces the STS Benchmark, finding strong performance for established languages but weaker correlations for Arabic and some cross-lingual tracks. These results motivate more robust models for new languages and less-supervised settings.
Problem
STS evaluation has focused heavily on English, leaving multilingual, cross-lingual, and MT quality-estimation performance less explored.
Method
The paper evaluates six multilingual and cross-lingual tracks, reviews participating methods, analyzes errors, and releases the STS Benchmark from English shared-task data.
Results
State-of-the-art systems achieve strong correlations for previously evaluated languages, while correlations are weaker for Arabic, Arabic-English, and Turkish-English.
Takeaways & Limitations
The findings indicate a need for robust semantic-similarity models that transfer to new languages and perform well with less supervised training data.
Takeaways & Limitations
MT-based systems can obscure original translation errors by retranslating MT output, contributing to poor performance on Spanish-English MT quality-estimation pairs.
Abstract
from arXiv · showhide
Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017).
1 Introduction
STS measures graded semantic overlap between sentence pairs and supports evaluation across many language-understanding applications. The 2017 task broadened evaluation toward multilingual and cross-lingual settings and introduced the STS Benchmark for common evaluation.
- Motivation: STS measures how semantically equivalent two sentences are, capturing graded meaning overlap rather than only binary relationships.It differs from textual entailment and paraphrase detection, which classify particular relationships.
- Motivation: STS supports applications including machine translation, summarization, question answering, semantic search, and dialogue systems.
- Shared task: The annual STS shared task has evaluated state-of-the-art similarity methods since 2012, with signals evolving from lexical and syntactic matching toward alignment-based methods.
- 2017 evaluation: The 2017 task emphasized Arabic, Spanish, and English pairings with Arabic, Spanish, and Turkish, while excluding English-Turkish from the primary combined criteria.
- 2017 evaluation: 31 teams produced 84 submissions in the 2017 task despite its expanded language coverage.
- Benchmark: The STS Benchmark is a publicly available selection of English STS shared-task data from 2012–2017 for common evaluation of sentence similarity and semantic representations.
2 Task Overview
The task assigns real-valued semantic-similarity scores to sentence pairs and evaluates them against human judgments using an interpretable ordinal scale.
- Task definition: STS produces real-valued similarity scores for sentence pairs and measures Pearson correlation with human judgments.
- Annotation scale: The annotation scale ranges from 0 for no meaning overlap to 5 for meaning equivalence, with intermediate values representing partial overlap.
- Annotation scale: The scale is designed for reasonable human judges without formal linguistics expertise and incorporates pragmatic and world knowledge.
3 Evaluation Data
The evaluation data spans four languages and combines SNLI-based multilingual tracks with a cross-lingual MT quality-estimation track. Data preparation includes translation, pair selection, and restrictions against using gold quality-estimation labels.
- SNLI data: SNLI supplies entailment-labeled image-caption pairs, but the authors construct new pairings so gold entailment labels cannot directly inform STS evaluation.
- MT quality estimation: Track 4b relates STS to MT quality estimation using Spanish translations from rule-based, statistical, hybrid, and human translation systems.
- MT quality estimation: Participants cannot use gold quality-estimation annotations to inform their STS scores.
- MT quality estimation: HTER measures the minimum edits needed to correct a translation divided by its corrected length.
- 3.1 Tracks: Six evaluation tracks span Arabic, English, Spanish, and Turkish, with Track 4 divided into SNLI-based 4a and WMT quality-estimation 4b.
- Translation: SNLI sentences are independently translated into Arabic, Spanish, and Turkish for multilingual and cross-lingual evaluation.
4 Annotation
Most STS pairs receive crowdsourced labels, while the MT quality-estimation track uses a single bilingual expert because translation errors create subtler annotation challenges.
- Annotation procedure: Crowdsourcing supplies STS labels for most tracks, whereas Track 4b uses a single expert annotator.
- Crowdsourced annotation: Mechanical Turk workers label English SNLI pairs, and those labels are transferred to translated pairs in cross-lingual and non-English tracks.
- Crowdsourced annotation: Five annotations are collected per pair, and the gold score averages the individual annotations.
- Expert annotation: Track 4b uses a native Spanish, fluent English graduate student because MT pairs are highly similar and their error differences can be subtle.
5 Training Data
Training data combines prior STS evaluations with translated and newly annotated material across English, Spanish, Spanish-English, and Arabic conditions. The data includes label-scale differences and substantial variation in sentence length and similarity distributions.
- Training-data coverage is organized into English, Spanish, Spanish-English, Arabic, and Arabic-English tables, with separate Arabic-English parallel data supplied by translating English training data.
- Spanish data from 2014 and 2015 collapses labels 4 and 3 on a five-point scale, removing their distinction between unimportant and important details.
- English, Spanish, and Spanish-English training data comes from prior STS evaluations, while Arabic and Arabic-English data is created by translating English training data and transferring similarity scores.
- 1,000 new Spanish-English STS pairs from the 2013 WMT translation task are expert annotated for MT quality estimation.The pairs were produced by a phrase-based Moses SMT system and have a label distribution similar to the track 4b test data.
- Track 4b sentences average 19.4 words, compared with 8.7 words for track 5 English data and 8.8–13.8 words across the listed English training years.
6 System Evaluation
The evaluation ranks STS systems by Pearson correlation across multilingual, cross-lingual, and MT quality-estimation tracks. ECNU leads overall, while results reveal cross-lingual degradation and especially difficult Arabic and MTQE conditions.
- Participation: 31 teams produced 84 submissions, including 17 teams with 44 systems participating in all tracks.English attracted the most specialized participation, while other teams focused on Spanish-English or Arabic tracks.
- Baseline: The baseline uses cosine similarity between binary word-presence sentence vectors and reaches an average correlation of 53.7 across tracks 1–5.For cross-lingual pairs, non-English sentences are translated into English before comparison; the baseline would rank 23rd among 44 all-track submissions.
- Rankings: ECNU ranks first overall with average r 0.7316 and leads Arabic-English, Spanish, and Turkish-English tracks.
- Rankings: SEF@UHH leads the difficult WMT quality-estimation track 4b with r 0.3407, while RTV leads English track 5 with r 0.8547.
- Evaluation limitations: MT quality-estimation evaluation may benefit from metrics beyond Pearson correlation because Spanish-English SNLI performance is much higher than track 4b MTQE performance.
- Cross-lingual performance: Cross-lingual results generally decrease, with the baseline dropping more than 10% relative on Arabic-English and Spanish-English SNLI versus monolingual Arabic and Spanish.
- Methods: Systems combine surface, alignment, MT-evaluation, conceptual-similarity, and embedding features, often using MT for cross-lingual and non-English tracks.
7 Analysis
Analysis shows that top systems perform best on English, with broader errors in Arabic and weak agreement on Spanish-English MT pairs. Difficult cases expose failures in disambiguation, composition, negation, and semantic blending, while STS and MT quality scores capture partly different properties.
- Model–human comparison: Approximately 80% of top English-system scores fall within 1.0 point of human labels, whereas Arabic errors are more broadly distributed.The Arabic dispersion is especially pronounced for model scores between 1 and 4.
- Model–human comparison: Spanish-English MT plots show a weak relationship between predicted and gold STS scores.Cross-lingual errors can be magnified when systems translate sentences before applying a monolingual similarity model.
- Common errors: Difficult sentence pairs reveal errors involving word-sense disambiguation, attribute importance, compositional meaning, negation, and semantic blending.Examples include reversing agent roles, treating antonymic forms as equivalent, and bringing “boy” closer to “girl” through appended context.
- Cross-lingual STS and MTQE: 53% of MT quality-estimation test instances receive an STS score of 5, while 13% score below 3.The distribution also includes 22% at score 3 and 12% at score 4.
- Cross-lingual STS and MTQE: The Pearson correlation between gold MT quality and gold STS scores is 0.41, indicating only moderate alignment between the measures.MT quality penalizes all source–translation mismatches, whereas STS focuses on meaning differences.
8 STS Benchmark
The STS Benchmark provides a carefully selected, partitioned English evaluation resource from 2012–2017 shared-task data. It supports comparable model assessment, while results show 2017 systems outperforming strong sentence-embedding baselines but leaving substantial room for improvement.
- Benchmark construction: The STS Benchmark selects English data from SemEval and *SEM STS shared tasks conducted between 2012 and 2017.The benchmark is intended as a common evaluation set for semantic-representation research.
- Benchmark construction: The benchmark partitions data into training, development, and test sets for model design, tuning, and evaluation.The test set should be used sparingly after model design and hyperparameters are fixed.
- Purpose: Using the STS Benchmark enables comparable assessments across research efforts and improved tracking of state-of-the-art performance.Its standardized partitions make results easier to compare over time.
- Baselines: Baseline sentence embeddings compute sentence representations by averaging word embeddings and then applying cosine similarity.The implementation discussion covers embeddings including GloVe, LexVec, Word2Vec, FastText, and Paragram.
- Results: Top 2017 STS shared-task systems outperform state-of-the-art sentence-embedding baselines on the benchmark.The paper nevertheless reports substantial room for further improvement.
9 Conclusion
The 2017 STS shared task shifted substantially toward multilingual and cross-lingual evaluation across Arabic, Spanish, English, and Turkish. It also introduced the STS Benchmark to support standardized evaluation of English STS models.
- The task’s primary emphasis shifted from English STS to multilingual and cross-lingual evaluation.
- 31 teams submitted 84 systems, while 17 teams submitted 44 systems covering all four 2017 languages.
- Systems achieved strong human correlations for previously evaluated languages but weaker correlations for Arabic, Arabic-English, and Turkish-English.
- The results indicate a need for robust models that transfer to new languages and settings with less supervised training data.
- The STS Benchmark selects English datasets from the 2012–2017 tasks and evaluates competitive baselines and participant systems on them.