Source-linked AI summary
BLEU is Not Suitable for the Evaluation of Text Simplification
Elior Sulem, Omri Abend, Ari Rappoport
TL;DR
BLEU is widely used for text simplification, but its suitability for structural operations such as sentence splitting is uncertain. The paper builds a manual splitting corpus and compares BLEU with human judgments across reference settings. It finds that BLEU often penalizes simplicity and is not informative for evaluating text simplification or sentence splitting.
Problem
BLEU is widely used for text simplification, although text simplification includes structural operations such as sentence splitting that may challenge its evaluation reliability.
Method
The authors manually compile the HSplit sentence-splitting corpus and perform correlation analyses with human judgments across standard and split-focused reference sets and multiple evaluation metrics.
Results
BLEU often negatively correlates with simplicity and is not informative or is misleading for text simplification and sentence-splitting evaluation.
Takeaways & Limitations
The findings motivate alternative methods for evaluating structural text simplification instead of using BLEU.
Abstract
from arXiv · showhide
BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS). TS includes both lexical and structural aspects. In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation. We manually compiled a sentence splitting gold standard corpus containing multiple structural paraphrases, and performed a correlation analysis with human judgments. We find low or no correlation between BLEU and the grammaticality and meaning preservation parameters where sentence splitting is involved. Moreover, BLEU often negatively correlates with simplicity, essentially penalizing simpler sentences.
1 Introduction
BLEU is widely used in text simplification despite known deficiencies, while sentence splitting introduces a major structural dimension that BLEU may not capture reliably. This paper examines BLEU using split-focused references and human-generated splitting data.
- BLEU is widely used for text simplification despite deficiencies documented in prior work.
- Sentence splitting rewrites one sentence as multiple simpler sentences while preserving meaning and is a major structural simplification operation.
- Sentence splitting is common in text simplification systems and is central to the Split-and-Rephrase task, where BLEU is used as the automatic metric.
- The study compiles HSplit, a human-generated sentence-splitting corpus, and performs correlation analyses with human judgments using two reference-set configurations.
- BLEU is often negatively correlated with simplicity and is not informative or is misleading for text simplification and Split-and-Rephrase evaluation.
2 Related Work
BLEU was developed for machine translation and has known limitations in correlating with human judgments. Prior text-simplification studies found inconsistent relationships between BLEU and human-rated simplicity or adequacy.
- The BLEU Metric: BLEU uses modified n-gram precision, multiple references, and a brevity penalty to evaluate reference-based text generation.
- BLEU limitations: BLEU may assign the same score to many potential translations, and its correlation with human judgments decreases when translation quality is low.
- BLEU limitations: Reported shortcomings of BLEU become especially relevant when structural operations are involved.
- BLEU in TS: In text simplification, prior studies reported positive correlation between BLEU and simplicity but no correlation with adequacy on a small set of rated outputs.
3 Gold-Standard Splitting Corpus
The authors construct a manually annotated sentence-splitting benchmark with multiple guideline settings and annotators, then examine how splitting behavior varies across the resulting HSplit corpora.
- The corpus contains source sentences modified by four annotators according to explicit sentence-splitting guidelines.
- Unlike a semi-automatically compiled dataset, HSplit is fully manual, assumes neither predetermined split points nor that every sentence should be split.
- Set 1 requires splitting as much as possible while preserving grammaticality, fluency, and meaning; Set 2 encourages splitting only when it simplifies the original.
- Annotators in both guideline sets may leave the source unchanged when splitting violates grammaticality, fluency, or meaning preservation.
- The four HSplit corpora each contain 359 sentences, and within-set differences in splitting exceed between-set differences.This suggests inter-annotator variability affects split counts more than whether simplicity is explicitly mentioned.
4 Experiments
The experiments compare automatic metrics with human judgments across standard and sentence-splitting reference settings. BLEU and iBLEU fail to capture simplicity and structural simplicity, while BLEU can behave mainly as a conservatism measure.
- Experimental setup: The study evaluates BLEU, iBLEU, -FK, SARI, and -LDSC against human ratings of grammaticality, meaning preservation, simplicity, and structural simplicity.The standard setting uses Simple Wikipedia and eight crowdsourced references; the HSplit setting uses HSplit as the reference set.
- Standard Reference Setting: BLEU produces much higher scores for Identity than HSplit: BLEU-1ref is 59.85 versus 43.90, and BLEU-8ref is 94.63 versus 73.03.The corresponding human-score decreases are smaller for grammaticality and meaning preservation, indicating that BLEU does not track relative simplicity well.
- Standard Reference Setting: BLEU and iBLEU negatively correlate with simplicity and structural simplicity, while their correlations with grammaticality and meaning preservation become weak or disappear when splitting is included.BLEU’s meaning-preservation correlation is considerably lower than -LDSC’s, and its grammaticality correlation is comparable to -LDSC’s when splitting is involved.
- Standard Reference Setting: BLEU correlates strongly with -LDSC: BLEU-1ref reaches 0.86 without splits and 0.52 with splitting, while BLEU-8ref reaches 0.82 and 0.55.These correlations suggest BLEU may primarily measure conservatism rather than simplification quality.
- Standard Reference Setting: SARI correlates positively with simplicity at 0.52 without splits and 0.26 across all systems and corpora, but correlates with structural simplicity in neither setting.The authors relate this pattern to SARI’s focus on lexical rather than structural simplification.
- HSplit as Reference Setting: With HSplit as the reference, BLEU correlates 0.57 with grammaticality, 0.11 with meaning preservation, -0.70 with simplicity, and -0.60 with structural simplicity.Sentence-level BLEU and iBLEU correlations are positive but lower than those obtained by LDSC.
- HSplit as Reference Setting: Even references focused on sentence splitting do not make BLEU capture the output’s simplicity or structural simplicity.This section’s results support the conclusion that BLEU is inadequate for evaluating text simplification when splitting is involved.
5 Conclusion
The paper concludes that BLEU is unsuitable for Text Simplification evaluation: it negatively correlates with simplicity and performs no better than, or worse than, reference-free -LDSC for grammaticality and meaning preservation.
- BLEU negatively correlates with simplicity in Text Simplification evaluation.
- For grammaticality and meaning preservation, BLEU is comparable to or worse than -LDSC, which requires no references.
- The findings argue against using BLEU for Text Simplification generally and sentence splitting specifically.
- The paper motivates alternative methods for evaluating structural Text Simplification.