Source-linked AI summary
EASSE: Easier Automatic Sentence Simplification Evaluation
Fernando Alva-Manchego, Louis Martin, Carolina Scarton, Lucia Specia
TL;DR
Sentence Simplification evaluation is hindered by fragmented tools and limited support for understanding diverse simplification behaviors. EASSE consolidates metrics, transformation analyses, quality-estimation features, datasets, and reports in a Python package. Experiments show that these resources improve comparison and interpretation of system performance, while further transformation and QE coverage remains needed.
Problem
Fragmented implementations, inconsistent requirements, and insufficient documentation hinder standardized automatic evaluation and comparison of Sentence Simplification systems.
Method
EASSE provides one Python package containing automatic metrics, word-level transformation analysis, reference-independent QE features, public datasets, and HTML reporting.
Results
EASSE’s metric, transformation, and QE analyses support comparison of simplification systems and help explain their overall automatic scores.
Takeaways & Limitations
Combining overall scores with transformation and QE analyses gives a more detailed understanding of each system’s simplification capabilities.
Takeaways & Limitations
Future work must extend transformation analysis to splitting and subject-verb-object reordering and add QE features such as dependency-tree depth.
Abstract
from arXiv · showhide
We introduce EASSE, a Python package aiming to facilitate and standardise automatic evaluation and comparison of Sentence Simplification (SS) systems. EASSE provides a single access point to a broad range of evaluation resources: standard automatic metrics for assessing SS outputs (e.g. SARI), word-level accuracy scores for certain simplification transformations, reference-independent quality estimation features (e.g. compression ratio), and standard test data for SS evaluation (e.g. TurkCorpus). Finally, EASSE generates easy-to-visualise reports on the various metrics and features above and on how a particular SS output fares against reference simplifications. Through experiments, we show that these functionalities allow for better comparison and understanding of the performance of SS systems.
1 Introduction
Sentence Simplification evaluation relies on diverse automatic metrics, but fragmented implementations and poor documentation hinder consistent use. EASSE addresses this by consolidating evaluation resources in one package to facilitate comparison and reproducibility.
- 1 Introduction: Automatic evaluation commonly combines machine-translation, simplicity, and readability metrics to assess simplification outputs.BLEU, SARI, and FKGL represent these three metric categories.
- 1 Introduction: Fragmented implementations, differing software requirements, and insufficient documentation make existing evaluation metrics difficult for researchers to use.Examples include corpus-level and sentence-level SARI implementations in different languages and SAMSA workflows requiring hard-coded paths.
- 1 Introduction: EASSE consolidates popular sentence simplification metrics, transformation analysis, quality-estimation features, datasets, and HTML reporting in one Python package.Its command-line interface provides access to these resources and is intended to facilitate evaluation and reproducibility.
2 Package Overview
EASSE packages complementary automatic metrics, transformation analyses, quality-estimation features, datasets, and reporting tools for sentence simplification evaluation. Its components cover both reference-based scoring and reference-independent analysis while exposing important dataset and metric caveats.
- 2.1 Automatic Corpus-level Metrics: EASSE wraps BLEU, SARI, SAMSA, and FKGL, covering n-gram overlap, paraphrasing-oriented simplicity, sentence splitting, and readability.SARI is re-implemented in Python with operation-level precision, recall, and F1 scores averaged across n-gram orders; corpus-level deletion also uses F1 to match the Java implementation.
- 2.2 Word-level Analysis and QE Features: Word-level analysis uses alignments to identify DELETE, MOVE, REPLACE, and COPY transformations and evaluates system annotations against reference annotations.The automatic annotation procedure achieved a micro-averaged F1 score of 0.61 against manual labels from four annotators.
- 2.2 Word-level Analysis and QE Features: Reference-independent QE features compare source and output behavior through compression ratio, Levenshtein similarity, sentence splits, and word-addition proportions.These features provide information about simplification behaviors not reflected in reference-based metrics such as SARI.
- 2.3 Access to Test Datasets: PWKP’s single reference is not ideal for automatic metrics requiring multiple references, such as SARI.TurkCorpus instead provides multiple simplification references for each original sentence.
- 2.3 Access to Test Datasets: EASSE provides PWKP, TurkCorpus, and HSplit, while supporting custom datasets and matching datasets to evaluation needs.TurkCorpus has eight references per source and is suited to SARI and multireference BLEU; HSplit targets sentence splitting and can provide an expected-performance upper bound.
- 2.4 Reporting: A single command generates a comprehensive HTML report comparing system outputs with human references using simplification metrics and analyses.The report integrates quantitative and qualitative evaluation in one output.
3 Experiments
Experiments use EASSE to compare diverse sentence simplification systems through automatic metrics, transformation analysis, QE features, and HTML reporting. The analyses show that metric rankings reflect different transformation behaviors and that QE features clarify system outputs.
- Automatic Metrics: The SAMSA comparison uses only the first 70 TurkCorpus sentences also released in HSplit, limiting the structural-simplification analysis at submission time.The full HSplit corpus was expected to become available in EASSE.
- Automatic Metrics: DMASS-DCSS is state-of-the-art on TurkCorpus by SARI, but has the lowest SAMSA and third-to-last BLEU scores; PBSMT-R leads the latter two metrics.Human references achieve the highest values across all metrics, significantly exceeding the top-performing systems.
- Word-level Transformations: SBSMT-SARI’s strong SARI score aligns with leading replacement performance and second-place copying, while PBSMT-R leads copying and DMASS-DCSS ranks second in replacements.Hybrid leads deletions but performs worst on replacements, the transformation SARI mainly measures.
- Automatic Metrics: TurkCorpus mainly contains paraphrases with virtually no deletions, so systems that copy words better more closely match its human simplification characteristics.This dataset composition helps explain the observed metric and transformation differences.
- Quality Estimation Features: Hybrid compresses inputs far more than other systems, with a 0.57 compression ratio versus ≥0.78, while adding new words in only 0.01 of cases.These QE values explain Hybrid’s high deletion and low replacement performance; DRESS-LS leaves inputs unchanged 26% of the time.
- Report: The HTML report compares DMASS-DCSS with human simplifications and shows close matches across QE aspects, helping explain its high SARI score.It combines quantitative comparisons with visualizations and qualitative displays of source sentences and their simplifications.
4 Conclusion and Future Work
EASSE supports comparison of sentence simplification systems through automatic metrics, word-level transformation analysis, and quality-estimation features. Experiments show that these analyses provide insights into systems’ simplification capabilities and help explain overall automatic scores, while future work targets more sophisticated transformations and additional QE features.
- EASSE enables comparison of different SS systems using automatic metrics, word-level transformation analysis, and QE features.
- Transformation analysis reveals each system’s simplification capabilities and helps explain its initial automatic scores.
- Future work will identify more sophisticated transformations, such as splitting and subject-verb-object reordering, and integrate QE features measuring additional aspects such as syntactic complexity.