PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural CRF Model for Sentence Alignment in Text Simplification

Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, Wei Xu

arXiv:2005.02324v4cs.CL

TL;DR

Text simplification depends on reliable complex-simple sentence pairs, yet commonly used corpora contain alignment errors and surface-based matches that miss paraphrases and context. The paper introduces manually annotated datasets and a neural CRF aligner combining semantic similarity with document-level sequence structure. The aligner improves sentence alignment by more than 5 points in F1, and derived datasets support state-of-the-art simplification results.

  • Problem

    Existing text simplification corpora rely on automatically aligned sentence pairs that can miss paraphrases and context, while WIKILARGE contains many errors.

  • Method

    The paper creates manually annotated Newsela and Wikipedia alignment datasets and uses a neural CRF model with fine-tuned BERT semantic similarity and sequential document structure.

  • Results

    The aligner outperforms previous monolingual sentence alignment approaches by more than 5 points in F1, and a Transformer trained on the resulting datasets achieves state-of-the-art simplification performance.

  • Takeaways & Limitations

    The resulting NEWSELA-AUTO and WIKI-AUTO datasets are larger and higher quality, enabling improved automatic and human-evaluated text simplification.

  • Takeaways & Limitations

    The alignment construction assumes that sentence pairs outside aligned paragraphs are unaligned, causing a small number of missing alignments that require manual correction.

Abstract

from arXiv · show

The success of a text simplification system heavily depends on the quality and quantity of complex-simple sentence pairs in the training corpus, which are extracted by aligning sentences between parallel articles. To evaluate and improve sentence alignment quality, we create two manually annotated sentence-aligned datasets from two commonly used text simplification corpora, Newsela and Wikipedia. We propose a novel neural CRF alignment model which not only leverages the sequential nature of sentences in parallel documents but also utilizes a neural sentence pair model to capture semantic similarity. Experiments demonstrate that our proposed approach outperforms all the previous work on monolingual sentence alignment task by more than 5 points in F1. We apply our CRF aligner to construct two new text simplification datasets, Newsela-Auto and Wiki-Auto, which are much larger and of better quality compared to the existing datasets. A Transformer-based seq2seq model trained on our datasets establishes a new state-of-the-art for text simplification in both automatic and human evaluation.

1 Introduction

The paper addresses unreliable sentence alignments in text simplification corpora by combining manually annotated data with a neural CRF aligner. The resulting datasets improve both corpus scale and simplification performance.

  • Motivation: Text simplification training depends on high-quality complex-simple sentence pairs, but existing corpora often use surface-level alignment that misses paraphrases and sentence context.Such alignments are also associated with conservative models that mostly delete content and rarely paraphrase.
  • Datasets: The authors create manually annotated alignment datasets for Newsela and Wikipedia to support systematic training and evaluation.The datasets contain 50 Newsela article sets and 500 Wikipedia article pairs.
  • Alignment model: The neural CRF aligner combines semantic similarity with the sequential structure of parallel documents and captures multi-sentence editing operations.The approach uses fine-tuned BERT for semantic similarity and models operations such as splitting and elaboration.
  • Results: >5 points in F1 separate the proposed aligner from previous monolingual sentence alignment approaches.The comparison covers approaches by Stajner et al., Paetzold et al., and Xu et al.
  • Results: 666,645 and 488,332 sentence pairs comprise the NEWSELA-AUTO and WIKI-AUTO datasets, respectively.The datasets are constructed from 1,882 Newsela article sets and 138,095 Wikipedia article pairs.
  • Results: 3.4% higher SARI and 25% more rephrasing are achieved by a BERT-initialized Transformer trained on the new datasets.The SARI comparison is against the state of the art, while the rephrasing comparison is against models trained on existing datasets.

2 Neural CRF Sentence Aligner

The neural CRF aligner combines BERT-based sentence similarity with sequential alignment structure, modeling transitions and paragraph context to align sentences in parallel documents.

  • Problem Formulation: Each simple sentence receives a complex-article alignment index, with 0 representing no alignment; the full article alignment is a label sequence.The model uses a variable number of labels for each article pair rather than a fixed label set.
  • Neural CRF Sentence Alignment Model: The alignment score decomposes into semantic similarity between sentence pairs and a transition score for the current label following the previous label.The transition score is computed from features describing label distance and unaligned-sentence patterns through a two-layer feedforward network.
  • Neural CRF Sentence Alignment Model: The model scores sentence alignments using BERT semantic similarity and pairwise transitions between neighboring alignment labels.Its transition features capture document order and whether adjacent sentences are aligned.
  • Neural CRF Sentence Alignment Model: Inference uses the Viterbi algorithm in O(mn^2) time, while training’s alignment normalization is computed with the forward algorithm in O(mn^2) time.Both procedures operate over the sequence of alignment labels for the simple and complex documents.
  • Paragraph Alignment: Paragraphs are aligned before sentences using semantic similarity and relative position, improving efficiency and reducing the CRF label space.The paragraph algorithm also supports paragraph splitting or fusion when two continuous complex paragraphs correspond to one simple paragraph.

3 Constructing Alignment Datasets

The paper combines manual sentence-alignment annotation with a neural aligner to automatically construct larger, higher-quality Newsela and Wikipedia simplification corpora.

  • Annotation and automatic alignment: The alignment workflow first aligns paragraphs, then annotates candidate sentence pairs and trains the model to align remaining documents automatically.Paragraph-first alignment reduces annotation effort and label-space size; the procedure uses crowdsourced sentence labels and an alignment model trained on annotated data.
  • Corpus quality: Manual inspection finds at least 44% more complex rewrites and 27% fewer defective pairs in the new corpora than in existing corpora.Complex rewrites include deletion-plus-paraphrase or splitting-plus-paraphrase; defective pairs are not aligned or not simpler.
  • Manual annotation: Manual annotation covers 50 Newsela article groups and 500 Wikipedia document pairs, providing training data for automatic alignment.The Newsela annotation split is 35/5/10 for train/dev/test; Wikipedia annotation covers 10,123 sentence pairs.
  • Newsela corpus: NEWSELA-AUTO contains 666,645 predicted aligned or partially-aligned sentence pairs, compared with 141,582 pairs in the previous NEWSELA dataset.The new corpus is produced by automatically aligning the remaining 1,882 Newsela article groups.
  • Wikipedia corpus: WIKI-AUTO contains 604k non-identical aligned and partially-aligned sentence pairs generated from 138,095 English/Simple English Wikipedia document pairs.Document pairs are identified using Wikidata, and the alignment model is trained on annotations from 500 sampled pairs.

4 Evaluation of Sentence Alignment

The evaluation compares the neural CRF aligner with similarity-based and prior alignment systems on two manually annotated datasets, including ablations of its components.

  • Baselines: The neural CRF aligner is compared with Jaccard, TF-IDF, logistic regression, JaccardAlign, MASSAlign, and CATS.These baselines use lexical or character n-gram similarity with greedy or dynamic-programming alignment strategies.
  • Evaluation setup: The evaluation reports Precision, Recall, and F1 for two binary classification tasks, excluding identical sentence pairs as trivial cases.Task 1 groups aligned and partially-aligned pairs against not-aligned pairs; Task 2 groups aligned pairs against the other categories.
  • Main results: The neural CRF aligner outperforms previous monolingual sentence-alignment approaches by more than 5 points in F1 on NEWSELA-MANUAL.It performs particularly better on partial alignments involving sentence splitting and paraphrasing with deletion.
  • Main results: 85.1 F1 is achieved for Task 1 on WIKI-MANUAL, exceeding CATS by 15.1 points in F1.Task 1 classifies aligned or partially-aligned pairs versus not-aligned pairs.
  • Ablation study: 93.2 F1 and 88.1 F1 are achieved on Tasks 1 and 2, respectively, around 3 points above the variant without the CRF component.The authors associate sequential prediction and label-transition modeling with better handling of sentence splitting under dramatic rewriting.
  • Ablation study: Paragraph alignment raises Task 1 precision from 93.3 to 98.4 with a negligible recall decrease, while generated paragraph alignments approach gold-alignments performance.The F1 differences between generated and gold paragraph alignments are 0.9 on Task 1 and 0.3 on Task 2.
  • Ablation study: BERTfinetune performs better than Infersent, ESIM, BERTScore, and pre-trained BERT embedding for semantic similarity.For BERTScore, the evaluation uses idf weighting and treats the simple sentence as the reference.

5 Experiments on Automatic Sentence Simplification

The experiments compare simplification systems trained on NEWSELA-AUTO and WIKI-AUTO with systems trained on existing datasets using automatic and human evaluation. Models trained on the new datasets generally achieve stronger simplification quality, with the largest gains for BERT-initialized Transformers.

  • Models trained on the new datasets outperform equivalent models trained on existing datasets according to SARI.
  • 6.4% and 3.7% SARI increases were observed for Transformerbert on the NEWSELA-AUTO test set and TURK corpus, respectively.
  • Transformerbert trained on the new datasets outperforms existing state-of-the-art systems for automatic text simplification.
  • The new datasets improve addition and deletion F1 scores for LSTM-based models despite modest SARI gains.
  • Human evaluation rates Transformerbert trained on NEWSELA-AUTO higher than its old-dataset variant and the compared state-of-the-art systems on fluency, adequacy, and overall ratings.
  • Transformerbert trained on NEWSELA-AUTO performs 25% more paraphrasing and deletions than its variant trained on the previous NEWSELA dataset.

6 Related Work

Prior text simplification and sentence alignment systems relied heavily on rules, statistical methods, neural seq2seq models, and lexical similarity. These approaches often omit paraphrases and sentence splits, encouraging conservative deletion-based simplification and biased evaluation.

  • Text simplification research includes hand-crafted rules, statistical machine translation, and neural seq2seq models.
  • Existing simplification datasets built with lexical similarity metrics frequently omit paraphrases and sentence splits.
  • Training on such datasets produces conservative systems that rarely paraphrase.
  • Evaluation on these datasets favors deletion-based simplification over paraphrasing.
  • Earlier sentence alignment methods used TF-IDF cosine similarity, Jaccard similarity, or other lexical features before greedy or dynamic-programming alignment.

7 Conclusion

The paper concludes that its neural CRF aligner improves sentence alignment, its manually annotated datasets support training and evaluation, and its automatically constructed corpora enable stronger text simplification performance.

  • The neural CRF sentence alignment model substantially outperformed existing approaches.
  • The authors created NEWSELA-MANUAL and WIKI-MANUAL as high-quality manually annotated datasets for training and evaluation.
  • The neural CRF aligner was used to construct NEWSELA-AUTO and WIKI-AUTO, described as the largest sentence-aligned datasets to date.
  • A BERT-initialized Transformer trained on the new datasets established new state-of-the-art performance for automatic sentence simplification.

A.1 Implementation Details

The implementation uses BERT-based sentence representations within a neural CRF aligner and tailored paragraph-alignment procedures for Newsela and Wikipedia. Wikipedia preprocessing removes certain pages and short or colon-ending sentences, then filters repetitive or low-similarity pairs.

  • The neural CRF aligner uses a 12-layer BERTbase sentence encoder and the [CLS] representation for classification.The model uses cross entropy loss and updates weights in all BERT layers.
  • Paragraph-alignment thresholds were selected using development data from NEWSELA-MANUAL and WIKI-MANUAL.
  • Wikipedia preprocessing removes meta-pages, disambiguation pages, sentences shorter than four tokens, and sentences ending with a colon.
  • 13,036 article pairs contained simple articles with only one sentence after preprocessing and matching.
  • Regular expressions filter repetitive sentence patterns before BERT-based semantic-similarity filtering retains pairs above a development-tuned threshold.
  • After filtering, 970 aligned sentence pairs remained from the 13,036 one-sentence article pairs.

C Sentence Alignment on Wikipedia

This section evaluates sentence-alignment approaches on the WIKI-MANUAL dataset, reporting Task 1 performance on development and test sets. Several comparison systems first select candidate sentence pairs before applying their similarity metrics.

  • Task 1 distinguishes aligned or partially aligned sentence pairs from not-aligned pairs.
  • Performance is reported on both the WIKI-MANUAL development and test sets.Tables 12 and 13 contain the corresponding results.
  • MASSAlign, CATS, and two BERTfinetune methods use a candidate-pair selection step before similarity scoring.The section states that this procedure was found to improve their accuracy.

D.1 Implementation Details

The implementation uses Fairseq for Transformer and LSTM baselines, with BERT-based Transformer initialization and specified optimization and preprocessing choices. The LSTM baseline follows a prior encoder-decoder model with vocabulary and embedding preprocessing.

  • Fairseq implements the Transformer and LSTM baselines.
  • The Transformer uses BERTbase architecture for both encoder and decoder and initializes its encoder from a BERTbase uncased checkpoint.
  • Transformer training uses Adam, learning rate 0.0001, 40k warmup steps, and 200k total training steps.Data are tokenized with the BERT WordPiece tokenizer; other hyperparameters are listed in Table 14.
  • The LSTM baseline replicates Zhang and Lapata's encoder-decoder model and replaces named entities and words occurring fewer than three times during preprocessing.Predicted <UNK> tokens are replaced with aligned source words, and 300-dimensional GloVe embeddings initialize the embedding layer.

D.2 Human Evaluation

The paper uses crowdsourced interfaces and instructions for evaluating generated simplifications and correcting alignment labels. Its examples compare models trained on new versus existing Newsela data and contrast the best model with other baselines.

  • Amazon Mechanical Turk workers evaluate generated simplified sentences using instructions from Kriz et al. (2019).
  • Models trained on NEWSELA-AUTO rephrase input sentences more often than models trained on the older NEWSELA dataset.Table 16 compares LSTM and Transformerbert outputs, with bold marking deletions or paraphrases.
  • Table 17 compares Transformerbert and LSTM trained on NEWSELA-AUTO with EditNTS and Rerank on the old NEWSELA test set.Bold marks new phrases introduced by the model.
  • Crowdsourcing annotation on Figure Eight is illustrated with instructions and an example question.
  • A separate annotation interface supports correction of crowdsourced alignment labels.
Loading 2005.02324v4…