PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning To Split and Rephrase From Wikipedia Edit History

Jan A. Botha, Manaal Faruqui, John Alex, Jason Baldridge, Dipanjan Das

arXiv:1808.09468v1cs.CL

TL;DR

Split-and-rephrase models face training examples with unnatural expressions and limited vocabulary. The paper introduces WikiSplit, a million rewrites mined from Wikipedia edits, and reports that incorporating it more than doubles BLEU on WebSplit, from 30.5 to 62.4.

  • Problem

    WebSplit examples use fairly unnatural linguistic expressions and a small vocabulary, while neural encoder-decoder models perform poorly even with copying mechanisms.

    Source: §1 · p. 1

  • Method

    The authors mine one million split-and-rephrase examples from English Wikipedia edit histories and release them as the WikiSplit training dataset.

    Source: §1 · p. 1

  • Results

    BLEU on WebSplit rises from 30.5 to 62.4 when WikiSplit is incorporated into training, more than doubling Aharoni and Goldberg (2018)’s score.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The results show a large positive impact from training on large, diverse data with some noise, suggesting that progress may also come from finding other such data sources.

    Source: §4 · p. 5

    Further progress ideally requires evaluation data from naturally occurring sentences and a metric more sensitive to the task’s particularities.

    Source: §4 · p. 5

Abstract

from arXiv · show

Split and rephrase is the task of breaking down a sentence into shorter ones that together convey the same meaning. We extract a rich new dataset for this task by mining Wikipedia's edit history: WikiSplit contains one million naturally occurring sentence rewrites, providing sixty times more distinct split examples and a ninety times larger vocabulary than the WebSplit corpus introduced by Narayan et al. (2017) as a benchmark for this task. Incorporating WikiSplit as training data produces a model with qualitatively better predictions that score 32 BLEU points above the prior best result on the WebSplit benchmark.

1 Introduction

Split-and-rephrase breaks complex sentences into simpler sentences that retain their meaning, an important operation in text simplification. The paper introduces WikiSplit, mined from Wikipedia edits, and reports improved WebSplit performance when it is used for training.

  • Split-and-rephrase is a core text-simplification operation, and sentence splitting may benefit systems whose predictive quality degrades with sentence length.The cited examples include relation extraction and translation; the task's schema-free nature may also allow crowd-sourced supervision.Source: §1 · p. 1
  • The paper presents a scalable, language-agnostic method for extracting Wikipedia split-and-rephrase edits and releases WikiSplit with one million rewrites.The extracted examples have naturally expressed sentences and a rich, varied vocabulary, though they contain some noise.Source: §1 · p. 1
  • BLEU on WebSplit rises from 30.5 to 62.4 when WikiSplit is incorporated into training, more than doubling the result of Aharoni and Goldberg (2018).Source: §1 · p. 1

2 The WikiSplit Corpus

WikiSplit is a large corpus of sentence splits mined from Wikipedia edits. The paper describes its extraction and quality filtering, reports corpus size and noise, and contrasts its diversity and intended use with WebSplit.

2.1 Mining Wikipedia Edits

  • The extraction compares temporally adjacent Wikipedia snapshots in both directions, then filters candidate splits using trigram-prefix and suffix constraints and BLEU similarity.When multiple candidates remain, the method retains the one maximizing the sum of BLEU similarities to the two split sentences.Source: §2.1 · p. 2

2.2 Corpus Statistics and Quality

  • The similarity threshold δ=0.2 provides the best reported trade-off between quality and corpus size.Table 2 reports the quality-versus-size trade-off for a random sample of 100 examples.Source: Table 2 · p. 2, §2.2 · p. 3
  • WikiSplit contains 989,944 unique complex training sentences, compared with WebSplit's 16,938, after extraction from around 18 million English documents.The corpus has over one million examples overall; Table 3 summarizes training-corpus statistics for complex sentences, simple sentences, and tokens.Source: Table 3 · p. 3, §2.2 · p. 3
  • In a 100-example sample, 68% of split-and-rephrase examples were perfect, while 32% contained noise from unsupported or missing information.The authors position this noisy extraction as training data and regard its use for evaluation as an imperfect signal.Source: §2.2 · p. 3

2.3 Comparison to WebSplit

  • WikiSplit has a 633k-word vocabulary across 33m tokens in distinct complex sentences, versus WebSplit's 7k-word vocabulary across 344k tokens.The authors describe WikiSplit text as more natural and diverse; WebSplit's small vocabulary is repetitive, and its examples are often unnatural.Source: §2.3 · p. 3
  • WikiSplit is harder than WebSplit because of greater diversity and sparsity, but potentially easier because each example uses a single split.Source: §2.3 · p. 3
  • WebSplit is better suited for evaluation because its construction yields cleaner data and multiple reference decompositions per complex sentence.Source: §2.3 · p. 3

3 Experiments

Experiments hold the model architecture fixed while varying training data, and show that WikiSplit improves WebSplit performance, including when combined with WebSplit training data. Manual evaluation and example predictions also favor the combined model's output quality.

3 Experiments

  • Using fixed Copy512 architecture, experiments compare WEBSPLIT, WIKISPLIT, and BOTH; training uses unique (C, S) pairs and one randomly selected WebSplit reference per complex sentence.SOURCE predicts the unmodified input, while SPLITHALF deterministically divides it into two equal-length token sequences.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1 Results

  • WEBSPLIT-only scored 35.3 BLEU on WebSplit validation but 4.2 on WikiSplit; WIKISPLIT-only scored 59.4 on WebSplit without in-domain data, and BOTH further improved WebSplit performance.Source: Table 4 · p. 4, §3.1 · p. 4
  • 32 BLEU points above the prior best result, the combined WikiSplit and downsampled WebSplit training set achieves the best BLEU on the WebSplit v1.0 test set.Source: Table 5 · p. 4, §3.1 · pp. 4–5
  • In blind manual evaluation, BOTH produced 95% correct simple sentences, while AG18 outputs were 22% accurate.The evaluation sampled predictions for 50 WebSplit validation inputs.Source: §3.1 · p. 4, Table 6 · p. 5
  • WIKISPLIT and BOTH examples are more coherent and faithful, while another shown output drops content, hallucinates common training-set elements, and can be incoherent.The combined model's example produces three fluent sentences despite the majority of its training examples having two-sentence outputs.Source: §3.1 · p. 4, Table 7 · p. 5

4 Conclusion and Outlook

The authors report strong gains from large, diverse, somewhat noisy training data, and argue that further progress also requires naturally occurring evaluation data and a more task-sensitive metric.

  • Large, diverse training data containing some noise had a large, positive impact on split-and-rephrase.Source: §4 · p. 5
  • Future improvements may come from finding other large, diverse data sources as much as from modeling.Source: §4 · p. 5
  • WikiSplit is intended for training; further progress ideally needs naturally occurring evaluation data and a metric more sensitive to task particularities.Source: §4 · p. 5

Related papers

Loading 1808.09468v1…