Controllable Text Simplification with Explicit Paraphrasing

1 Introduction

• End-to-end systems mostly rely on deletion, with fewer than 1% of outputs split and fewer than 11.2% introducing new words, while professional editors combine deletion, paraphrasing, and splitting.The passage describes deletion-only simplification as suboptimal and unsatisfying.Source: §1 · p. 1
• Because simplification needs vary by audience, systems should control characteristics such as output length, readability, and number of split sentences.What counts as simplified for one group may not be acceptable to another.Source: §1 · p. 1
• The proposed hybrid combines linguistically motivated rules with neural models, integrating them dynamically through ranking and data augmentation.The rules offload some splitting and deletion decisions so the seq2seq model can learn lexical and structural paraphrases; earlier hybrid approaches applied rules deterministically before paraphrasing (Narayan and Gardent, 2014; Siddharthan and Mandya, 2014; Sulem et al., 2018c).Source: §1 · p. 2
• The model achieves overall better performance than state-of-the-art systems on SARI and other metrics in automatic and human evaluations.Its outputs are reported as more similar to those written by human editors.Source: §1 · p. 2
• The system controls operation extent by constraining the percentage of input words copied and selecting candidates with a desired amount of splitting or deletion.The paper also introduces NEWSELA-TURK, a Newsela test dataset with multiple human references for evaluating lexical paraphrasing.Source: §1 · p. 2

2 Our Approach

• The pipeline generates intermediate simplifications, ranks them, rewrites a selected candidate with the paraphrasing model, and reuses candidates for paraphrasing-focused data augmentation.The intermediate candidates have undergone splitting and deletion.Source: §2 · p. 2

2.1 Splitting and Deletion

• DisSim applies 35 hand-crafted grammar rules to generate hierarchically organized split candidates, while a neural module adds split and non-split outputs filtered by Jaccard similarity above 0.5.Candidates with compression ratios outside [0.5, 1.5] are excluded; in a pilot, DisSim split 92% of 100 complex training sentences longer than 20 words, with errors in 6.8% of those splits.Source: §2.1 · pp. 2–3

2.2 Candidate Ranking

• A pairwise neural ranker orders candidates using length, compression, similarity, and rule-application features, then sends the top-ranked candidate to the paraphrasing model.Its gold score is a length-penalized BERTScore that favors candidates with compression ratios close to the reference; λ is set to 1 in the experiments.Source: §2.2 · p. 3

2.3 Paraphrase Generation

• The paraphrasing model is a BERT-initialized Transformer encoder-decoder with an encoder-side copy mechanism trained jointly with generation as an auxiliary task.The copy ratio is supplied as input, and the decoder generates from token states adjusted according to predicted copy probabilities; experiments use the training set’s average copy ratio of 0.7 unless specified otherwise.Source: §2.3 · pp. 3–4
• For augmentation, filtered DisSim candidates that match the reference’s sentence-split count are paired with that reference to train paraphrasing alongside original sentence-reference pairs.Candidates scoring below 0.5 under the length-penalized BERTScore are discarded; φy and λ are set to 1 and 2, respectively, during this filtering.Source: §2.3 · p. 4

2.4 Controllable Generation

• The system selects candidates by desired length or split count for split- or delete-focused output, and changes the copy ratio to control paraphrasing while preserving the earlier splitting and deletion extent.Paraphrase-focused generation runs only the paraphrasing step, which is designed to avoid splitting and minimize deletion.Source: §2.4 · p. 4

3 Experiments

• Automatic and manual evaluations compare the approach with simplification models and test adaptation to paraphrasing and splitting without deletion.The operation-focused automatic tests include NEWSELA-TURK and split- and deletion-focused subsets of NEWSELA-AUTO.Source: §3 · p. 4, §3.1 · p. 5

3.1 Data and Experiment Setup

• NEWSELA-AUTO has 259,778/32,689/33,391 train/validation/test pairs (~30% splits); NEWSELA-TURK adds 500 sentences with four human paraphrases each.The paraphrase references were collected for sentences selected to favor paraphrasing, with no sentence splits and compression ratios between 0.8 and 1.2.Source: §3.1 · p. 5

3.2 Existing Methods

• Baselines include a BERT-initialized Transformer, EditNTS, an LSTM encoder-decoder, and Hybrid-NG; all were retrained on NEWSELA-AUTO.EditNTS predicts word-level edit operations, while Hybrid-NG combines probabilistic splitting and deletion with phrase-based lexical substitution.Source: §3.2 · p. 5

3.3 Automatic Evaluation

• Automatic evaluation uses SARI edit scores and measures paraphrasing, readability, sentence length, splits, compression, and output length.SARI’s add score measures n-gram additions; BLEU is omitted because it often does not correlate with simplicity.Source: §3.3 · p. 6
• Our model beats Transformerbert and EditNTS on SARI; it has lowest self-BLEU (48.7), FK (7.9), and %eq (0.4), and highest add (3.3) and %new (16.2).The authors report that these results make it the least conservative system and that its outputs better mimic human references.Source: Table 2 · p. 5, §3.3 · pp. 6–7

3.4 Human Evaluation

• Human ratings are best for fluency, simplicity, and overall quality; correct splits reach 90%, with syntactic-simplification fluency 4.19.Adequacy is also close to Transformerbert and EditNTS despite more paraphrasing.Source: §3.4 · p. 7, Table 6 · p. 7

4.1 System Ablations

• 3.2 SARI points separate the BERTScorelen ranking model from random candidate selection, while random selection also performs fairly well.The comparison is reported for the dev-set ablation study.Source: §4.1 · p. 7, Table 8 · p. 8
• Random candidate selection performs fairly well, suggesting the chosen sentence-splitting and deletion models produce good candidates.Source: §4.1 · p. 8
• Removing data augmentation or copy attention lowers SARI and new-word use, while using only DisSim or Transformer candidates remains close to the full model in SARI.The SARI drops are 1.0 points without augmentation and 2.6 points without copy attention; both variants also reduce new words by at least 3.0%.Source: §4.1 · p. 8

4.2 Error Analysis

• The error analysis manually labels 200 simplifications for good outputs, hallucinations, fluency errors, anaphora resolution, bad substitutions, and reference errors; categories can overlap.Table 7 provides examples of the categories, with red marking errors and blue marking good paraphrases.Source: §4.2 · p. 8, Table 7 · p. 8

Related approaches

• Earlier systems handled simplification operations in pipelines, whereas later neural systems performed them end-to-end, with controllability and performance costs noted by the authors.Source: §5 · pp. 8–9

Controllable simplification

• Prior controllable systems used input tokens, word-complexity losses, embedding constraints, or linguistic scoring to guide simplification.Source: §5 · p. 9

Single-operation research

• Operation-specific research covers lexical substitution, sentence splitting, and compression by removing irrelevant input content.Source: §5 · p. 9

• The hybrid sentence simplification approach performs better and produces more diverse outputs than existing systems.Source: §6 · p. 9
• The new data augmentation method is designed to encourage paraphrasing, and NEWSELA-TURK evaluates paraphrasing-focused simplifications.Source: §6 · p. 9
• The model controls attributes including sentence-split count, output length, and the number of words copied from the input.Source: §6 · p. 9

• The deletion-and-splitting and paraphrase components are separate Transformers; both use BERTbase architecture, and the former uses a BERTbase-initialized encoder.Deletion-and-splitting candidate generation uses beam width 10.Source: A Implementation and Training Details · p. 14
• Inference avoids repeated trigrams and aggressive deletion, retaining outputs with compression ratios from 0.9 to 1.2 and selecting checkpoints by dev-set SARI.Source: A Implementation and Training Details · p. 14
• The copy-attention network has three 1,000-node hidden layers; the pairwise ranker has three 100-node hidden layers and uses dropout of 0.2.The copy-attention network uses tanh and sigmoid activations; the ranker uses tanh hidden layers.Source: A Implementation and Training Details · p. 14
• The authors used Adam and did not perform extensive hyperparameter tuning, examining only a few learning rates for the ranker.The paraphrase model trained for 100,000 steps with learning rate 0.0001; the ranker trained for 10 epochs with learning rate 0.01.Source: A Implementation and Training Details · p. 14

B Annotation Interface

• Figure 2 presents annotation guidelines for the NEWSELA-TURK corpus alongside example questions.Source: Figure 2 · p. 15

C System Outputs

• Table 10 compares paraphrase-focused simplifications generated by the neural paraphrasing component with baseline outputs.The model skips splitting and deletion; cp is a soft constraint on how many input words can be copied.Source: Table 10 · p. 16

• The Newsela paraphrasing subset contains 8,371 complex-simple sentence pairs with compression ratio above 0.9 and no splits.Source: Table 11 · p. 16
• Table 11 evaluates models on this paraphrasing-focused Newsela test subset.Source: Table 11 · p. 16
• The models control paraphrasing extent by specifying the percentage of input words to copy as a soft constraint.Source: Table 11 · p. 16

• Figure 3 gives Mechanical Turk workers guidelines for evaluating simplified sentences.The interface is based on the one proposed by Kriz et al. (2019).Source: Figure 3 · p. 17
• Figure 4 gives workers guidelines specifically for evaluating sentence splitting.Source: Figure 4 · p. 17

• Automatic evaluation is reported on TURK for lexical paraphrasing and on ASSET for all three simplification operations.Source: Table 12 · p. 18, Table 13 · p. 18
• WIKI-AUTO provides 138,095 article pairs and 604k non-identical aligned or partially aligned sentence pairs for the complex-simple data.Simple sentences mapped to the same complex sentence are joined to capture splitting; pairs with BLEU above 0.9 or below 0.1 are removed.Source: F Evaluation on Wikipedia · p. 18
• TURK and ASSET provide validation and test sets with 8 and 10 human-written references, respectively, for the same sentences.The validation and test sentences are removed from the training corpus.Source: F Evaluation on Wikipedia · p. 18