BiSECT: Learning to Split and Rephrase Sentences with Bitexts

1 Introduction

• Split and Rephrase addresses the need to turn long, complex sentences into shorter ones, a step used in text simplification.The passage notes that machine translation and dependency parsing also tend to perform poorly on long sentences.Source: §1 · p. 1
• Around 25% of WIKISPLIT pairs contain significant errors, as Wikipedia edits can change the input sentence’s meaning while splitting it.The paper attributes this noise to editors making changes for purposes beyond sentence splitting.Source: §1 · p. 1
• BiSECT combines bilingual-pivot corpus construction with a model designed to target the amount of rephrasing a split requires.The method extracts 1-2 and 2-1 sentence alignments from parallel corpora and machine-translates the foreign sentences into English.Source: §1 · pp. 1–2
• BiSECT is presented as higher quality than WIKISPLIT, with a wider variety of splitting operations and better model outputs for Split and Rephrase.The paper also introduces a classification task for the types of splitting outputs, based on how much rephrasing is needed.Source: §1 · p. 2

2 Related Work

• WEBSPLIT derives shorter sentences from subsets of RDF triples, but its limited vocabulary is associated with unnatural expressions and repeated syntactic patterns.Later corpora addressed training and evaluation needs; WIKISPLIT, mined from Wikipedia edit histories, is the main comparison in this work.Source: §2 · p. 2
• Unlike Wikipedia-edit approaches, BiSECT is created from aligned parallel documents, whose edits need not preserve the original meaning.Concurrent work used a WIKISPLIT subset for sentence decomposition and could transfer across languages.Source: §2 · p. 2
• Bilingual pivoting has supported paraphrase resources and neural approaches to sentential paraphrasing and sentence compression.The technique assumes that English phrases translating to the same foreign phrase have similar meaning.Source: §2 · p. 2
• Earlier Split and Rephrase models used LSTMs and copy-attention; this work starts from a BERT-initialized transformer and uses adaptive loss to emphasize split-based edits.Concurrent work also introduced a neural graph approach.Source: §2 · p. 2

3.1 Corpus Creation Procedure

• For English BISECT, the authors extract all 1-2 and 2-1 sentence alignments from parallel corpora with English as A and a foreign language as B, then translate foreign sentences into English.The testing set uses different corpora from the training set to prevent domain overlap.Source: §3.1 · p. 3
• Filtering removes malformed sentence alignments and pairs with lexical-overlap ratio below 0.25, a missing verb in any sentence, or BERTScore below 0.4.Source: §3.1 · p. 3
• The filtered English corpus contains 938,102 pairs and can grow as additional OPUS parallel corpora are processed.Source: §3.1 · p. 3

3.2 Comparison to Existing Corpora

• BiSECT contains longer aligned pairs and a higher %new score than WIKISPLIT, indicating more complex pairs with more rephrasing.Source: §3.2 · pp. 3–4
• Manual inspection found significantly more high-quality pairs and fewer pairs with significant errors in BiSECT than in WIKISPLIT.The assessment compared 100 randomly selected pairs from each corpus.Source: §3.2 · p. 4
• Manual inspection found 77% of the sampled German BiSECT pairs to be high-quality.Source: §3.2 · p. 4

3.3 Categorization for Split and Rephrase

• The corpus categorizes rephrasing as Direct Insertion, Changes near Split, or Changes across Sentences, with finer syntactic subcategories.The categories were derived across web, newswire, medical, legal, and other domains.Source: §3.3 · p. 4
• DisSim produces erroneous splits in 55% of Changes near Split cases and 63% of Changes across Sentences cases, despite identifying split locations.It fails to modify sentences effectively when more expansive rephrasing is required.Source: §3.3 · pp. 4–5

4 Our Model

• The approach targets variable rephrasing needs because some pairs require only edits near the split, while others need broader changes to preserve fluency.Linguistic rules assign sentence categories and identify the edits used to guide the model.Source: §4 · p. 5, §4.2 · p. 6

4.1 Base Model

• The base system is a BERT-initialized Transformer trained with cross-entropy, with [SEP] marking the boundary between split reference sentences.Its encoder and decoder follow the BERTbase architecture, and the encoder uses a BERT checkpoint for initialization.Source: §4.1 · p. 5

4.2 Adaptive Loss using Distant Supervision

• The model predicts split categories and generates output jointly, using the encoder’s [CLS] representation for category prediction.The three categories are Direct Insertion, Changes Near Split, and Changes Across Sentence; the rules classified 83% of 100 manually inspected training pairs correctly.Source: §4.2 · pp. 6–7
• The adaptive loss uses category-specific edit labels to focus generation on split-related changes, copying source words elsewhere.For Direct Insertion, edits cover the split and adjacent positions; Changes Near Split uses a variable window, while Changes Across Sentence marks all positions as editable.Source: §4.2 · p. 7

5.1 Data and Baselines

• The evaluation uses BISECT and HSPLIT-WIKI test sets to represent high-rephrasing and minimal-rephrasing splits, respectively.The manually selected BISECT test set contains 583 high-quality splits; WIKISPLIT is used for training but not evaluation because of inherent noise and biases.Source: §5.1 · p. 7

5.1 Data and Baselines; 5.2 Automatic Evaluation

• The authors compare Copy512 and a WIKISPLIT-trained Transformer baseline with models trained on BISECT.BLEU and SAMSA can underestimate models that rephrase often, so the automatic evaluation focuses on BERTScore and SARI, including an extension that recognizes lexical paraphrases.Source: §5.1 · p. 7, §5.2 · pp. 7–8

5.2 Automatic Evaluation; 5.3 Human Evaluation

• BISECT-trained models outperform WIKISPLIT-trained equivalents on SARI and BERTScore, and receive better human quality ratings; adaptive loss performs better in four of seven split categories.The human-evaluation advantage is especially evident for Preceding Relative Clause splits, which require changes across sentences.Source: Table 5 · p. 8, §5.2 · p. 8, Table 6 · p. 8

5.3 Human Evaluation

• Annotators rated sentence-split quality on a 0–100 scale, from erroneous to perfect meaning-preserving splits, using 100 random sentences drawn from the BISECT and HSPLIT-WIKI test sets.Only one rating was collected because grammatical and meaning-changing errors were difficult to distinguish; randomly corrupted control outputs received low ratings (<20).Source: §5.3 · pp. 8–9

6 Conclusion

• BISECT is a multilingual corpus for Split and Rephrase, created by translating instances of aligned split sentences from bilingual parallel corpora.The conclusion describes the resource as covering several languages.Source: §6 · p. 9
• Models trained on BISECT generate fewer errors than counterparts trained on existing datasets.Source: §6 · p. 9
• The proposed BISECT-trained approach outperforms existing systems in both automatic and human evaluations.The model uses split-based edits to further improve meaning preservation and diversity.Source: §6 · p. 9

A Implementation and Training details

• The BERT-initialized Transformer uses a BERTbase encoder, a randomly initialized decoder, and a three-class sentence classifier.Training uses Adam for 100k steps; inference uses beam search of width 10 with trigram repetition blocked.Source: A Implementation and Training details · p. 12

B BiSECT Language Composition

• Table 8 presents the English BiSECT corpus composition by pivoted language over bilingual parallel corpora.Source: Table 8 · p. 13

C Examples from Different Corpora

• Table 9 gives random sentence-pair examples from existing corpora, marking split positions, good edits, and hallucinations.Source: Table 9 · p. 13

• Training uses only split-related edits in y, determined by edit δ and sentence-category labels l.Source: Figure 3 · p. 14
• For a Changes Near Split training pair, Figure 3 illustrates padding and the construction of δ.Source: Figure 3 · p. 14

• Table 10 presents BISECT test-set outputs for a source sentence in the Changes Near Split category.Source: Table 10 · p. 15
• In Table 10, blue marks required edit locations, while green and red indicate good edits and errors, respectively.Source: Table 10 · p. 15

• Figure 4’s annotation interface presents each system output with a slider ranging from 0 to 100, labeled “Very Low Quality” on the left and “Perfect Quality” on the right.Source: Figure 4 · p. 16
• Each HIT includes a control text with 4 to 8 randomly substituted words, which workers are expected to score low.Source: Figure 4 · p. 16
• Outputs are shuffled for every HIT to eliminate position bias; highlighted additions and red ticks expose changes from the source.Source: Figure 4 · p. 16

G.1 French

• Table 11 reports statistics for OPUS datasets used to create the French version of BiSECT.Source: Table 11 · p. 17

G.2 Spanish

• Table 12 reports statistics for OPUS datasets used to create the Spanish version of BiSECT.Source: Table 12 · p. 17

G.3 German

• Table 13 reports statistics for OPUS datasets used to create the German version of BiSECT.Source: Table 13 · p. 17