Source-linked AI summary
Sentence Simplification with Deep Reinforcement Learning
Xingxing Zhang, Mirella Lapata
TL;DR
Sentence simplification seeks to reduce linguistic complexity while preserving meaning. The paper combines an encoder-decoder model with deep reinforcement learning and reports improved performance across three datasets over competitive systems.
Problem
Sentence simplification must reduce linguistic complexity while retaining the input's information and meaning.
Method
DRESS couples a recurrent encoder-decoder model with reinforcement learning that rewards simplicity, grammaticality, and semantic fidelity.
Results
The model achieves significant improvements over strong simplification models across three datasets.
Takeaways & Limitations
Reinforcement learning provides a way to inject prior knowledge into simplification and achieves good results across three datasets.
Takeaways & Limitations
The reward's use of SARI is constrained by scarce multiple references and noise in mostly automatically constructed datasets.
Abstract
from arXiv · showhide
Sentence simplification aims to make sentences easier to read and understand. Most recent approaches draw on insights from machine translation to learn simplification rewrites from monolingual corpora of complex and simple sentences. We address the simplification problem with an encoder-decoder model coupled with a deep reinforcement learning framework. Our model, which we call {\sc Dress} (as shorthand for {\bf D}eep {\bf RE}inforcement {\bf S}entence {\bf S}implification), explores the space of possible simplifications while learning to optimize a reward function that encourages outputs which are simple, fluent, and preserve the meaning of the input. Experiments on three datasets demonstrate that our model outperforms competitive simplification systems.
1 Introduction
Sentence simplification reduces linguistic complexity while preserving information and meaning. Recent systems learn complex-to-simple rewrites automatically, and this paper proposes reinforcement learning to optimize simplification-specific constraints.
- Sentence simplification aims to reduce linguistic complexity while retaining the original information and meaning.
- Common rewrite operations include lexical substitution, syntactic simplification, and deletion.
- Recent approaches treat simplification as monolingual text-to-text generation learned from complex-simple sentence pairs.
- The proposed model combines an encoder-decoder architecture with reinforcement learning to encourage simpler, meaningful, and grammatical outputs.The encoder reads the source into continuous representations, while the decoder generates the target sequence.
- Experiments on three datasets show significant improvements over strong simplification models across datasets.
2 Neural Encoder-Decoder Model
The neural simplification model uses an attention-based recurrent encoder-decoder to generate a simplified target from a complex source. The encoder represents the source sequence, while the decoder generates target words using prior outputs and a dynamic source context.
- The model predicts a simplified target sequence from a complex source sequence as an attention-based sequence-to-sequence problem.
- An LSTM encoder transforms the source sentence into a sequence of hidden states.
- A decoder LSTM generates one target word at a time, conditioned on previously generated words and a dynamic context vector.
- The neural network uses output and hidden-state parameters, while attention scores use vector dot products for efficiency.
- The context vector is a weighted sum of source hidden states, with weights determined by attention.
3 Reinforcement Learning for Sentence Simplification
DRESS uses reinforcement learning to counter encoder-decoder copying bias by rewarding simplifications that are simple, relevant, and fluent. REINFORCE samples outputs and updates the policy toward higher expected reward.
- Motivation: DRESS addresses encoder-decoder copying bias by using reinforcement learning to encourage diverse rewrite operations while preserving fluency and meaning.A generic encoder-decoder often copies the source or makes only trivial changes.
- Policy and training: The model treats generation as an agent taking vocabulary actions until EOS, then updates its policy using the reward received by the complete simplification.The encoder-decoder supplies the policy over action sequences.
- Reward: The reward combines simplicity, relevance, and fluency as r(Ŷ) = λS rS + λR rR + λF rF.The components are weighted by λS, λR, and λF and depend on the source, reference, and system output as specified.
- Reward: SARI rewards addition, copying, and deletion patterns that align system output with references while differing appropriately from the input.It is used because it correlates with human judgments of simplicity.
- Reward: Relevance is measured as cosine similarity between LSTM sentence encodings of the source and predicted target, while fluency uses normalized LSTM language-model probability.The fluency score is normalized to [0, 1] using the exponential of perplexity.
- REINFORCE algorithm: Because exact expected reward is intractable, training approximates it with one sampled action sequence and uses a learned baseline to reduce gradient variance.The baseline predicts expected future reward and is trained by mean squared error without back-propagating into encoder states or context.
4 Lexical Simplification
DRESS additionally models lexical simplification explicitly, using source-target alignments to encourage substitutions while interpolating them with the reinforcement-learning model to retain fluency.
- Lexical simplification: Lexical simplification explicitly targets replacement of complex words with simpler alternatives, complementing jointly learned rewrite operations.The authors motivate this because contextually natural predictions can still be poor substitutes for complex-sentence content.
- Lexical simplification: A pretrained complex-simple encoder-decoder provides probabilistic word alignments, represented by attention scores, for the lexical simplification model.The source is encoded into LSTM hidden states used with alignment scores to estimate lexical simplification probabilities.
- Integration: The lexical model is linearly interpolated with the reinforcement-learning model so substitution preferences can be combined with previously generated context and fluency.The interpolation weight η lies in [0, 1].
5 Experimental Setup
The evaluation tests DRESS across three automatically collated simplification datasets using standard automatic metrics and comparisons with established simplification systems.
- Datasets: Experiments cover WikiSmall, WikiLarge, and Newsela, combining Wikipedia-derived aligned pairs with professionally edited news simplifications.Newsela contains articles rewritten at multiple grade levels, while WikiLarge aggregates earlier Wikipedia simplification corpora.
- Datasets: WikiLarge contains 296,402 training sentence pairs after combining prior corpora and removing duplicates and development or test sentences.Its development and test sets contain 2,359 sentences with eight reference simplifications.
- Datasets: Newsela uses 94,208 training pairs, 1,129 development pairs, and 1,076 test pairs after removing overly similar level transitions.The corpus contains 1,130 news articles rewritten four times for children at different grade levels.
- Training: Training proceeds from encoder-decoder pretraining to reinforcement learning and lexical-model training, using shared hyperparameters across datasets.Reinforcement training uses stochastic gradient descent with λS = 1, λR = 0.25, and λF = 0.5.
- Evaluation: Automatic evaluation uses BLEU5 for similarity to references, FKGL for readability, and SARI for simplification quality.Lower FKGL indicates simpler output.
- Comparison systems: Comparisons include PBMT-R, Hybrid, and SBMT-SARI, with SBMT-SARI evaluated only using models trained on Wikilarge.Hybrid is described as state of the art on WikiSmall.
6 Results
Across three datasets, the experiments compare neural, reinforcement-learning, lexical-simplification, and traditional systems using automatic and human evaluations. DRESS-based models generally improve simplicity and rewriting, while trade-offs appear across fluency, adequacy, and other metrics.
- Newsela: All neural models achieve higher BLEU, lower FKGL, and higher SARI than PBMT-R on Newsela.Hybrid has the lowest FKGL and highest SARI; DRESS improves FKGL and SARI over EncDecA, while DRESS-LS improves BLEU but slightly worsens FKGL and SARI.
- Newsela: Human ratings on Newsela show DRESS and DRESS-LS outperforming PBMT-R and Hybrid on Fluency, Simplicity, and overall scores.DRESS-LS and DRESS are significantly better than EncDecA, PBMT-R, and Hybrid on Simplicity at p < 0.01.
- Rewrite analysis: Higher TER indicates more rewriting; Hybrid has the highest TER, while DRESS and DRESS-LS perform more deletion and other rewriting operations than EncDecA and PBMT-R.More deletions correlate strongly with shorter sentences, and reinforcement learning encourages deletion while retaining substitutions and shifts.
- WikiSmall: DRESS-LS is significantly better on Simplicity than PBMT-R, Hybrid, and the Reference on WikiSmall, while its combined rating is also significantly higher.It is on par with PBMT-R on Fluency and worse on Adequacy, though closer to the human Reference on Adequacy than PBMT-R or Hybrid.
- Wikilarge: On Wikilarge, DRESS-LS significantly outperforms all comparison systems on the combined human rating and is significantly better than all except Hybrid on Simplicity.It is better than Hybrid but worse than other comparison systems on Adequacy, and is on par with PBMT-R on Fluency.
7 Conclusions
The paper presents reinforcement learning as a way to jointly model simplicity, grammaticality, and semantic fidelity in text simplification. It also adds lexical simplification and identifies broader applications for the framework.
- The model jointly represents simplicity, grammaticality, and semantic fidelity to the input.
- A lexical simplification component further boosts performance.
- Across three datasets, reinforcement learning injects prior knowledge into simplification and achieves good results.
- The authors plan to explicitly model sentence splitting and extend simplification from individual sentences to entire documents.
- The reinforcement learning framework may also apply to sentence compression, programming-code generation, and poem generation.