Discourse-Aware Neural Rewards for Coherent Text Generation

1 Introduction

• Cross-entropy does not always align with common evaluation measures, while local n-gram scores may improve without improving overall coherence or discourse structure.Prior studies also report misalignment between commonly used measures and desired generation quality (Rennie et al., 2017; Li et al., 2016).Source: §1 · p. 1
• The paper addresses the challenge of defining a global score for complex text quality beyond local n-gram patterns with automatically constructed, unsupervised neural rewards.The approach shares the spirit of learning neural scores to approximate human judgments of dialogue quality, but does not require human judgments.Source: §1 · p. 1
• Neural rewards trained to capture cross-sentence ordering approximate desired document discourse structure and guide a generator trained with self-critical reinforcement learning.The paper also introduces a method for distributing sentence-level rewards for more accurate credit assignment.Source: §1 · pp. 1–2
• The learned discourse-aware rewards yield text that is more coherent and less repetitive than models trained with cross-entropy or commonly used scores.The evaluation uses cooking recipes, automatic overlap metrics, and human judgments.Source: §1 · p. 2

2 Neural Teachers

• Two neural teachers score ordered sentence sequences as approximations of discourse structure; they are trained offline on gold sequences without supervision and remain fixed during policy learning.Their scores formulate rewards for policy-gradient systems generating coherent long text.Source: §2 · p. 2

2.1 Notation

• A document S comprises n sentences, with each sentence s_j containing L_j words.Source: §2.1 · p. 2

2.2 Absolute Order Teacher

• The absolute order teacher encodes sentence sequences in forward and reverse order, minimizing their cosine similarity to capture temporal coherence.It uses sentence-level bag-of-words representations so the encoder captures sentence ordering rather than word ordering; its learned score rewards generated sequences for ordering similarity to gold.Source: §2.2 · pp. 2–3

2.3 Relative Order Teacher

• The relative order teacher scores subsequences to learn local ordered-sentence patterns, addressing the equal-reward assumption that can penalize appropriate subsequences.It trains on gold subsequences of lengths ℓ between ℓ_min and ℓ_max and is intended to reward temporal coherence among neighboring sentences.Source: §2.3 · p. 3

3.1 Notation

• The paper denotes recipe title words and ingredient phrases as separate input sets, with projection matrices and bias vectors used in the following formulation.Each ingredient may be a single word or a multiword phrase.Source: §3.1 · p. 3

3.2 Encoder

• Title words are represented as a bag of embeddings, while ingredient phrases are encoded as bags of embeddings and passed through a bidirectional GRU.The encoder concatenates the title and ingredient representations.Source: §3.2 · p. 3

3.3 Decoder

• The decoder is a separate GRU initialized from the encoder output and pretrained to predict the next recipe token by negative log-likelihood.At each step it receives both the current token embedding and the encoder representation.Source: §3.3 · pp. 3–4
• During reward-guided generation, sampled recipe sentences are passed to a teacher for sentence-level rewards, alongside a greedily decoded baseline recipe.The task generates full recipe text from a title and ingredient list.Source: §3 · p. 3, Figure 3 · p. 4

4.1 Self-critical sequence training

• The model learns a policy that receives teacher rewards for sampled sequences exceeding the reward of a greedily decoded sequence.The sampled sequence is generated token by token; the teacher supplies token rewards, while the greedy sequence serves as a reward baseline.Source: §4.1 · p. 4

4.2 Rewards

• Absolute ordering rewards compare a generation with forward and reverse gold order, while relative ordering scores subsequences against nearby gold-recipe sentences.The relative-order score is bounded in [−1, 1].Source: §4.2 · pp. 4–5
• Absolute-order training gives every token the same sequence reward, whereas relative-order training assigns each token the reward for its own sentence.Relative rewards reflect how a sentence imitates nearby sentences in the gold recipe.Source: §4.2 · p. 5

4.3 Mixed Training

• Optimizing teacher reward alone can yield simple, incoherent recipes, so mixed training balances the discourse-focused policy with the generator’s language-model objective.The mixing parameter γ lies in [0, 1].Source: §4.3 · p. 5

5.1 Datasets

• The Now You’re Cooking dataset uses the splits from Kiddon et al. (2016), with 109,567 training recipes and 1,000 recipes each for development and test.Source: §5.1 · p. 5

5.2 Training

• Teachers are fixed during generator training, and separate policies use absolute ordering, relative ordering, or relative ordering combined with BLEU-4 rewards.The mixed-reward model is selected by the development-set geometric mean of BLEU-4 and relative-order reward; the best mixed-objective models use γ = 0.97.Source: §5.2 · pp. 5–6

5.3 Baselines

• Baselines include a cross-entropy model and reimplementations of prior reinforcement-learning models, with additional γ settings tested to control low fluency from high-γ configurations.The selected baseline configurations are reported in Table 1, whose columns cover word-level, action-level, and state-change-level scores.Source: §5.3 · p. 6, Table 1 · p. 6

6.1 Overlap Metrics

• Action metrics evaluate matching actions, consecutive action subsequences, and the full action sequence; state-change metrics assess corresponding induced changes and their order.The evaluation maps recipe words to actions and actions to state changes, then compares those sequences with the gold recipe using BLEU-1, BLEU-4, and ROUGE-L.Source: §6.1 · p. 6
• Relative-ordering feedback improves action- and state-change sequence metrics over baselines, while word-overlap optimization yields its greatest gains on the word-overlap scores.The relative-ordering reward focuses on co-occurrences between recipe events, whereas word-level optimization favors common word sequences.Source: §6.1 · pp. 6–7

6.2 Human Evaluation

• For recipes over 100 words, the teacher-trained model’s performance rises over BLEU-1 on every human-evaluation question.Across all recipes, the model is consistently preferred to the cross-entropy baseline on aggregate, while BLEU-1 is preferred for fluency and action order.Source: Table 3 · p. 7, §6.2 · p. 7, Table 2 · p. 7

6.3 Insights

• The neural teacher rewards generations that better imitate gold recipe actions and discourages repetitive phrases, but can encourage common temporal patterns inappropriate to a particular recipe.The examples include completing steps that the MLE model misses and mentioning baking a pie when that action does not fit the recipe.Source: §6.3 · pp. 7–8, Table 4 · p. 8
• Comparing generated sequences with both forward and reverse gold orders dampens reward exploitation; mixing policy learning with maximum-likelihood learning also prevents language-model deterioration.Without reverse-order comparison, the generator learns simple high-reward sequences such as “Serve.”Source: §6.3 · p. 8
• Higher ℓmax improves global-coherence scores, while γ must balance teacher guidance against language-model deterioration.The study fixes ℓmin = 3, varies ℓmax from 3 to 6 and γ from 0.95 to 0.98, and reports better global-coherence performance at higher ℓmax.Source: §6.3 · pp. 8–9, Figure 5 · p. 9

• Earlier generation work emphasized architectures with structural biases, whereas this work uses a simple architecture and focuses on improving learner optimization.Source: §7 · p. 9
• Exposure bias from misaligned training and testing was addressed by augmenting training data with examples generated by pretrained models.Bengio et al. (2015) outlined the limitation, and subsequent work used generated examples to make models more robust to their own errors.Source: §7 · p. 9
• Policy-gradient generation has used global sequence rewards such as ROUGE, CIDEr, and BLEU, while this work rewards captured discourse semantics with a neural teacher.Source: §7 · p. 9
• Prior neural and embedding rewards largely use single-sentence similarity for short tasks, whereas these teachers reward long-range ordering patterns and implicitly learned recipe scripts.The paper connects its teachers to script knowledge learned from long texts, including recipes.Source: §7 · p. 9

8 Conclusion

• Absolute- and relative-ordering teachers score adherence to discourse structure and reward recipe generation for capturing cooking-domain temporal semantics.The teacher-trained generator better models latent recipe event sequences, and human evaluation attributes the improvement mainly to semantic coherence in longer recipes.Source: §8 · p. 9

A Hyperparameters

• Models are selected by the metric they were trained on, or by lowest loss for the MLE model.Source: A Hyperparameters · p. 12

• Recipes are batched by token and ingredient counts, using a minibatch size of 32.Source: §A.1 · p. 12

• The reward generator has a hidden size of 100.Source: §A.2 · p. 12
• Word embeddings have dimensionality 100.Source: §A.2 · p. 12
• Dropout at rate 0.3 is applied between the bag-of-words and recurrent layers.Source: §A.2 · p. 12

A.3 Pretrained Recipe Generator

• The recipe generator uses 256-dimensional encoder and decoder hidden states and three 256-dimensional embedding sets for titles, ingredients, and text.Models are single-layer, use dropout 0.3, and train with Adam at η = 0.0003; β = 2 sharpens the output distribution, and scheduled sampling decays by 5% every 5 epochs up to 50%.Source: §A.3 · p. 12

A.4 Policy Learning

• Policy learning retains pretraining hyperparameters but reinitializes Adam, lowers the learning rate to η = 3 × 10−5, and omits scheduled sampling.Source: §A.4 · p. 12

B Baseline Selection

• Baseline γ settings are selected by development-set performance on each model’s training reward, and checkpoints by highest reward.The chosen γ values are 0.97 for BLEU-1, ROUGE-L, and CIDEr baselines, and 0.99 for BLEU-4.Source: B Baseline Selection · p. 12