Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

• RL offers a natural way to optimize non-differentiable scalar objectives, but imperfect learned metrics can reward nonsensical generations through Goodhart’s Law.Human-in-the-loop feedback is often prohibitively dense, motivating cheaper automated preference metrics despite their limitations.Source: §1 · pp. 1–2
• The authors release RL4LMs, a library for training HuggingFace generative models with on-policy reinforcement learning and arbitrary reward functions.The library supports models such as GPT-2 and T5 with methods including PPO and A2C.Source: §1 · p. 2
• GRUE evaluates reinforcement-learning algorithms on NLP generation tasks using reward functions instead of supervised target strings while requiring fluent language generation.The benchmark pairs each task with reward function(s) and supports training with or without task-specific supervised pre-training.Source: §1 · p. 2
• NLPO better balances preference rewards and language fluency than alternatives including PPO, as shown in GRUE experiments and human evaluations.In the sentiment-guided example, NLPO with KL regularization achieves higher reward without the perplexity cost of removing the KL penalty.Source: §1 · p. 2, Figure 1 · p. 2
• RL can be more data- and parameter-efficient than supervised learning: a reward signal with 5 times less data can outperform supervised training, and a 220 million-parameter NLPO model can beat a 3 billion-parameter supervised model.The strongest data-efficiency result combines reinforcement learning with supervision, and the parameter-efficiency comparison also uses combined supervision and NLPO.Source: §1 · p. 2

Imitation learning for NLP

• Imitation-learning methods inspired by DAGGER and SEARN address sequence generation but are known to suffer from exposure bias and the cliff MDP problem.Examples include Schedule Sampling (Bengio et al., 2015), LOLS (Lampouras & Vlachos, 2016; Chang et al., 2015), TextGAIL (Wu et al., 2021b), and SEARNN (Leblond et al., 2017).Source: §2 · p. 3

RL for Large Action Spaces

• RL methods for language generation have addressed variance and large action spaces using actor-critic approaches and action-space reduction during exploration.MIXER (Ranzato et al., 2016) combined schedule sampling with REINFORCE, while later methods included KG-A2C (Ammanabrolu & Hausknecht, 2020), TrufLL, AE-DQN (Zahavy et al., 2018), and GALAD (Ammanabrolu et al., 2022).Source: §2 · p. 3

RL for NLP

• RL has been applied across many NLP tasks and to preference alignment, but its stability and overall value relative to supervised LM training remain contested.Ouyang et al. (2022) used PPO (Schulman et al., 2017) for preference alignment, but their non-public dataset prevents comparison; RL has also been criticized as less stable than supervised training (Choshen et al., 2020).Source: §2 · p. 3

3 RL4LMS: A LIBRARY FOR TRAINING LMS WITH RL

• RL4LMs supports decoder-only and encoder-decoder HuggingFace models with stable-baselines-3 algorithms, customized environments and rewards, and four tailored algorithms across six NLP tasks.The initial release provides 16 evaluation metrics and rewards, including PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015a), A2C (Mnih et al., 2016), and NLPO.Source: §3 · p. 3

3.1 ENVIRONMENTS: GENERATION AS A TOKEN-LEVEL MDP

• RL4LMs models generation as a token-level Markov Decision Process, where vocabulary tokens are appended deterministically until a horizon or end-of-sentence token, then receive sequence-level rewards.Its OpenAI Gym-style API enables rapid addition of compatible NLP tasks and supports automated metrics such as PARENT (Dhingra et al., 2019).Source: §3.1 · pp. 3–4

3.2 REWARD FUNCTIONS AND EVALUATION METRICS

• The library provides per-token or per-sequence rewards spanning n-gram overlap, semantic, task-specific, diversity, fluency, naturalness, and human-preference metrics.Examples include ROUGE, BLEU, BertScore, BLEURT, PARENT, SummaCZS, perplexity, Distinct-1/2, and preference classifiers trained on human preference data.Source: §3.2 · p. 4

3.3 ON-POLICY ACTOR-CRITIC ALGORITHMS

• RL4LMs trains language models with on-policy actor-critic methods initialized from a pretrained policy and value network, using generalized advantage estimation and a dynamically adapted token-level KL penalty.The KL regularization addresses sparse sequence-level rewards by limiting divergence from the initialized LM π0.Source: §3.3 · p. 4

Motivation and core mechanism

• NLPO extends PPO with a learned mask that removes less relevant tokens in context during training.The method is motivated by language-generation vocabularies of 50K tokens for GPT-2/3 and 32K for T5, which greatly exceed typical discrete RL action spaces.Source: §4 · p. 4

Mask construction

• Top-p sampling constructs the mask by retaining the smallest token set whose cumulative probability exceeds p and zeroing probabilities for remaining tokens.The mask is derived from the current policy and applied when sampling actions during training.Source: §4 · pp. 4–5

Periodic policy updates

• NLPO maintains a masking policy πψ copied from the current policy πθ and updates it only every µ steps.This periodic update is inspired by off-policy Q-learning and constrains πθ while preserving more task-relevant information than a KL penalty from π0 and reducing reward-hacking risk.Source: §4 · pp. 4–5

Optimization procedure

• Each iteration collects trajectories with πψ, computes preference and KL-penalty rewards and advantages, then updates πθ and the value function with PPO-Clip.The parameterized masked policy is updated every µ iterations.Source: §4 · p. 5

5.1 RESULTS ON GRUE: WHICH ALGORITHM SHOULD BE USED TO LEARN PREFERENCES?

• Supervised+NLPO and Supervised+PPO usually outperform isolated RL or supervised training, while RL is strongest on text continuation and summarization and supervised training on Commongen and ToTTo.DailyDialog is the exception, with isolated RL outperforming warm-started variants; all methods outperform zero-shot on low-zero-shot tasks.Source: §5.1 · p. 6, Figure 2 · p. 7

5.2 PREFERENCE REWARD LEARNING, SELECTION, AND HACKING

• Removing KL constraints causes reward hacking, while using a supervised model as the regularizing base mitigates nonsense outputs on tasks where the pretrained model lacks task-specific signal.Warm-started policies are therefore critical for Commongen and ToTTo, where models otherwise repeat input fragments.Source: §5.1 · p. 7, §5.2 · p. 8
• NLPO generally outperforms PPO and supervised training, especially after supervised warm-starting, because its masking policy adds a task-relevant constraint beyond the KL penalty.Performance rises and then falls as top-p increases, indicating that masking strength requires balancing constraint and exploration.Source: §5.1 · p. 6, §5.2 · p. 8
• Human-preference optimization beats METEOR-only optimization, with preference-model generations preferred in 682 cases versus 587, while the reward model achieves 69.5 test ROC AUC.The comparison used 417 annotated training pairs and a T5-11B reward model trained on majority preferences.Source: §5.2 · pp. 8–9

5.3 DATA BUDGET: IMPROVE YOUR REWARD OR GATHER MORE DEMONSTRATION?

• Improving a learned reward model yields greater LM performance than supervised training with five times more data, making reward improvement more data-efficient than collecting demonstrations.On IMDB, better classifier accuracy produces better RL performance, evaluated with the full-data sentiment metric.Source: §5.3 · p. 9

5.4 PRACTICAL CONSIDERATIONS: WHICH IMPLEMENTATION DETAILS MATTER MOST?

• Token-level MDP training with γ = 0.95 improves naturalness stability over bandit-style γ = 1, while dropout destabilizes training and mismatched exploration-inference sampling harms test performance.The γ comparison preserves approximately equal sentiment scores but significantly improves language naturalness; dropout greater than 0 often causes loss divergence.Source: §5.4 · p. 9

6 CONCLUSIONS

• GRUE and RL4LMs aim to provide a standard means of comparing reinforcement-learning methods for aligning language models with human preferences.The authors hope these resources will push progress in RL-based alignment.Source: §6 · p. 9
• Improved training stability and consistency could enable iterative language-technology improvement through deployment, user feedback collection, and re-optimization.This process is intended to produce better user experiences with generative models.Source: §6 · p. 9

A.1 PPO DETAILS

• PPO dynamically adapts the KL coefficient β toward a user-specified target divergence, generally using update rate Kβ = 0.2.The target divergence is measured between the initial model h and current policy π.Source: §A.1 · p. 18
• PPO further uses Generalized Advantage Estimation to improve training stability, with λ controlling the bias–variance trade-off.The advantage estimator is defined from the temporal-difference residual.Source: §A.1 · p. 18

A.2 NLPO DETAILS

• NLPO learns to mask irrelevant language with a masking policy copied from the current policy and updated every µ steps.The method defines a parameterized top-p vocabulary by selecting the highest-probability tokens under the policy.Source: §A.2 · p. 18

B.1 CROWDWORKING DETAILS

• Crowd judgments used 600 IMDB generations rated by three annotators each, yielding 56 qualified workers after agreement, participation, and instruction-quality screening.Annotators were compensated to meet a target hourly wage of $15, with bonuses when needed.Source: §B.1 · p. 19

B.2 GRUE EXPERIMENT SETUP

• The GRUE experiments benchmark five training algorithms across six tasks using encoder or encoder-decoder language models, with PPO, NLPO variants, and supervised policy baselines.Actor and critic networks use separate language-model instances, AdamW with a fixed learning rate, and no scheduling.Source: §B.2 · p. 19
• Figure 3 summarizes automated metrics across seven GRUE tasks and human studies on five suitable tasks, averaging multiple reward functions per task.Metrics include task-specific measures such as positive sentiment, plus naturalness measures including perplexity and human-perceived coherence.Source: Figure 3 · p. 19

B.3.1 SETUP

• For IMDB, GPT-2 completes 64-token movie-review prefixes with 48 tokens using top-k sampling with K = 50 and a sentiment classifier as the RL reward.The dataset contains 25k training, 5k validation, and 5k test examples; PPO and NLPO train for 64k steps with adaptive KL targets of 0.02, 0.05, 0.1, or inf, and initial β = 0.1. Table 4 lists the complete hyperparameter settings.Source: §B.3.1 · p. 20, Table 4 · p. 20

B.3.2 RESULTS AND DISCUSSION

• A target KL of 0.1 yields higher rewards and sentiment, but target KL values of 0.02 or 0.05 better preserve fluency and proximity to the pretrained LM.Figure 4 reports that NLPO retains lower perplexities and greater stability than PPO, including at higher KL targets.Source: §B.3.2 · pp. 20–21, Figure 4 · p. 21, Table 5 · p. 21
• With an infinite KL penalty, the model generates context-insensitive positive phrases, illustrating reward hacking and the fluency cost of unconstrained optimization.The reported examples include “I highly recommend this movie to all!” and “worth watching.”Source: Table 5 · p. 21
• Improving the reward classifier generally improves overall task performance more efficiently than adding expert demonstrations for supervised training.Table 7 compares data allocated to reward-classifier training for RL with positive-review data used by the supervised baseline.Source: §B.3.2 · p. 22, Table 7 · p. 22
• A discount factor of 0.95 is preferred because the bandit setting (1.0) reduces NLPO performance and causes PPO reward hacking and instability.These findings indicate that discounted environments are important for stable language-generation training.Source: §B.3.2 · p. 22, Table 6 · p. 22
• NLPO’s hyperparameter ablation varies mask-model synchronization frequency and top-p masking, with higher top-p generally improving performance and intermediate update frequency preferred.Performance is low when the mask model is updated either too infrequently or too frequently.Source: §B.3.2 · p. 22, Table 8 · p. 23

B.3.3 HUMAN PARTICIPANT STUDY

• The human participant study evaluates algorithm outputs using 100 randomly sampled test-set examples per model, each rated by three annotators for coherence and sentiment.The study reports mean Likert scores, annotator agreement, skew, and statistical significance tests.Source: Table 9 · p. 23, §B.3.3 · p. 23, Figure 5 · p. 24
• A one-way ANOVA finds significant differences between models for both coherence and sentiment, with post-hoc Tukey HSD tests identifying significant pairwise differences.The reported overall p-values satisfy p ≪0.05 for both evaluated human-judgment dimensions.Source: §B.3.3 · p. 23, Table 10 · p. 25

B.3.4 QUALITATIVE RESULTS

• Qualitative results present sample generations from each algorithm for three randomly selected prompts.The prompts cover movie summaries, war-film commentary, and Shakespearean language.Source: §B.3.4 · p. 26
• The qualitative examples show outputs that can diverge substantially from their prompts, including altered plots, characters, and review content.This pattern appears across zero-shot, PPO, PPO-no-KL, NLPO, NLPO-no-KL, and supervised generations.Source: §B.3.4 · p. 26

B.4.1 SETUP

• CommonGen generates coherent sentences describing input concept sets, using T5-base with a prompt prefix designed to encourage exploration.The study evaluates Rouge-1, Rouge-avg, meteor, CIDEr, SPICE, and SPiDer rewards.Source: §B.4.1 · p. 28

B.4 COMMONGEN

• The CommonGen experiments combine traditional lexical rewards with task-specific CIDEr, SPICE, and SPiDer rewards.Rouge-avg averages Rouge-1, Rouge-2, and Rouge-L, while SPiDer linearly combines CIDEr and SPICE with equal weights.Source: §B.4.1 · p. 28

B.4.1 SETUP

• Direct language-model fine-tuning caused reward hacking through repetition of prompted concepts, so a −1 penalty is applied when prompt n-grams overlap generated text.The penalty is omitted for policies initialized from supervised training, where this behavior was not observed.Source: §B.4.1 · p. 28
• Evaluation uses beam search, whereas rollout generation uses top-k sampling to favor exploration over exploitation.The implementation and hyperparameter settings are summarized in Table 11.Source: §B.4.1 · p. 28, Table 11 · p. 28

B.4.2 RESULTS AND DISCUSSION

• Warm-started initial policies are crucial for generating coherent sentences with common sense, whereas policies without warm-start suffer from reward hacking despite repetition penalties and task-specific metrics.The finding is based on benchmarking with six reward functions and supervised baselines across development and test sets.Source: §B.4.2 · p. 29
• RL fine-tuned models achieve very high concept coverage, while supervised models more often omit some concepts from their generations.This pattern is also observed in the referenced coverage results.Source: §B.4.2 · p. 29
• RL fine-tuning on supervised models performs better across most CommonGen metrics, especially Coverage, which measures the ratio of concepts included in generated texts.This comparison is reported on the official CommonGen hold-out evaluation.Source: Table 12 · p. 29, §B.4.2 · p. 29
• The study evaluates best-performing algorithm–reward combinations and supervised baselines on CommonGen development metrics before submitting their generations for official test evaluation.Development results include lexical, semantic, and diversity metrics, with test scores reported separately.Source: Table 13 · p. 30, §B.4.2 · p. 30

B.4.3 HUMAN PARTICIPANT STUDY

• Human evaluation rates model generations on coherence and sentiment using averaged Likert scores, with three annotators rating 100 randomly drawn test samples per model.The resulting dataset contains 300 ratings per algorithm, and the study also reports participant counts, annotator agreement, and skew.Source: Table 14 · p. 31, §B.4.3 · p. 32
• A one-way ANOVA finds significant overall differences between models for both coherence and sentiment, with p ≪0.05, followed by pairwise Tukey HSD tests.Individually significant pairwise comparisons are bolded in the post-hoc results.Source: Table 15 · p. 31, §B.4.3 · p. 32
• The human study uses CommonGen instructions and examples without providing a prompt, because the word set is unnecessary for rating either evaluation axis.The interface and task instructions are shown in Figure 6.Source: §B.4.3 · p. 32, Figure 6 · p. 32

B.4.4 HUMAN PREFERENCE LEARNING EXPERIMENTS

• The human study filtered prompts to cases where both models attempted all input concepts before collecting three-way pairwise fluency and commonsense preferences.This filtration avoided rewarding fluent outputs that omitted required concepts.Source: §B.4.4 · pp. 32–33
• Human feedback favored preference-optimized RL over METEOR-only RL, with 682 versus 587 preferred generations (p < 0.01).The comparison used a rerun of the pairwise preference procedure on Commongen test generations.Source: §B.4.4 · p. 33
• The preference reward model achieved 69.5 test ROC AUC when predicting majority human preferences from 417 annotated completion pairs.Three crowdworkers annotated each pair, and the data were split 60/20/20 into train, validation, and test sets.Source: §B.4.4 · p. 33
• The preference-based reward combined METEOR with preference-model scores discounted by uncovered concepts: r = meteor + pref/(1 + |miss|).The design aimed to capture both concept coverage and human preferences for sentence form within the task constraints.Source: §B.4.4 · p. 33

B.4.5 QUALITATIVE ANALYSIS

• Qualitative examples show Supervised+NLPO generally producing fluent sentences that cover prompted concepts, whereas PPO, NLPO, and supervised variants sometimes omit concepts or introduce errors.Examples include kitchen-sausage, badminton, and pasta prompts, with outputs varying in fluency, coverage, and factual consistency.Source: §B.4.5 · pp. 33–34

B.5 CNN DAILY MAIL

• For CNN/Daily Mail summarization, the study uses 287k training, 13k validation, and 11k test examples from long news articles paired with author-written highlights.This establishes the dataset scope for the subsequent RL summarization experiments.Source: §B.5.1 · p. 35

B.5.1 SETUP

• The CNN/Daily Mail experiments train RL methods with Rouge-1, Rouge-avg, and Meteor rewards using T5 as the base language model.T5 was selected for its unified text-to-text pretraining and zero-shot capabilities.Source: §B.5.1 · p. 35
• All CNN/Daily Mail models use multinomial sampling with temperature 0.7 during decoding, with additional hyperparameters reported in Table 16.The controlled decoding configuration applies across the evaluated models.Source: §B.5.1 · p. 35, Table 16 · p. 35

B.5.2 RESULTS AND DISCUSSION

• RL fine-tuning on top of a supervised model consistently improves lexical, semantic, factual-correctness, and diversity metrics on CNN/Daily Mail.PPO and NLPO are on par with supervised performance on Rouge-2, Rouge-L, and Bleu, while fine-tuned models are factually consistent according to SummaCZS.Source: §B.5.2 · p. 35, Table 17 · p. 36
• The CNN/Daily Mail evaluation compares lead-3, Zero-Shot, supervised, PPO, and NLPO baselines across lexical, semantic, factual-correctness, and diversity metrics.The test-set table reports the benchmarked model families and metric categories; no individual cell values are supplied here.Source: Table 17 · p. 36
• PPO model selection varies reward functions and rollout top-k values, with the best model selected for each algorithm–reward combination.These comparisons are reported on the validation set in Table 18.Source: Table 18 · p. 37
• NLPO model selection varies reward functions, rollout top-k values, and target-update iterations, selecting the best model for each algorithm–reward combination.These comparisons are reported on the validation set in Table 19.Source: Table 19 · p. 37

B.5.3 HUMAN PARTICIPANT STUDY

• Human evaluation rates model outputs for coherence and sentiment using 50 randomly sampled test examples per model, each rated by three annotators.This produces 150 data points per algorithm and reports participant counts, average Likert scores, inter-annotator agreement, and skew.Source: Table 20 · p. 38
• One-way ANOVA finds significant overall model differences for both coherence and sentiment, with post-hoc Tukey HSD tests identifying individually significant pairwise comparisons.The reported overall p-values are significant with p ≪0.05, and individually significant results are bolded in Table 21.Source: Table 21 · p. 38
• The human study uses a summarization interface with a 15-second minimum window time, while Tables 20 and 21 report averaged ratings, agreement, and significance tests.Participants were not required to read the entire article, but sliders remained hidden until the minimum time elapsed.Source: §B.5.3 · p. 39, Figure 8 · p. 39

B.5.4 QUALITATIVE ANALYSIS

• Samples from three randomly selected prompts illustrate qualitative differences among the evaluated generation algorithms.The prompts cover Manchester City’s financial restrictions, an emergency flight landing, and an Inverness–Celtic football match.Source: §B.5.4 · pp. 39–41
• For the Manchester City article, NLPO preserves the central UEFA financial-fair-play topic and named transfer targets, while other outputs vary in specificity and factual detail.The NLPO sample mentions the restrictions being lifted and names Raheem Sterling, Kevin de Bruyne, and Ross Barkley; supervised and hybrid outputs retain related financial figures and claims.Source: §B.5.4 · p. 40
• For the emergency-landing article, the generated summaries retain the landing, pressurization issue, passenger distress, and reported loss of consciousness with differing levels of detail.NLPO includes the flight number and three passengers’ loss of consciousness, while supervised and hybrid outputs emphasize the FAA investigation, passenger quotations, or the emergency landing.Source: §B.5.4 · p. 41

B.6.1 SETUP

• ToTTo training uses T5, five reward functions, beam-search inference, and top-k sampling for rollout generation.The reward functions are BLEU, SacreBLEU, METEOR, PARENT, and a METEOR–PARENT combination.Source: §B.6.1 · p. 43, Table 22 · p. 43

B.6.2 RESULTS AND DISCUSSION

• Supervised+NLPO outperforms all models on the ToTTo leaderboard by PARENT, while warm-started policies avoid reward hacking and sub-optimal solutions.The benchmark compares five reward functions with supervised baselines; Tables 23 and 24 report dev and test evaluations.Source: §B.6.2 · p. 43, Table 23 · p. 44, Table 24 · p. 45

B.6.3 HUMAN PARTICIPANT STUDY

• Human evaluation measures model generations using averaged coherence and sentiment ratings, annotator agreement, and statistical significance tests.Each algorithm contributes 150 ratings from 50 randomly sampled test examples and three annotators per sample; one-way ANOVA finds significant differences for both coherence and sentiment.Source: Table 25 · p. 46, Table 26 · p. 46, §B.6.3 · p. 47

B.6.4 QUALITATIVE ANALYSIS

• Qualitative ToTTo examples compare zero-shot, PPO, NLPO, supervised, and supervised-plus-RL generations on table-description prompts.The examples illustrate outputs describing tour dates, a French monarch, and a United States senator succession.Source: §B.6.4 · p. 48

B.7.1 SETUP

• NarrativeQA experiments use T5-base with Rouge Combined and Rouge-L-Max rewards, while supervised baselines are trained on UnifiedQA.UnifiedQA supervised fine-tuning is reported to outperform training only on NarrativeQA in prior work.Source: §B.7.1 · p. 49, Table 27 · p. 49

B.7.2 RESULTS AND DISCUSSION

• NarrativeQA results show that warm-started initial policies are crucial for generating answers that successfully use the input context.The evaluation uses Rouge Combined and Rouge-L-Max rewards on the NarrativeQA test set, with supervised baselines based on UnifiedQA.Source: Table 28 · p. 50, §B.7.2 · p. 51

B.7.3 QUALITATIVE RESULTS

• The qualitative language-generation examples show PPO and NLPO producing the same key facts for sampled prompts, while supervised variants differ in answer completeness.For the Mark Hunter prompts, PPO and NLPO retain central details such as the pirate radio station and expelling students; for Maskull, supervised variants preserve the reported deaths.Source: §B.7.3 · pp. 51–52

B.8.1 SETUP

• The WMT-16 setup translates English to German and trains models with SacreBLEU, chRF, TER, and BertScore rewards.The experiment frames WMT-16 as a standard English-to-German machine-translation task.Source: §B.8.1 · p. 53, Table 29 · p. 53

B.8.2 RESULTS AND DISCUSSION

• NLPO + Supervised performs better than PPO and supervised models on the WMT-16 and IWSLT machine-translation benchmarks.Results compare four reward functions against supervised baselines using lexical, semantic, and diversity metrics.Source: §B.8.2 · p. 53, Table 30 · p. 54, Table 31 · p. 55

B.8.3 QUALITATIVE RESULTS

• Machine-translation qualitative examples show all methods producing similar German translations for the sampled TED prompts, with minor wording and formatting differences.The examples compare Zero-Shot, PPO, NLPO, supervised, and supervised-plus-RL variants across three prompts.Source: §B.8.3 · p. 56

B.9.1 SETUP

• The DailyDialog setup generates the next utterance from five-turn contexts with GPT-2, top-k sampling at k = 20, and a reward combining METEOR with intent-match accuracy.The intent component is supplied by a fine-tuned RoBERTa classifier, with reward coefficients selected using validation lexical and intent results.Source: §B.9.1 · p. 57

B.9.3 HUMAN PARTICIPANT STUDY

• In the DailyDialog human study, 100 test samples per model were rated by three annotators, with coherence and sentiment summarized alongside agreement and pairwise significance tests.The overall one-way ANOVA differences are significant at p ≪0.05 for both coherence and sentiment.Source: Table 34 · p. 59, §B.9.3 · p. 59, Table 35 · p. 59

B.9.2 RESULTS AND DISCUSSION

• NLPO variants generally outperform other methods on DailyDialog intent accuracy and automatic metrics, while RL methods overall improve on supervised baselines.The benchmark uses lexical and intent-based rewards for next-utterance generation.Source: §B.9.1 · p. 57, §B.9.2 · p. 59

B.9.4 QUALITATIVE ANALYSIS

• Across three prompts, zero-shot, PPO, NLPO, supervised, and combined methods generate distinct responses to identical conversational contexts.The samples cover discussions about Chinese tea, food, and an auto-reverse device.Source: §B.9.4 · pp. 60–61
• The tea example shows responses ranging from topical statements about Oulong tea to unrelated or contradictory claims across methods.Zero-Shot discusses meals, PPO compares tea strength, NLPO denies having tea, and the supervised variants provide different conversational replies.Source: §B.9.4 · p. 60
• The food and device examples likewise show differing degrees of contextual engagement, with methods offering replies that vary from relevant follow-ups to apparent misunderstandings.For the food prompt, outputs discuss saltiness, preferences, or choices; for the device prompt, outputs address features, uncertainty, or unrelated details.Source: §B.9.4 · p. 61