PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi

arXiv:2210.01241v3cs.CLcs.LG

TL;DR

The paper asks whether reinforcement learning is practical for aligning pretrained language models with human preferences despite instability and limited shared resources. It introduces RL4LMs, the GRUE reward-based benchmark, and NLPO, and finds that NLPO and reinforcement learning generally improve preference–fluency trade-offs over alternatives. The results also indicate data and parameter efficiency, while performance depends on the initial policy containing task-relevant signal.

  • Problem

    Aligning pretrained language models with human preferences is difficult because supervised target strings are only an imperfect proxy, while RL lacks stable, open-source benchmarks and implementations for NLP.

  • Method

    The paper releases RL4LMs, a modular library for on-policy RL with HuggingFace language models and arbitrary rewards, introduces GRUE reward-based generation tasks, and proposes NLPO’s token-level constraints.

  • Results

    RL methods, especially supervised warm-starting combined with NLPO or PPO, generally outperform isolated approaches across preference and naturalness metrics, with NLPO also matching human-evaluation trends.

  • Takeaways & Limitations

    The released library and benchmark provide a standard basis for comparing RL alignment methods, while the results support RL as a practical approach within the evaluated tasks.

Abstract

from arXiv · show

We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open-source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community: is RL a practical paradigm for NLP? To help answer this, we first introduce an open-source modular library, RL4LMs (Reinforcement Learning for Language Models), for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al. 2020) with an arbitrary reward function. Next, we present the GRUE (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference. GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language Policy Optimization) that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al. 2017)), based on both automatic and human evaluations.

1 INTRODUCTION

The paper addresses the difficulty of aligning language models with human preferences using practical reinforcement-learning resources and methods. It releases RL4LMs, GRUE, and NLPO, reporting improved reward–fluency balance and efficiency over alternatives.

  • Motivation: Automated preference metrics provide a cheaper alternative to dense human-in-the-loop feedback, but evaluate only full generations.Such metrics can correlate better with human judgment than earlier measures, yet imperfect neural metrics risk reward hacking.
  • Motivation: RL offers a natural framework for optimizing non-differentiable scalar objectives in language generation, but NLP progress is hindered by instability and limited open-source resources.The paper specifically highlights combinatorial action spaces, Goodhart’s Law, and the lack of open benchmarks and implementations.
  • Contributions: The authors release RL4LMs, the GRUE benchmark, and NLPO to support research on aligning language models with human preferences.GRUE pairs contemporary NLP tasks with reward functions, while NLPO dynamically learns token-level task constraints.
  • Results: NLPO better balances preference rewards and language fluency than alternatives, including PPO, in GRUE experiments and human evaluations.The broader experiments also report data and parameter efficiency for reinforcement learning with learned rewards.

2 RELATED WORK

Prior NLP work has applied imitation learning and reinforcement learning to sequential text generation, including methods designed for large action spaces. Despite successes across tasks, RL remains criticized for instability and limited comparability.

  • Imitation learning for NLP: Imitation-learning methods for NLP draw on DAGGER and SEARN but are associated with exposure bias and the cliff MDP problem.The cited methods include schedule-sampling variants, LOLS, TextGAIL, and SEARNN.
  • RL for Large Action Spaces: Prior approaches address large language-generation action spaces by using actor-critic methods or reducing the action space during exploration.Examples include MIXER, KG-A2C, TrufLL, AE-DQN, and GALAD.
  • RL for NLP: RL has been used across translation, summarization, dialogue, captioning, question generation, text games, and preference alignment.Preference-alignment work includes PPO-based fine-tuning, although non-public data can prevent direct comparison.
  • RL for NLP: RL has achieved successes in NLP but is also criticized as less stable than supervised language-model training.The paper connects this criticism with relatively limited evidence and implementations for comparing RL methods.

3 RL4LMS: A LIBRARY FOR TRAINING LMS WITH RL

RL4LMs provides a modular interface for training language models as token-level sequential decision processes with on-policy reinforcement learning. It supports arbitrary environments, reward functions, metrics, and KL-regularized objectives.

  • RL4LMs library: RL4LMs supports decoder-only and encoder-decoder HuggingFace transformers with on-policy algorithms including PPO, TRPO, A2C, and NLPO.The modular library combines HuggingFace and stable-baselines-3 components and allows customized environments, rewards, metrics, and algorithms.
  • Environments: generation as a token-level MDP: Each generation environment is modeled as an MDP in which the state is the prompt plus generated tokens and each action is a vocabulary token.The transition appends the selected token, and an episode ends at the horizon or when EOS is generated.
  • Environments: generation as a token-level MDP: RL4LMs provides an environment API that enables new NLP tasks to be added while remaining compatible with implemented algorithms.The interface supports per-token or per-sequence rewards and diverse textual metrics.
  • On-policy actor-critic algorithms: On-policy actor-critic training initializes the agent policy from a pretrained language model and optimizes discounted trajectory rewards.The value network is similarly initialized from the pretrained model, with a randomly initialized scalar-output final layer; generalized advantage estimation improves stability.
  • On-policy actor-critic algorithms: A token-level KL penalty regularizes sparse sequence rewards to keep the learned policy near the initialized language model.The KL coefficient β is dynamically adapted, and the regularized reward subtracts β times the policy-to-reference KL divergence.

4 NLPO: NATURAL LANGUAGE POLICY OPTIMIZATION

NLPO extends PPO with a learned token mask that reduces language-generation action spaces during reinforcement learning. The mask is updated periodically from a delayed copy of the current policy, balancing task-relevant exploration against excessive constraint.

  • NLPO is a parameterized-masked extension of PPO that learns to exclude less relevant vocabulary tokens during training.Top-p sampling restricts actions to the smallest token set whose cumulative probability exceeds p.
  • The masking policy πψ is a periodically updated copy of the current policy πθ, with update frequency μ.It selects top-p tokens and assigns zero probability to the remaining tokens when πθ samples actions.
  • NLPO’s mask adds a constraint based on a policy from μ iterations earlier rather than only the initial policy used for KL regularization.This can retain task-relevant information learned during RL while constraining the current policy.
  • Algorithm 1 collects trajectories, computes preference and KL-penalty rewards and advantages, updates the PPO-Clip policy and value function, then periodically updates the masked policy.The procedure highlights the operational differences between NLPO and PPO.

5 GRUE (GENERAL REINFORCED-LANGUAGE UNDERSTANDING EVAL)

GRUE evaluates reinforcement-learning methods across generative NLP tasks using task-preference and naturalness metrics, supplemented by human studies and targeted ablations. Results show that performance depends on warm starts, reward constraints, token-level discounting, and the choice between NLPO and PPO.

  • GRUE benchmark: GRUE comprises 7 generative NLP tasks evaluated with task-preference and naturalness metric mixes to reduce reward hacking by any single metric.Preference metrics assess task desiderata, while naturalness metrics assess fluency and readability.
  • Experimental setup: Human studies cover five GRUE tasks and test whether automated metrics agree with human model rankings while comparing NLPO, PPO, and KL-related ablations.The evaluated tasks are IMDB, Commongen, ToTTo, DailyDialog, and CNN Daily Mail.
  • Results on GRUE: RL tends to outperform supervised training for text continuation and summarization, whereas supervised training performs best on low-zero-shot tasks such as Commongen and ToTTo.Both supervised and RL approaches outperform zero-shot on the latter tasks.
  • Results on GRUE: Supervised warm-starting combined with RL usually outperforms either method alone, especially for Commongen and ToTTo, although DailyDialog is an exception.The results associate warm starts with reduced reward-hacking risk on tasks where the initial policy has weak task performance.
  • Human agreement: Human-evaluation trends generally match automated metrics, but human judgments identify discrepancies such as supervised training outperforming Supervised+PPO on 2 of 5 tasks.The authors report that automated metrics usually correlate with human judgments above a naturalness threshold, but may miss reward hacking.
  • Preference reward learning: Removing the KL constraint causes reward hacking, while deriving it from a warm-started supervised policy mitigates nonsense behavior when the initial policy performs poorly.On Commongen and ToTTo, weak initial policies can lead the KL penalty toward repeated input fragments.
  • PPO versus NLPO: NLPO generally outperforms PPO and supervised baselines, with performance peaking at an intermediate top-p value for its masking policy.The pattern suggests a balance between constraining exploration and retaining available actions.
  • Practical considerations: Using γ = 0.95 in a token-level MDP preserves approximately the same sentiment scores as γ = 1 while improving naturalness on IMDB.The γ = 1 bandit-equivalent setting produces significantly less natural language for both PPO and NLPO.

6 CONCLUSIONS

The authors hope GRUE and RL4LMs will accelerate alignment research by providing a standard way to compare reinforcement-learning methods.

  • GRUE and RL4LMs provide a standard means of comparing methods for aligning language models to human preferences.

A ON-POLICY ALGORITHM IMPLEMENTATION DETAILS

The implementation combines adaptive KL regularization, advantage estimation, and NLPO’s periodically updated masking policy with LM and human-evaluation infrastructure.

  • PPO details: The KL coefficient β is dynamically adapted during training, following prior work.The update rate Kβ is generally set to 0.2.
  • PPO details: Generalized Advantage Estimation defines the advantage estimator from Temporal Difference residuals to increase training stability.The parameter λ controls the bias–variance trade-off.
  • NLPO details: NLPO maintains a masking policy copied from the current policy and updates it only every µ steps.The masking policy supports parameterized top-p vocabulary masking.
  • Experiment setup: The experiments use GPT-2 for IMDB continuation and T5 for the remaining tasks, training PPO, NLPO, supervised, and hybrid policies.Human studies use qualified Mechanical Turk annotators and multiple algorithmic baselines.
  • Evaluation: Figure 3 averages automated metrics across seven GRUE tasks and human studies across five suitable tasks, separating task-specific and naturalness measures.

B.3.2 RESULTS AND DISCUSSION

Results show that KL regularization, reward-model data quality, and discounted environments materially affect RL performance, while NLPO remains competitive across evaluations.

  • Target KL ablation: A target KL of 0.1 increases rewards but causes drift from the pretrained LM and reduced fluency; targets of 0.02 or 0.05 preserve closeness better.These learning-curve trends are averaged over five runs, with shaded regions showing one standard deviation.
  • Target KL ablation: NLPO achieves better sentiment and perplexity scores than PPO in the target-KL ablation.The table reports means and standard deviations over five random seeds, alongside fluency and diversity metrics.
  • Training data size ablation: Improving reward-function quality increases overall task performance more than adding supervised-training data, indicating greater data efficiency for reward-model improvement.
  • Discount factor ablation: A discount factor of 1.0 causes NLPO performance loss and PPO reward hacking, whereas the discounted setting with 0.95 is preferred.
  • Human evaluation: Human evaluations assess coherence and sentiment across models, with one-way ANOVA results significant at p ≪0.05 for both measures.The study uses three annotators per sample and reports agreement statistics.

B.3.4 QUALITATIVE RESULTS

Qualitative examples illustrate that unconstrained or poorly regularized policies can produce repetitive or contextually inappropriate text, while the benchmark also evaluates task-specific rewards and coherence.

  • Sample generations: The qualitative samples compare Zero-Shot, References, PPO, PPO-no-KL, NLPO, NLPO-no-KL, and Supervised generations across three prompts.
  • Sample generations: The examples include repetitive recommendations, fragmented continuations, and outputs that drift from the prompt’s subject matter.These patterns appear in several PPO-no-KL, supervised, and other sample continuations.
  • CommonGen: Direct LM fine-tuning on CommonGen caused repetition of prompted concepts to maximize rewards, motivating a −1 overlap penalty for RL policies without supervised initialization.This penalty is not applied when a supervised policy initialization avoids the problem.

B.4.2 RESULTS AND DISCUSSION

On CommonGen, warm-started policies are crucial for coherent, commonsense generation, while RL fine-tuning improves concept coverage and can outperform supervised baselines. Human evaluations assess coherence and sentiment across algorithms.

  • Benchmark results: Warm-started initial policies are crucial for generating coherent sentences with common sense, whereas uninitialized policies suffer reward hacking.This remains true despite repetition penalties and task-specific metrics such as CIDEr.
  • Benchmark results: RL fine-tuned models obtain very high concept coverage, while supervised models tend to miss some input concepts.
  • Benchmark results: RL fine-tuning on a supervised model yields better CommonGen performance across most metrics, especially Coverage.
  • Human evaluation: The CommonGen human study reports participant counts, average coherence and sentiment ratings, annotator agreement, and skew for each model.
  • Human evaluation: Post-hoc Tukey tests compare algorithm means, while one-way ANOVA finds significant model differences for both coherence and sentiment.The reported overall p-values are p ≪ 0.05.

B.4.4 HUMAN PREFERENCE LEARNING EXPERIMENTS

The human preference experiment filters CommonGen outputs for concept use, collects pairwise crowdworker judgments, and trains a reward model to predict majority preferences. Sample generations illustrate qualitative differences among RL and supervised methods.

  • Preference data collection: The experiment samples one completion from Supervised and Supervised+NLPO, retaining prompts where both models attempt all input concepts.The filter avoids preferring a more fluent sentence that omits concepts crowdworkers should consider necessary.
  • Preference data collection: Crowdworkers choose between paired generations for commonsense or fluency, producing 3 annotations for each of 417 pairs.The annotator agreement was Krippendorf α = .28.
  • Preference model: A T5-11B reward model predicts which completion a majority of three annotators preferred, achieving 69.5 test ROC AUC.The model is conditioned on the prompt and completion.
  • Qualitative analysis: Qualitative examples compare Zero-Shot, PPO, NLPO, Supervised, and hybrid models on CommonGen prompts involving kitchen objects, sports, and food.
  • Related evaluation setup: The summarization setup uses T5 on CNN/DM with Rouge-1, Rouge-avg, and Meteor as automated reward metrics.The dataset contains 287k training, 13k validation, and 11k test examples.

B.5.2 RESULTS AND DISCUSSION

On CNN/Daily Mail summarization, PPO and NLPO are competitive with supervised performance on several metrics, while RL fine-tuning atop supervised models improves results consistently. Human studies evaluate coherence and sentiment with statistical comparisons.

  • Benchmark results: PPO and NLPO are on par with supervised performance on Rouge-2, Rouge-L, and Bleu.
  • Benchmark results: Fine-tuning RL methods on a supervised model improves performance consistently across all reported metrics.
  • Benchmark results: RL-fine-tuned models are factually consistent according to the SummaCZS metric.
  • Ablations: PPO and NLPO model selection varies reward functions and rollout top-k values, with NLPO additionally varying target update iterations.
  • Human evaluation: The summarization human study reports average coherence and sentiment ratings, annotator agreement, and skew from 50 samples rated by three annotators per model.
  • Human evaluation: Post-hoc Tukey tests compare algorithm means, and one-way ANOVA reports significant differences for both coherence and sentiment.The reported overall p-values are p ≪ 0.05.

B.5.4 QUALITATIVE ANALYSIS

Qualitative examples and benchmark findings span summarization, ToTTo, and NarrativeQA. Across these tasks, warm-starting is associated with avoiding reward hacking and producing outputs that use the input context effectively.

  • Summarization examples: The qualitative summarization example compares PPO, NLPO, supervised, and hybrid outputs on a Manchester City financial-fair-play article.
  • Summarization examples: The article’s source discusses UEFA spending and wage restrictions, transfer targets, and the possibility of penalties being lifted.
  • Summarization examples: The example outputs preserve parts of the 49-million transfer-spend limit and 205-million annual wage bill, with some variants adding transfer targets.
  • Summarization examples: The source further describes City’s requested relief from UEFA restrictions and comparisons with spending by other English and European clubs.
  • ToTTo results: Warm-started policies are crucial for generating ToTTo descriptions from highlighted table cells, while uninitialized policies suffer reward hacking.
  • ToTTo results: Supervised+NLPO outperforms all models on the ToTTo leaderboard according to the PARENT metric.
  • NarrativeQA results: Warm-started policies are crucial for NarrativeQA answers that successfully use the input context.

B.7.3 QUALITATIVE RESULTS

The qualitative examples contrast generations from zero-shot, supervised, PPO, and NLPO systems across narrative continuation and question-answering prompts. The benchmark results report that NLPO variants outperform other methods, while NLPO generations often preserve more of the source content and task context.

  • NLPO continues the pirate-radio narrative with specific details about Hunter, Phoenix, and his parents’ basement.
  • The longer reference passage includes the radio show’s influence, student reactions, the FCC investigation, and the principal’s expulsion of low-scoring students.
  • For the question about Mark Hunter, NLPO identifies his pirate radio station and describes him as a loner and outsider.
  • For the Maskull prompt, PPO repeats the question and setup, while NLPO gives a shorter continuation centered on Maskull meeting Krag.
  • NLPO + Supervised performs better than PPO and supervised models in the benchmark results.
  • NLPO variants achieve better intent accuracy and automatic metric scores than the other evaluated methods.

B.9.4 QUALITATIVE ANALYSIS

The Daily Dialogue examples show substantial variation in how systems continue multi-turn conversations. NLPO sometimes produces contextually appropriate replies, but the samples also contain terse, irrelevant, or incomplete outputs across methods.

  • In the tea conversation, NLPO continues with a brief question about Oolong tea, while other systems produce varied and sometimes incomplete replies.
  • For the dahl conversation, NLPO replies positively, whereas other systems mention saltiness, dryness, thirst, or unrelated bus details.
  • In the auto-reverse discussion, NLPO gives an uncertain and disengaged response that does not address the preceding product recommendation.
Loading 2210.01241v3…