PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation

Zongxia Li, Yapei Chang, Yuhang Zhou, Xiyang Wu, Zichao Liang, Yoo Yeon Sung, Jordan Lee Boyd-Graber

arXiv:2506.15068v1cs.CLcs.LG

TL;DR

Open-ended long-form generation is difficult to evaluate because human-valued qualities are hard to capture with objective criteria, and existing metrics can miss them. The paper proposes PrefBERT as a lightweight semantic-quality reward model for GRPO; its use yields policy models better aligned with human preferences than traditional metric rewards.

  • Problem

    Open-ended long-form generation lacks objective criteria for judging qualities such as coherence, relevance, and style, which limits reliable reward design for GRPO.

    Source: §1 · p. 1

  • Method

    PrefBERT is a lightweight long-form reward model used to score response quality and provide semantic rewards for GRPO.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Across model sizes, semantic verifiable rewards produce better long-form GRPO policies than traditional metric-based rewards.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    PrefBERT-trained 1.5B and 3B models match or exceed Qwen2.5-7B-Instruct across all reported metrics.

    Source: §5.2 · p. 6

    The study did not train or use a larger, more powerful evaluator such as a 7B-scale model, primarily because of computational constraints.

    Source: §9 · p. 9

Abstract

from arXiv · show

Evaluating open-ended long-form generation is challenging because it is hard to define what clearly separates good from bad outputs. Existing methods often miss key aspects like coherence, style, or relevance, or are biased by pretraining data, making open-ended long-form evaluation an underexplored problem. To address this gap, we propose PrefBERT, a scoring model for evaluating open-ended long-form generation in GRPO and guiding its training with distinct rewards for good and bad outputs. Trained on two response evaluation datasets with diverse long-form styles and Likert-rated quality, PrefBERT effectively supports GRPO by offering better semantic reward feedback than traditional metrics ROUGE-L and BERTScore do. Through comprehensive evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis, we show that PrefBERT, trained on multi-sentence and paragraph-length responses, remains reliable across varied long passages and aligns well with the verifiable rewards GRPO needs. Human evaluations confirm that using PrefBERT as the reward signal to train policy models yields responses better aligned with human preferences than those trained with traditional metrics. Our code is available at https://github.com/zli12321/long_form_rl.

1 Introduction

Open-ended long-form generation lacks objective criteria for distinguishing high- from low-quality responses, making reward design a challenge for GRPO. The paper proposes PrefBERT as a lightweight reward model and reports improved human-preference alignment for policies trained with it.

  • Open-ended generation requires balancing coherence, fluency, and informativeness, unlike structured tasks with explicit correctness signals.Source: §1 · p. 1
  • ROUGE and BERTScore often miss human-valued qualities, while general reward models can be costly and their scores may track response length.Source: §1 · p. 1
  • PrefBERT is a lightweight model trained on diverse long-form responses and human ratings to provide fine-grained rewards for GRPO.Source: §1 · pp. 1–2
  • Human evaluations and benchmark results show PrefBERT-trained GRPO policies align better with preferences than policies trained with traditional metrics or SFT.Source: §1 · p. 2

2 Related Work

Prior GRPO work relies on verifiable, rule-based rewards, leaving long-form generation without a fixed correct answer underexplored. Existing evaluation options trade off alignment with human judgments against computational cost.

  • DPO and PPO require substantial human-annotated data or computational resources, whereas GRPO uses self-generated data and verifiable rewards for tasks with clear correctness signals.Source: §2 · p. 2
  • GRPO extensions broaden rule-based reward designs for structured tasks, but long-form generation without a fixed correct response remains underexplored.Source: §2 · pp. 2–3
  • Reference-based metrics such as ROUGE and BERTScore poorly reflect long-form semantic or pragmatic qualities and can misalign with human judgments.Source: §2 · p. 3
  • LLM-as-a-judge can align better with human preferences, but repeated evaluations are computationally costly; small models trained on human ratings are less explored as training rewards.Source: §2 · p. 3

3 Conceptual Backgrounds

GRPO can learn from scalar feedback, but open-ended generation lacks reliable evaluation signals that reflect human judgments. The section introduces PrefBERT, a lightweight reference-based reward model trained on human ratings of generated responses.

3.1 Preliminaries on GRPO

  • GRPO samples candidate responses, assigns scalar rewards, and normalizes each response’s advantage against the rewards within its group.The resulting policy update combines a clipped surrogate loss with a KL-divergence penalty against a reference model.Source: §3.1 · p. 3

3.2 Existing Methods for Scoring Open-ended Generation

  • Reference-based metrics such as ROUGE and BERTScore, and preference-trained reward models such as GRM-LLaMA-3B, have different limitations for scoring open-ended generation.Reference-based metrics correlate poorly with human preferences on long-form outputs; reward models may inherit training-data biases.Source: §3.2 · pp. 3–4

3.3 PrefBERT: Training a Lightweight BERT-Based Reward Model

  • PrefBERT is a 150M-parameter, reference-based model designed to provide fine-grained reward signals while remaining computationally lightweight.It is trained on 19K examples drawn from Prometheus-preference and MOCHA, rather than the 80K examples used for GRM-llama-3B.Source: §3.3 · p. 4
  • PrefBERT concatenates a reference answer and generated response, predicts a normalized Likert score, and supplies that score as GRPO’s reward.ModernBERT encodes the pair; a linear regressor and sigmoid produce the prediction, and training minimizes mean-squared error.Source: §3.3 · p. 4

4 Experiment Setup

The experiments use three open-ended datasets and compare GRPO policies trained with different reward signals against supervised fine-tuning. The datasets span varied long-form response styles and topics.

4.1 Free-Form and Open-Ended Datasets

  • The combined free-form training and test sets draw from ELI5, Alpaca, and LongForm, with coding tasks excluded from LongForm.The datasets contain long-form responses averaging 185 words and call for evaluation of coherence, conciseness, and informativeness rather than correctness.Source: §4.1 · pp. 4–5

4.2 Training Setup

  • The GRPO setup trains Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct separately with ROUGE-L, BERTScore, GRM-llama-3B, or PrefBERT rewards.Training uses OpenRLHF and prompts that encourage factual, relevant, clear, concise, and complete responses.Source: §4.2 · p. 5
  • The supervised fine-tuning comparison trains 1.5B and 3B Qwen2.5-Instruct models using reference responses as ground truth.Source: §4.2 · p. 5

5 Automatic Evaluation

Across the reported evaluations, PrefBERT-trained policies perform strongly against other policy models and can rival a larger instruction-tuned model. Automatic ratings also show weaker results for traditional metric rewards and SFT, while reward curves differ in their progression.

5.1 Evaluation Metrics

  • Point-wise evaluation uses GPT-4 Likert scores and success rates, while pairwise comparisons use Bradley–Terry win rates derived from ratings between every model pair.The training curves show minimal reward change for ROUGE-L (0.05) and BERTScore (0.25); GRM-llama-3B reaches the 1,024-token maximum by step 60, while PrefBERT’s reward progression is not strictly tied to response length.Source: §5.1 · p. 5, §5.3 · p. 6

5.2 Results and Discussion

  • 3B-PrefBERT and 3B-GRM-llama-3B have the highest LLM-judge average ratings and success rates among policy models, while PrefBERT uses 20 times fewer parameters and 19K training examples versus 80K for GRM-llama-3B.Human evaluations later find that GRM-llama-3B tends to produce long, overly verbose outputs, which likely inflate its Table 1 scores.Source: §5.2 · p. 5
  • PrefBERT-trained policies perform competitively with GRM-llama-3B policies and substantially outperform token-overlap-reward policies and SFT; 1.5B and 3B PrefBERT policies rival or exceed Qwen2.5-7B-Instruct.The table caption identifies comparisons among SoTA baselines, RL-finetuned models using various rewards, and SFT; larger models are generally stronger, and GRM scores may be inflated by verbosity.Source: §5 · p. 5, Table 1 · p. 6, §5.2 · p. 6
  • 3B-PrefBERT scores 3.98 versus 3.69 for Qwen2.5-7B-Instruct on Likert score, 70.70% versus 59.10% on success rate, and 11.56% versus 6.37% on win rate.Qwen2.5-32B and 72B are stronger overall.Source: §5.2 · p. 6
  • Across both model sizes, ROUGE-L- and BERTScore-trained policies perform worse across all evaluation dimensions than PrefBERT- and GRM-trained policies; SFT scores are lower than GRPO models except for 1.5B-BERTScore.The 1.5B PrefBERT and GRM policies exceed 3B-SFT by around 1.1 average Likert points, 50% in success rate, and 6% in Bradley–Terry win rate.Source: §5.2 · p. 6

6 Human Evaluation

Human evaluation favors 3B-PrefBERT among the evaluated policy models, while exposing a mismatch between automatic and human judgments of verbose GRM outputs. Annotators also identify differences in instruction following, writing quality, and the shallowness of SFT responses.

6.1 Human Evaluation Setup

  • Four author annotators rated and ranked responses from seven models on 150 prompts sampled equally across three datasets, with model names hidden.They used 1–5 Likert scores and the same evaluation criteria given to LLM judges.Source: §6.1 · p. 7

6.2 Results and Qualitative Analysis

  • 3B-PrefBERT earns a 3.36 average human Likert rating and 51% success rate; Qwen2.5-72B-Instruct leads overall at 3.61 and 57%.ROUGE-L and BERTScore improve success rates by approximately 3% over the 3B base model, while PrefBERT gains around 10%.Source: §6.2 · p. 7, Table 2 · p. 8
  • Human rankings align closely with LLM-as-a-judge except for GRM-llama-3B, whose 710-word responses contrast with PrefBERT’s 258 and are consistently ranked low by annotators.The paper links GRM’s length-driven rewards to verbosity and reports that its 3B and 1.5B policy models take 10 and 7 more training hours, respectively, than other models at those scales.Source: §6.2 · p. 7, Figure 2 · p. 7
  • Across 26 examples where PrefBERT is preferred over the base model, annotators identify better instruction following and tone and fluency.PrefBERT reliably follows explicit constraints and produces more fluent, cohesive answers than the base model.Source: §6.2 · pp. 7–8
  • PrefBERT and GRM outputs use markdown formatting more often than other models—81.31% and 66.80%, respectively—and annotators often note their readability.By contrast, ROUGE-L and BERTScore models can give generic or repetitive answers; the authors suggest their low reward variance could indicate a weak training signal.Source: §6.2 · p. 8
  • SFT responses are often vague and shallow, while the authors note that SFT remains valuable when high-quality human-annotated data is available.They attribute some SFT behavior to noisy or imprecise reference answers in sources such as ELI5.Source: §6.2 · pp. 8–9

7 Conclusion

The conclusion proposes PrefBERT as a small fine-tuned evaluator that supplies semantic-quality rewards for GRPO on open-ended long-form generation. The authors report better response quality than traditional metrics or a generalized preference reward model, with quality close to larger models of the same backbone.

7 Conclusion

  • The paper proposes a small fine-tuned language model, PrefBERT, to evaluate long-form responses across dimensions and provide semantic-quality rewards for GRPO.The conclusion situates this approach against GRPO’s prior success on rule-based evaluation tasks.Source: §7 · p. 9
  • Models trained with PrefBERT generate higher-quality responses than models trained with ROUGE, BERTScore, or a generalized preference reward model, and approach larger models of the same backbone.The authors frame this as evidence for more efficient and robust verifiable reward design in GRPO for long-form, open-ended generation.Source: §7 · p. 9
  • Future work could test reward models and GRPO on creative writing, creative research and design, and open-ended math problems.The authors also propose developing more efficient and stronger verifiable reward models.Source: §7 · p. 9

8 Ethics

The authors state that their annotation protocol is exempt from Institutional Review Board review.

8 Ethics

  • The annotation collected no private information and involved no extensive tool usage; the authors therefore consider it exempt under the Institutional Review Board annotation protocol.Source: §8 · p. 9

9 Limitations

The authors identify incomplete exploration of GRPO reward design and the absence of a larger reward provider as limitations, citing computational constraints. The appendix supplies technical details on GRPO regularization and the Bradley–Terry loss used to train GRM-llama-3B.

9 Limitations

  • The study did not fully explore GRPO and reward design for enabling self-improvement on complex tasks.The authors identify this as an area for further investigation.Source: §9 · p. 9
  • No larger, more powerful language model such as a 7B-scale evaluator was used as a verifiable reward provider, primarily because of computational constraints.The authors note that larger evaluators could increase GPU memory use and slow training, though they hypothesize stronger rewards might unlock more of GRPO’s potential.Source: §9 · p. 9

A Technical Details

  • The appendix provides further technical details on GRPO regularization and the Bradley–Terry loss used by GRM.Source: A Technical Details · p. 13

A.1 GRPO Regularization Objective

  • The GRPO regularization description defines ρ_i(ϕ) as the probability ratio for y_i and identifies ϵ as the clipping hyperparameter and β as the KL penalty coefficient.Source: §A.1 · p. 13

A.2 GRM Bradley and Terry Loss

  • GRM-llama-3B is trained to minimize the Bradley–Terry loss; rθ(x, y) is its predicted reward score and σ(·) is the sigmoid function.The GRM supplies reward signals for GRPO, and its real-valued outputs are normalized to [0, 1].Source: §A.2 · p. 13

B Dataset Details

The appendix describes three long-form datasets, prompt templates, a response-structure check, and the human annotation workflow. The datasets cover varied open-ended tasks, while annotators score and rank model responses.

B Dataset Details

  • ELI5 draws from Reddit questions seeking accessible explanations and uses 10,444 training and 1,056 test questions.Its topics include chemistry, psychology, biology, and earth science.Source: B Dataset Details · p. 13
  • The cleaned Alpaca set filters out short responses and specified problematic examples, leaving 10,444 training and 1,334 test examples.The source collection contains diverse instruction-response pairs and long-form responses.Source: B Dataset Details · p. 13
  • LongForm combines human-written documents from several sources with LLM-generated instructions, excluding code-generation examples.It contains 8,648 training and 956 test examples.Source: B Dataset Details · p. 13

C Prompt Template

  • The appendix provides separate templates for model training, point-wise evaluation, and pairwise preference evaluation.Source: C Prompt Template · p. 13

D Markdown Expression Check

B Dataset Details

  • The experiments use long-form, open-ended datasets spanning diverse domains, with responses averaging 185 words.Table 4 includes dataset sizes and example inputs and reference responses.Source: B Dataset Details · p. 13, Table 4 · p. 14

E Annotation Tool

  • Annotators score each response on a 1–5 Likert scale, add comments, and rank responses by their preferences and ratings.The tool displays the prompt and answers from seven models.Source: E Annotation Tool · p. 14

F Showcases for Qualitative Analysis

The qualitative showcases compare PrefBERT with baseline models across readability, content logic, instruction following, and tone and fluency. They also include a comparison framed as a failure of SFT.

Readability

Content Logic

Instruction Following

Tone & Fluency

Failure of SFT

Loading 2506.15068v1…