PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models Can Learn from Verbal Feedback Without Scalar Rewards

Renjie Luo, Zichen Liu, Xiangyan Liu, Chao Du, Min Lin, Wenhu Chen, Wei Lu, Tianyu Pang

arXiv:2509.22638v1cs.CLcs.AIcs.LG

TL;DR

Scalar rewards can discard distinctions in verbal feedback and are difficult to keep consistent across tasks. FCP trains a policy to condition on verbal feedback, then bootstraps it using fresh feedback; across benchmarks, it matches or slightly exceeds scalar-based baselines.

  • Problem

    In LLM reinforcement learning, verbal feedback is often reduced to scalar rewards, which can lose information, obscure mixed feedback, and create scale imbalance across tasks.

    Source: §1 · p. 1

  • Method

    FCP learns responses conditioned on verbal feedback, then bootstraps the policy using positive-conditioned rollouts annotated with fresh feedback.

    Source: §1 · p. 2

  • Results

    Across math, IFEval, and general reasoning, bootstrapped FCP matched scalar-based baselines; math-suite average accuracy rose from 28.8% to 38.7%, slightly surpassing GRPO at 38.4%.

    Source: §3.2 · p. 7

  • Takeaways & Limitations

    FCP offers a framework for learning directly from rich verbal feedback without verifiers, scalar conversion, or data filtering.

    Source: §1 · p. 2, §5 · p. 9

    Length-related positive feedback conditions, such as conciseness, can destabilize bootstrapping and shrink average response length over time.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verbal feedback as a conditioning signal. Inspired by language priors in text-to-image generation, which enable novel outputs from unseen prompts, we introduce the feedback-conditional policy (FCP). FCP learns directly from response-feedback pairs, approximating the feedback-conditional posterior through maximum likelihood training on offline data. We further develop an online bootstrapping stage where the policy generates under positive conditions and receives fresh feedback to refine itself. This reframes feedback-driven learning as conditional generation rather than reward optimization, offering a more expressive way for LLMs to directly learn from verbal feedback. Our code is available at https://github.com/sail-sg/feedback-conditional-policy.

1 INTRODUCTION

The paper proposes learning directly from verbal feedback rather than reducing it to scalar rewards, and reports that FCP matches or surpasses strong scalar-based baselines. Its approach combines offline feedback conditioning with online bootstrapping from fresh feedback.

1 INTRODUCTION

  • Scalarizing verbal feedback loses critique distinctions, makes mixed or uncertain comments difficult to map, and creates inconsistent reward scales across tasks.Source: §1 · pp. 1–2
  • LLMs already integrate feedback prompts to refine responses in agentic tasks, but this feedback understanding is implicit and relies on test-time trial and error.Source: §1 · p. 2
  • FCP conditions responses on verbal feedback by reweighting a reference policy according to how likely each response is to elicit that feedback.This combines prior knowledge from the reference policy with feedback, including mixed feedback.Source: §1 · p. 2
  • Online bootstrapping samples responses under positive feedback conditions, obtains fresh environment feedback, and iteratively strengthens the policy.Source: §1 · p. 2
  • Pilot experiments show FCP matches or surpasses offline RFT and online GRPO without verifiers, scalar conversion, or data filtering.The authors describe the current implementation as naive and suggest advanced training techniques could further improve performance.Source: §1 · p. 2

2 LEARNING DIRECTLY FROM VERBAL FEEDBACK

The section proposes learning response distributions conditioned on verbal feedback, first from offline response-feedback pairs and then through online rollouts conditioned on positive feedback.

2.1 OFFLINE TRAINING: INITIALIZING FEEDBACK-CONDITIONAL POLICY

  • Offline maximum-likelihood training on response-feedback pairs approximates the feedback-conditional posterior and supports generation conditioned on arbitrary feedback.The pairs come from reference-policy responses and environment feedback; the learned policy matches the posterior on its support.Source: §2.1 · pp. 3–4
  • The positive-feedback posterior favors responses more likely to elicit favorable feedback, but cannot guide generation step by step because feedback arrives after the full response.The method learns an approximation rather than sampling directly from this posterior.Source: §2.1 · p. 3
  • Because positive-only feedback likelihoods are unavailable or intractable, the objective generalizes to learning from arbitrary feedback through forward-KL minimization and maximum likelihood.The passage notes that exclusively sampling positive feedback is infeasible without designed rubrics or filtering.Source: §2.1 · p. 3

2.2 ONLINE TRAINING: BOOTSTRAPPING BY CONDITIONING ON POSITIVE FEEDBACK

  • Online bootstrapping samples responses under positive feedback, obtains fresh environment feedback, and iteratively updates the policy to strengthen alignment with the specified feedback.The number of gradient steps between rounds can vary, interpolating between fully online and semi-online training.Source: §2.2 · pp. 4–5

3 EXPERIMENTS

The experiments compare FCP with scalar-based pipelines on mathematical, general-reasoning, and instruction-following tasks. Offline FCP is competitive with RFT, while bootstrapping brings it to results comparable to or slightly better than scalar-based RL; feedback conditions also shape behavior, though length-related conditions can destabilize bootstrapping.

3.1 SETUP

  • Experiments use Qwen2.5-7B-base, simulated GPT-5-nano feedback, and evaluations across math, general reasoning, and IFEval; the same feedback source supplies FCP conditions and scalar baseline rewards.Benchmarks are run under four seeds with mean accuracy reported; GRPO shares the feedback prompt and scalar scores.Source: §3.1 · pp. 5–6

3.2 MAIN RESULTS

  • Offline FCP scores 28.8% average math accuracy versus 30.4% for RFT, and 43.5% general-reasoning accuracy versus RFT’s 44.6%, while using noisier, unfiltered supervision.The base model scores 27.0% on math and 38.7% on general reasoning.Source: §3.2 · p. 6, Table 2 · p. 7
  • 38.7% average math accuracy for FCP+Bootstrap slightly exceeds GRPO’s 38.4%, while general-reasoning accuracy reaches 47.8%, matching the best scalar baseline at 47.5%.On IFEval, FCP+Bootstrap reaches 39.0%, comparable to GRPO at 38.5% and RFT+GRPO at 38.8%.Source: Table 1 · p. 6, §3.2 · p. 7, Table 2 · p. 7

3.3 LEARNING DYNAMICS OF FCP

  • On MATH500, FCP reaches 68.5% accuracy under fully_positive conditions and 17.1% under fully_negative conditions; code appears in 74.3% of has_code responses.Neutral conditions trade slightly lower accuracy for longer responses; unlike Qwen2.5-7B-Instruct, FCP varies behavior across conditions.Source: §3.3 · p. 7, Table 3 · p. 8
  • FCP reaches peak accuracy within 30 online steps like GRPO, although its scalar scores later lag because it does not optimize directly against the reward model’s preference.The authors report sustained high accuracy despite lower scores, which they interpret as avoiding reward hacking often seen in scalar-based methods.Source: Figure 2 · p. 6, §3.3 · p. 7
  • Length-related positive conditions destabilize bootstrapping: response lengths shrink and loss becomes less stable, whereas filtering those conditions yields steadily longer responses.The authors suggest a feedback loop in which concise rollouts receive affirming feedback and subsequent updates further shorten responses.Source: Figure 3 · p. 7, §3.3 · p. 8

4 ABLATION STUDIES

The ablations examine feedback quality and training choices, including comparisons between user-style and reviewer-style feedback and between FCP and CFT. Reviewer-style feedback performs better, but user-style feedback remains competitive; several design choices also improve or preserve accuracy.

4.1 REAL-WORLD USER VS. PROFESSIONAL REVIEWER STYLE

  • Using only user-style feedback lowers math-suite accuracy by 2.5 points versus reviewer-style feedback, yet gains +5.8 over the base model (27%; Table 1).User-style feedback is typically subjective and colloquial, while reviewer-style feedback is precise and structured.Source: §4.1 · p. 8

4.2 ADDITIONAL TRAINING DESIGN CHOICES AND COMPARISON TO CFT

  • A prompt batch size of 2048 with mini-batch size 512 yields four gradient updates per rollout step and better accuracy than the fully online setup.The authors suggest partial online updates can improve optimization efficiency.Source: §4.2 · p. 9
  • Sequence-level averaging with token-level summation gives a consistent +0.7% gain over token-level mean loss aggregation.The authors note that cross-entropy on self-sampled responses reduces to policy gradient with unit advantages, which suffers from length bias.Source: §4 · p. 8, §4.2 · p. 9
  • Reducing the training batch size to 256 improves accuracy by about +2.4%; a different online dataset slightly underperforms the default but remains +5.5% above offline-only initialization.The results indicate that offline and online datasets need not be strictly aligned for FCP to be effective.Source: §4.2 · p. 9
  • CFT with the same coarse, lightweight feedback used for FCP performs worse than the base model, despite CFT’s ability to work with high-quality detailed critiques.The comparison highlights FCP’s ability to leverage coarse, high-level feedback without costly fine-grained annotations.Source: §4.2 · p. 9

5 DISCUSSION AND FUTURE DIRECTIONS

The discussion frames online environmental interaction, rather than scalar rewards or a particular algorithm, as central to RL. It presents FCP as a way to use diverse verbal feedback during training and user-specified feedback at inference, and outlines extensions for verifiable rewards, multi-turn use, adaptation, and multimodal feedback.

5 DISCUSSION AND FUTURE DIRECTIONS

  • During training, feedback is generated by the environment, while at test time users can freely specify the conditioning feedback, enabling diverse training signals and controllability at inference.The authors say FCP preserves feedback richness, bypasses reward imbalance, and retains diverse feedback including mixed and uncertain cases, unlike RFT, which discards many useful pairs.Source: §5 · p. 9
  • Proposed extensions include combining verbal feedback with verifiable rewards, incorporating feedback across multi-turn interactions, adapting at test time from user examples, and making feedback multimodal.The paper presents these as future directions, not evaluated capabilities.Source: §5 · p. 9

A.1 PROOF OF EQ. (2) AND ITS SPECIAL CASE

The proof section relates the offline feedback-conditional posterior to a reward-maximization objective. In the special case of verifiable rewards, it states that the posterior is an optimal solution to 0-1 reward maximization without KL regularization.

A.1 PROOF OF EQ. (2) AND ITS SPECIAL CASE

  • The section frames the positive-feedback objective as KL-constrained maximization of expected log feedback likelihood relative to a reference policy.The text then states that the objective is equivalent to minimizing reverse KL between the policy and the offline feedback-conditional posterior.Source: §A.1 · p. 15
  • With verifiable rewards, the offline positive-feedback posterior is an optimal solution to 0-1 binary reward maximization without KL regularization.The condition assigns positive feedback probability 1 to correct responses and 0 to incorrect responses.Source: §A.1 · p. 15
  • The proof notes that the posterior is one of the optimal solutions, but not necessarily the unique one.Source: §A.1 · p. 15

A.2 CONNECTION TO INVERSE DYNAMICS MODELING

The appendix aligns SFT, CFT, and FCP with behavior cloning, forward dynamics, and inverse dynamics, respectively. It uses this analogy to describe their conditional structures and position verbal feedback as a first-class supervision signal for FCP.

A.2 CONNECTION TO INVERSE DYNAMICS MODELING

  • The appendix maps SFT to behavior cloning, CFT to forward dynamics, and FCP to inverse dynamics.Source: §A.2 · p. 16
  • The analogy places FCP as the inverse-dynamics counterpart to existing finetuning objectives by treating verbal feedback as a first-class supervision signal.Source: §A.2 · p. 16
  • The graphical models are presented as clarifying the conditional structure behind each finetuning paradigm and how supervision drives model learning.Source: §A.2 · p. 16

B RELATED WORK

Prior reasoning work has developed SFT and RL methods across mathematics, coding, and agentic tasks. Existing approaches increasingly use verbal feedback directly, while FCP treats it as a conditioning signal that can also accommodate mixed or noisy feedback.

B RELATED WORK

  • Research on reasoning has developed SFT and RL methods, including algorithmic variants and studies of training curricula and reward formulations, across mathematics, code, and agentic applications.The passage names Dr. GRPO (Liu et al., 2025a), DAPO (Yu et al., 2025), REINFORCE++ (Hu, 2025), and VAPO (Yue et al., 2025).Source: B RELATED WORK · p. 16
  • Recent efforts explore learning directly from feedback or critiques: CFT (Wang et al., 2025a) imitates critiques, Critique-GRPO (Zhang et al., 2025) guides online RL refinements, and Salemi & Zamani (2025) jointly optimize feedback and policy models.Source: B RELATED WORK · p. 16
  • FCP treats verbal feedback as a conditioning signal, allowing it to use mixed or noisy feedback without requiring high-quality, rubric-constrained input.The framework contrasts with approaches that treat feedback as a prediction target.Source: B RELATED WORK · p. 17

C DETAILED EXPERIMENTAL SETUP

The experimental setup describes the implementation tools, token initialization, online bootstrapping configuration, and GPT-5-nano feedback prompt. Hyperparameters for offline and online training are listed in Table 6.

C DETAILED EXPERIMENTAL SETUP

  • FCP implementations use llama-factory and verl, with offline- and online-stage hyperparameters listed in Table 6.The supplied table caption identifies its scope but gives no hyperparameter values.Source: C DETAILED EXPERIMENTAL SETUP · p. 17, Table 6 · p. 17
  • The <EF> and </EF> token embeddings are initialized from a multivariate normal distribution using the mean and covariance of existing token embeddings.General reasoning bootstrapping uses a fully online setup with batch size 256, unlike the math setting.Source: C DETAILED EXPERIMENTAL SETUP · p. 17
  • The GPT-5-nano feedback prompt elicits three outputs: low-quality real-world user-style feedback, high-quality professional reviewer-style feedback, and a scalar quality score.Figure 6 presents the unified prompt template.Source: C DETAILED EXPERIMENTAL SETUP · p. 17, Figure 6 · p. 18
Loading 2509.22638v1…