PaperlayerView exact revision on arXiv

Source-linked AI summary

Expanding the Capabilities of Reinforcement Learning via Text Feedback

Yuda Song, Lili Chen, Fahim Tajwar, Remi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, Andrea Zanette

arXiv:2602.02482v2cs.LG

TL;DR

LLM reinforcement learning often relies on sparse scalar rewards, while dense demonstrations are costly and difficult to scale. This paper formalizes RL from Text Feedback, proposes Self Distillation and Feedback Modeling to internalize training-time feedback, and finds that both methods improve single-turn performance across diverse benchmarks. The paper also identifies practical limitations involving feedback quality, long-horizon interactions, and the scope of its theory.

  • Problem

    Sparse rewards reveal little about errors, whereas dense demonstrations are not scalable, motivating text feedback as an intermediate supervision signal.

  • Method

    RLTF uses feedback during multi-turn training but improves single-turn behavior through Self Distillation and Feedback Modeling.

  • Results

    Both proposed methods significantly improve single-turn test-time performance over strong baselines across reasoning, mathematics, and creative-writing benchmarks.

  • Takeaways & Limitations

    RL from text feedback provides a scalable alternative to expert demonstrations while addressing the sparsity of scalar rewards.

  • Takeaways & Limitations

    The paper notes that real-world feedback may be noisy or subjective, long-horizon interactions may require summarization, and its theory focuses on representations near the base-policy distribution.

Abstract

from arXiv · show

The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other extreme, distillation offers dense supervision but requires demonstrations, which are costly and difficult to scale. We study text feedback as an intermediate signal: richer than scalar rewards, yet cheaper than complete demonstrations. Textual feedback is a natural mode of human interaction and is already abundant in many real-world settings, where users, annotators, and automated judges routinely critique LLM outputs. Towards leveraging text feedback at scale, we formalize a multi-turn RL setup, RL from Text Feedback (RLTF), where text feedback is available during training but not at inference. Therefore, models must learn to internalize the feedback in order to improve their test-time single-turn performance. To do this, we propose two methods: Self Distillation (RLTF-SD), which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling (RLTF-FM), which predicts the feedback as an auxiliary objective. We provide theoretical analysis on both methods, and empirically evaluate on reasoning puzzles, competition math, and creative writing tasks. Our results show that both methods consistently outperform strong baselines across benchmarks, highlighting the potential of RL with an additional source of rich supervision at scale.

1 Introduction

The paper positions text feedback between sparse scalar rewards and costly demonstrations, then formalizes RLTF to use feedback during training while improving single-turn performance without feedback at test time. It proposes Self Distillation and Feedback Modeling, supported by theoretical analysis and broad empirical evaluation.

  • Sparse scalar rewards provide little information about what went wrong or how to fix an unsuccessful trajectory.
  • Demonstrations provide dense supervision, but distillation is not applicable to frontier-model training and human demonstration collection is not scalable.
  • Text feedback is richer than scalar rewards yet cheaper than complete demonstrations, and can localize errors, identify violated constraints, or suggest fixes.
  • Standard multi-turn RL can improve revision with feedback, but feedback is often unavailable at test time when users expect a good first response.
  • RLTF-SD distills feedback-conditioned second attempts into the single-turn policy, while RLTF-FM predicts critiques as an auxiliary objective.
  • Both methods significantly improve single-turn test-time performance over strong reward- and text-feedback baselines across reasoning, mathematics, and creative-writing benchmarks.

2 RL from Text Feedback

RLTF formalizes training with feedback-augmented multi-turn trajectories while evaluating the policy on initial prompts without feedback. The central objective is to convert training-time feedback into improved single-turn competence rather than merely optimizing revision performance.

  • The interaction samples an initial prompt and output, evaluates reward on the original prompt, and obtains text feedback from a feedback provider.
  • Later prompts incorporate previous outputs and feedback, potentially through simple concatenation, and the policy continues sampling outputs, rewards, and feedback.
  • Standard multi-turn RL maximizes cumulative rewards across the interaction, treating augmented prompts as states in an episodic MDP.
  • This objective does not isolate feedback's role because the policy may treat feedback as uninformative context or ignore it entirely.
  • RLTF instead evaluates single-turn performance on initial prompts without additional test-time feedback and asks how training trajectories can improve that objective.
  • The framework addresses this question with two complementary methods.

3 Self Distillation

Self Distillation uses feedback-conditioned second-turn outputs as implicit supervision for the original one-shot policy, while its baseline and importance-weighting analysis targets useful, stable learning signals. The analysis identifies gradient collapse and high variance as key hazards and motivates lower-variance alternatives.

  • Self Distillation: Self Distillation samples a first attempt and feedback, generates a revised second attempt, and uses that revision to update the policy on the original prompt.
  • Self Distillation: The distillation objective can recover an unbiased gradient for the single-turn objective when second-turn samples are importance-weighted under the stated support condition.
  • Baselines: Second-turn group-mean baselines can collapse the learning signal when second-turn rewards are constant or nearly constant.
  • Baselines: When feedback makes the second-turn policy highly reliable, the probability of a non-zero update scales approximately as 1 − p1^N ≈ N(1 − p1), leaving little first-turn signal.
  • Baselines: First-turn baselines avoid this collapse because the update remains non-trivial when the student is imperfect, even if the second-turn policy is highly reliable.
  • Bias-variance tradeoff in importance weighting: Importance weighting can create heavy-tailed, high-variance gradients when first- and second-turn policies shift, with token-level shifts compounding across long outputs.
  • Bias-variance tradeoff in importance weighting: Removing importance weighting consistently improves stability and final performance over full correction or clipping, indicating that variance dominates bias in the experiments.
  • Bias-variance tradeoff in importance weighting: Rejection Sampling avoids importance-weight variance but negative second-turn samples contribute nothing without a baseline, and it underperforms baselined methods empirically.

4 Feedback Modeling

Feedback Modeling trains the policy to predict critiques as an auxiliary objective, using dense feedback to improve representation learning and enable self-critique at inference.

  • Feedback Modeling: Feedback Modeling trains the policy to predict text feedback from interaction tuples as an auxiliary supervised objective.The feedback loss uses critique tokens, treats the model’s response as constant, and is combined with the multi-turn RL objective.
  • Theoretical analysis: Reward-only learning suffers from rare-event estimation and weak identification of representation directions under sparse base-policy rollouts.Estimating a gradient component can require on the order of 1/ε0 rollouts, while updates may have negligible projection onto a low-signal subspace.
  • Theoretical analysis: RLTF-FM supplies an additional supervised signal in representation directions weakly identified by sparse reward, improving representation conditioning under feedback coverage.The analysis characterizes feedback modeling as a representation preconditioner in an idealized frozen-rollout, log-linear-policy setting.
  • Test-time scaling: Because the same policy produces feedback predictions, RLTF-FM supports inference-time self-critique and iterative refinement without a separate learned judge.The model samples an answer, generates a critique, updates the prompt, and resamples an answer over multiple rounds.

5 Experiments

Experiments evaluate RLTF-SD and RLTF-FM across reasoning, mathematics, and creative-writing tasks, testing general performance and the design choices behind self distillation.

  • 5.2 Ablation on Self Distillation: The proposed self-distillation design choices outperform GRPO-baseline, PPO-clipping, and CISPO-clipping alternatives in both single-turn and multi-turn accuracy.Ablations compare baseline selection and importance-weight clipping on Knights and Knaves and MATH500.
  • General Results: RLTF-SD and RLTF-FM consistently outperform all baselines on single-turn performance across reasoning puzzles, competition math, and creative writing.The evaluation uses 2-turn training and compares final single-turn performance across diverse benchmarks.
  • General Results: Naive multi-turn GRPO performs similarly to single-turn GRPO, indicating that adding feedback as context alone is insufficient to internalize its learning signal.Feedback Descent also underperforms the proposed methods, supporting parameter-space optimization in this setting.
  • General Results: RLTF-SD is stronger on creative writing, whereas RLTF-FM performs better on math and reasoning tasks with more subjective feedback.The reported pattern is associated with teacher-student distribution mismatch and feedback subjectivity.
  • 5.3 Ablation on Feedback: Correctness-only feedback performs worse than semantically rich text feedback for RLTF-SD and multi-turn GRPO, with a noted exception for single-turn Knights and Knaves under multi-turn GRPO.Without distillation, neither feedback type substantially changes first-turn responses in that setting.
  • Test-time Scaling: Self-critique RL alone is insufficient for test-time scaling, while adding the RLTF-FM loss increases improvement that saturates after a handful of rounds.The reported benefit is primarily the magnitude of improvement rather than the number of improving rounds.

6 Related Work

Related work situates RLTF among language-correction learning, distillation, world modeling, and multi-turn RL, while distinguishing its use of unavailable-at-test-time feedback.

  • Learning from text feedback: Language-correction methods use human instructions grounded in robot perception and action spaces to update policies or value functions.These approaches provide corrections such as directional movement instructions in embodied settings.
  • Learning from text feedback: Other text-feedback methods densify feedback into numerical rewards, optimize in text space, propagate subgraphs, or use feedback as a hindsight goal.RLTF instead treats feedback as a rich training signal for policy learning without requiring it at inference.
  • LLM distillation: LLM distillation research includes teacher-distribution imitation, on-policy distillation from student generations, and self-distillation.The related work distinguishes these supervision paradigms from RLTF’s feedback-conditioned training setup.
  • LLM world models: World-modeling methods learn future states and rewards for imagined rollouts, while related LLM work models text feedback or environment interaction.These lines of work motivate modeling information beyond sparse reward signals.
  • Multi-turn RL: Multi-turn RL studies long-horizon interaction with external environments; RLTF treats the feedback provider as the environment affecting second-turn generation.The paper’s setting emphasizes internalizing feedback because it is often unavailable at test time.

7 Conclusion and Discussion

RL from text feedback addresses sparse reward supervision with a scalable alternative to expert demonstrations. Self Distillation and Feedback Modeling show favorable theoretical and empirical properties across reasoning, math, and creative writing tasks.

  • Conclusion and Discussion: RL from text feedback addresses reward sparsity while providing a scalable alternative to expert demonstration.The paper positions text feedback as an intermediate supervision source for reinforcement learning.
  • Conclusion and Discussion: Self Distillation and Feedback Modeling demonstrate strong empirical performance across reasoning, math, and creative writing tasks.These are the paper’s two proposed methods.
  • Conclusion and Discussion: The paper identifies noisy or subjective feedback, long-horizon interaction, context limits, and incomplete end-to-end theory as limitations.It also proposes studying interactions with other fine-grained supervision methods.

Consider the importance-corrected gradient estimator

The analysis distinguishes unbiasedness in expectation from point-wise signal collapse. It shows that second-turn group-mean baselines can eliminate updates when rewards are constant, whereas a first-turn mean baseline remains unbiased with importance correction.

  • Gradient estimator analysis: A first-turn group-mean baseline is unbiased in expectation when paired with importance correction.The analysis states that it does not introduce bias in expectation.
  • Gradient estimator analysis: Unbiasedness concerns the conditional mean of the estimator, whereas signal collapse concerns groups receiving an identically zero update.Thus an estimator can have the correct expected gradient while failing to provide a learning signal for a nontrivial fraction of groups.
  • Gradient estimator analysis: A second-turn mean baseline causes deterministic collapse when all group rewards are equal, regardless of importance weights.Every advantage becomes zero under constant second-turn rewards.
  • Gradient estimator analysis: For concentrated second-turn rewards, advantages become uniformly small and update magnitudes shrink even when the estimator is unbiased.This occurs because the advantage is tied to empirical reward variance.

B.3 Discussion on Alternative Baselines

The trajectory-level improvement advantage can have higher variance and discard useful successful revisions compared with the first-turn mean baseline. In sparse-reward settings, the first-turn baseline offers better normalization and retains more learning signal.

  • Alternative baselines: The improvement advantage measures revision gain for each trajectory but can have higher variance than the first-turn mean baseline.Its variance includes the first-turn reward variance and a covariance term with second-turn rewards.
  • Alternative baselines: When first-turn success is rare and reward correlation is weak or negative, the improvement baseline pays an extra variance term of order Var(R0).The first-turn mean baseline pays only Var(R0)/N in the corresponding comparison.
  • Alternative baselines: The improvement baseline discards trajectories where both attempts succeed, even though the corrected second-turn output may still be useful for distillation.The first-turn baseline continues reinforcing successful corrected outputs when the first-turn policy remains imperfect.
  • Alternative baselines: When first-turn rewards are near zero, the improvement baseline behaves like raw post-feedback reward and loses prompt-level normalization.This weakens the stability benefit of the first-turn baseline as post-feedback success becomes high.
  • Alternative baselines: Under a frozen rollout distribution, rare-event binary rewards bound directional SNR by √ε0 and require large batches to recover gradient signs reliably.Here ε0 is the base pass rate.

C.2.2 Weak identifiability of representation directions under success conditioning

Under frozen base rollouts, reward-only learning may provide weak information about some representation directions, even when conditioning on successful samples. The resulting low-signal subspace can constrain early progress toward reward improvements.

  • Reward-only updates can have negligible projection onto a nontrivial low-signal representation subspace under frozen base rollouts.This limitation is geometric and can persist beyond finite-sample estimation noise.
  • The analysis assumes a frozen rollout distribution and holds the linear head fixed, isolating representation directions rather than proving a general optimization limitation.
  • The low-signal subspace is defined from small eigenvalues of the success-conditioned representation-score second-moment matrix.Small eigenvalues indicate directions carrying little score information even among successful samples.
  • If successful samples provide insufficient cumulative score second moment, projected progress remains limited and an early-stage plateau can occur.

C.3.3 Result: representation-learning benefit

RLTF-FM supplies auxiliary feedback supervision that moves shared representations in directions where reward-only learning has weak signal under base rollouts. The result is an identifiability benefit, not a direct reward-improvement guarantee.

  • Theorem C.2 decomposes this movement into a covariance coverage term and a squared mean-drift term.
  • For large T, systematic feedback-model/feeder moment mismatch can dominate movement, while covariance coverage matters when drift is weak or broad conditioning is desired.
  • The theorem is a frozen-score, linearized initialization analysis and does not establish reward improvement; trajectory-level analysis would require tracking changing feedback statistics.
  • RLTF-FM induces nontrivial representation movement in the reward-only low-signal subspace before the rollout distribution shifts.

C.4 Proofs

The proofs formalize two limitations of sparse reward learning and illustrate how feedback supervision can address weak representation signal. They also include case studies where critiques correct failed reasoning and path-search attempts.

  • Proofs: Under rare-event rewards, estimating a gradient component with controlled relative error requires sample complexity scaling as Ω(1/SNR(u)^2), while SNR(u) is bounded by √ε0.
  • Proofs: In the frozen-rollout regime, reward-only iterates cannot reach a reward-improvement superlevel set until projected displacement satisfies ρ ε0 E_t ≥ r∆.
  • Proofs: The FM proof uses i.i.d. feedback samples with mean mFM and covariance CFM to establish movement in the reward-only low-signal subspace.

D.3 Details of Benchmarks

The evaluation spans reasoning puzzles, competition mathematics, and creative writing, with task-specific datasets and metrics. The experiments also examine importance-weighting variance and compare RLTF methods with baselines.

  • Benchmarks: Reasoning Gym evaluation covers Knights and Knaves, Binary Matrix, and Shortest Path; these tasks test logic, matrix distances, and obstacle-constrained path finding.
  • Benchmarks: Math evaluation uses MATH500’s 500 competition problems and the 30-question AIME24 2024 test set.
  • Benchmarks: Creative-writing evaluation trains on 43K LitBench pairwise examples and tests on 2K LitBench examples plus 1K real-world WritingBench tasks.
  • Variance analysis: Trajectory-level importance weighting is infeasible because direct policy-ratio variance is almost vacuously large; token-level ratios also show high-variance tails.
  • Metrics and comparisons: Table 4 reports second-turn accuracy after two-turn training, using mean@1 for reasoning and LitBench, mean@32 for math, and a 1–10 score for creative writing.

D.6 Additional Results

Additional results report evaluation curves for single-turn and multi-turn accuracy across reasoning, math, and creative-writing tasks, with evaluations every 40 training steps.

  • Figure 5 reports single-turn accuracy curves across reasoning puzzles, competition math, and creative writing tasks.Reasoning and math use mean@1 accuracy normalized from 0 to 1; creative-writing scores are normalized from 1 to 10.
  • Figure 6 reports multi-turn accuracy curves across reasoning puzzles, competition math, and creative writing tasks.The same task-specific mean@1 metrics and normalization ranges are used for multi-turn evaluation.
  • All training experiments are performed using Tinker.
Loading 2602.02482v2…