PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

Tianzhu Ye, Li Dong, Guanheng Chen, He Zhu, Xun Wu, Shaohan Huang, Furu Wei

arXiv:2607.18110v1cs.LGcs.CL

TL;DR

Scalar-reward RL discards rich evaluator feedback and obscures fine-grained differences among open-ended responses. Experiential Learning turns the evaluator into a coach that distills transferable guidance, and consistently outperforms rubric-based RL on held-out and unseen tasks while generalizing better beyond training distributions.

  • Problem

    Scalar-reward RL discards textual evaluator feedback, making responses with distinct quality profiles indistinguishable to the optimizer.

  • Method

    Experiential Learning uses an LLM-as-a-Coach to distill rubric-based assessments into transferable guidance for on-policy context distillation.

  • Results

    EL consistently outperforms rubric-based RL across policy families and feedback models on held-out and unseen open-ended tasks.

  • Takeaways & Limitations

    Experiential knowledge is a richer, more transferable learning signal that improves generalization beyond the training distribution and mitigates reward hacking.

  • Takeaways & Limitations

    EL does not address reward hacking caused by biased or miscalibrated feedback models, so improving feedback-model calibration remains complementary.

Abstract

from arXiv · show

Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.

1 Introduction

Open-ended tasks expose an information bottleneck in scalar-reward RL because evaluation captures multiple quality dimensions but discards textual distinctions. Experiential Learning addresses this by converting LLM coaching into transferable, dense supervision that improves learning and generalization beyond the training distribution.

  • Motivation: Open-ended responses depend on multiple qualities, while scalar-reward RL discards textual feedback and conflates responses receiving the same score.The passage identifies factuality, relevance, and completeness as examples of quality dimensions and describes the resulting information bottleneck.
  • Experiential Learning: Experiential Learning repurposes an LLM-as-a-Judge into an LLM-as-a-Coach that distills rubric-based response assessments into transferable guidance.The coach receives a prompt, on-policy response, and rubrics, then produces experiential knowledge for similar tasks.
  • Experiential Learning: EL internalizes coaching through on-policy context distillation, minimizing token-level reverse KL divergence between the policy and a context-conditioned teacher.This converts natural-language feedback into dense distributional supervision and consolidates it into policy parameters.
  • Feedback bandwidth: 17,600 bits is the theoretical upper bound for a 1024-token experiential context, compared with 3.3 bits for a discrete 1–10 score.The comparison uses a vocabulary of size 150,000 for the experiential context and describes the resulting bound as over 1,000× larger.
  • Results: EL consistently improves over rubric-based RL across two policy families and feedback models, including held-out in-distribution and unseen benchmarks.The feedback models comprise either the initial frozen policy checkpoint or a proprietary model accessed through an API.
  • Results: Scalar-reward optimization is susceptible to training-set reward overoptimization, whereas EL improvements transfer more effectively beyond the training distribution.A controlled distribution-matching analysis indicates that quantized scalar feedback preserves a binary target but discards finer distinctions in a multimodal target.

2 Method

Experiential Learning (EL) replaces scalar reward extraction with experiential knowledge from an LLM-as-a-Coach, then internalizes that feedback through context-conditioned teacher distillation. This preserves richer assessment information as dense token-level supervision for training policies on non-verifiable open-ended tasks.

  • RL Baseline: Standard RL extracts only a scalar reward from each on-policy response and discards the feedback LLM’s remaining textual output.The baseline is implemented with GRPO.
  • Experiential Learning: EL applies a feedback LLM to rubric-based assessments, distills transferable insights into experiential knowledge, and trains the policy toward a context-conditioned teacher.The teacher distribution is conditioned on both experiential knowledge e and prompt x, while the policy minimizes reverse KL divergence to that distribution.
  • Experiential Learning: EL uses experiential knowledge as its learning signal and converts its richer distributional information into policy weights through on-policy context distillation.The teacher may be the initial frozen policy checkpoint or an iteratively updated checkpoint from the preceding epoch.
  • Feedback Interfaces: Both interfaces use the same feedback LLM and prompt-specific rubrics, but an LLM-as-a-Judge exposes a scalar score whereas an LLM-as-a-Coach provides experiential knowledge.The distinction concerns which feedback the optimizer consumes, not the feedback model’s intrinsic capabilities.
  • Feedback Bandwidth: With L=1024 and |V|=150,000, EL’s textual channel has an upper bound of approximately 1024 × 17.2 ≈17,600 bits.By contrast, a discrete 1–10 RL score has a theoretical maximum bandwidth of log2(10) ≈3.3 bits per sample.

3 Experiments

Experiments compare rubric-based RL with EL across policy and feedback-model combinations, showing that EL generally improves held-out and unseen-task performance. Ablations and controlled analyses attribute this advantage to richer feedback while identifying teacher updates and context format as important for retaining out-of-distribution ability.

  • Experimental Setup: EL uses rubric-derived experiential knowledge as teacher context, whereas RL optimizes a 1–10 LLM-judge rating as a scalar reward.The experiments use Rubric-as-Reward with GRPO as the RL baseline and compare frozen-policy or GPT-4o feedback models.
  • Main Results: EL consistently outperforms RL on the WildChat test set and four additional benchmarks, with particularly large gains on AlpacaEval v2.0 and WildBench.The advantage holds when feedback comes from either the initial frozen policy checkpoint or GPT-4o.
  • Generalization and Reward Hacking: On a 1000-example WildChat training subset, RL gains more than EL, but EL scores higher on the WildChat test set, AlpacaEval v2.0, and WildBench.This train–test contrast is consistent with RL exploiting limited scalar feedback in ways that fail to transfer, while EL uses experiential-knowledge-induced distributional guidance.
  • Teacher Updates: Iterative teacher updates raise WildChat score from 80.0 to 80.7 but lower IFEval accuracy, while general-prompt OPD restores IFEval from 74.9 to 79.9 at the same WildChat score.General-prompt OPD therefore stabilizes iterative teacher updates against out-of-distribution forgetting.
  • Teacher Context: The default EL context performs best, while Full Critique and Rubrics Only reduce IFEval accuracy by biasing the teacher toward evaluation rather than task-solving behavior.Multiple-Choice preserves out-of-distribution performance but provides limited gains, whereas transferable experiential knowledge improves the WildChat test set.
  • Feedback Bandwidth: In the controlled five-level reward task, distributional matching recovers the target’s smooth bimodal shape, whereas RL produces a staircase distribution because equal maximum rewards cannot distinguish tokens with different target likelihoods.On the constructed binary task, both methods instead converge to nearly identical distributions matching the binary target.

4 Discussion

The discussion explains that EL mitigates reward hacking by preserving richer feedback, differs from RL through distribution matching rather than scalar reward maximization, and shares RL’s dominant online costs while adding modest method-specific overhead. Improving feedback-model calibration and accuracy remains complementary to EL’s focus on feedback bandwidth.

  • Reward Hacking: EL mitigates reward hacking by transmitting richer experiential knowledge instead of compressing feedback into a scalar reward.Scalar compression can let policies exploit score-inflating patterns without genuinely improving response quality.
  • Reward Hacking: Improving the feedback model’s calibration and accuracy remains complementary because EL addresses feedback bandwidth rather than feedback-model quality.
  • Distinction from Reinforcement Learning: RL maximizes a scalar reward and concentrates probability on highest-reward responses, whereas EL minimizes divergence from a context-conditioned teacher distribution.EL’s optimization is described as distribution-matching through on-policy context distillation, unlike RL’s reward-seeking objective.
  • Training Cost and Efficiency: Both RL and EL spend most end-to-end training time generating online data and invoking the feedback LLM.EL additionally performs a context-conditioned teacher forward pass and token-level distillation, while RL uses reward-based GRPO updates.

5 Related Work

Prior work studies learning from experience through environmental interaction, reflection, external memory, and textual feedback. Related distillation methods compress contextual knowledge into model parameters, including on-policy context distillation using the student’s own trajectories.

  • Learning from Experience: Learning from experience uses environmental interaction to accelerate future learning and help reasoning agents discover strategies through self-play and reflection.For language models, related approaches also reflect on past failures, store reusable knowledge in external memory, or optimize with multi-turn textual feedback.
  • Context Distillation and Self-Distillation: Context distillation compresses knowledge provided in context into model parameters, while on-policy distillation trains students on their own generated trajectories.On-policy context distillation extends these ideas with a context-conditioned teacher under on-policy training.

6 Conclusion

Experiential Learning (EL) is presented as a post-training framework for non-verifiable tasks that replaces scalar rewards with rich experiential knowledge. It uses an LLM-as-a-Coach to distill rubric-based assessments into transferable guidance and internalize that guidance through on-policy context distillation.

  • 6 Conclusion: EL post-trains language models on non-verifiable tasks using rich experiential knowledge rather than scalar rewards.The framework is designed to preserve information that scalar reward signals discard.
  • 6 Conclusion: EL repurposes the feedback model as an LLM-as-a-Coach that converts rubric-based assessments into transferable guidance.This reframes feedback from evaluation into experiential supervision for subsequent learning.
  • 6 Conclusion: EL internalizes the coach’s guidance through on-policy context distillation.The mechanism conditions learning on distilled experiential knowledge rather than only a scalar reward.
  • 6 Conclusion: The higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences that scalar rewards discard.This is the central information-preservation advantage attributed to EL in the conclusion.

A Details of Experiments · A.1 Prompt Templates

The experiments use distinct prompt templates for EL and rubric-based RL. EL extracts transferable experiential knowledge from coach evaluations, prepends it to the teacher context, while RL converts rubric evaluation into a scalar reward.

  • A.1 Prompt Templates: EL’s coach receives the same inputs as the evaluator and is additionally instructed to distill transferable experiential knowledge.The extracted knowledge is marked within <experience> tags.
  • A.1 Prompt Templates: Both evaluator prompts rate overall response quality on a 1-to-10 scale according to holistic rubric satisfaction.Responses violating multiple rubrics receive lower scores, whereas responses satisfying all rubrics receive higher scores.
  • A.1 Prompt Templates: Experiential knowledge is extracted from the coach’s <experience> tags.Figure 5 specifies the extraction mechanism.
  • A.1 Prompt Templates: The extracted experiential knowledge is prepended as context to the teacher model.This conditioning step uses the template shown in Figure 6.
  • A.1 Prompt Templates: RL’s judge evaluates each on-policy response against quality rubrics and produces a scalar score.The evaluator prompt asks for an overall quality rating from 1 to 10 based on holistic rubric satisfaction.
  • A.1 Prompt Templates: The RL prompt extracts an integer score from <score> tags for use as the scalar reward.Figure 7 defines the rubric-based scoring template and reward extraction.

A.2 Ablation Prompt Templates

The ablation configurations vary the scoring prompt and the context supplied to the teacher. They compare full textual critique, rubric-only context, and multiple-choice experiential directives.

  • Configuration Overview: Each ablation configuration specifies a scoring prompt sent to M and a context template used to condition the teacher.These templates define the implementation variants in Table 4.
  • Full Critique: Full Critique uses the RL scoring prompt and prepends M’s entire scoring output, including per-rubric analysis and score, as teacher context.The full textual output provides response-specific feedback.
  • Rubrics Only: Rubrics Only provides the rubrics as teacher context without a scoring prompt or response-specific feedback from M.This isolates rubric information from generated critique.
  • Multiple-Choice: Multiple-Choice extends the RL scoring prompt with an instruction to select one of 9 predefined directives.If M selects none, experiential knowledge is empty; otherwise, the selected directive conditions the teacher, yielding 10 context choices in total.

A.3 Rubric Examples

The section presents three rubric examples spanning different task types, with each training prompt paired with GPT-4o-generated rubrics that specify titles, descriptions, and importance weights.

  • A.3 Rubric Examples: Each training prompt is paired with GPT-4o-generated evaluation rubrics containing a title, description, and importance weight.The section provides three examples spanning different task types, as shown in Figure 11.

A.4 Experiential Knowledge Examples · A.5 End-to-End Example

A.4 presents three experiential-knowledge examples spanning different task types, emphasizing transferable strategies rather than instance-specific corrections. A.5 illustrates the complete EL pipeline, from prompt and rubrics through policy response, coaching, and teacher-model distillation.

  • A.4 Experiential Knowledge Examples: Three experiential-knowledge examples correspond to different task types and are extracted by the LLM-as-a-Coach during training.The examples are presented in Figure 12.
  • A.5 End-to-End Example: The end-to-end example shows the prompt, rubrics, on-policy response, and experiential knowledge extracted by the LLM-as-a-Coach.This complete EL pipeline is presented in Figure 13.
  • A.5 End-to-End Example: The scoring prompt requires selecting exactly one of nine improvement directives and outputting it unchanged inside <experience> and </experience> tags.The selected directive is extracted from the tags; with an empty output option, the multiple-choice ablation has 10 context choices.
  • A.5 End-to-End Example: The illustrated analytical task asks for pressing global issues, possible solutions, reasoning, and supporting evidence or examples.Its rubrics assess issue coverage, solution proposals, evidence and examples, reasoning clarity, and factual accuracy.
  • A.4 Experiential Knowledge Examples: The examples span open-ended analytical, technical coding, and recipe-scaling tasks, including a negative-weight pitfall criterion.These rubric examples are shown in Figure 11.
  • A.4 Experiential Knowledge Examples: Each example captures transferable strategies rather than corrections specific to a single instance.Figure 12 identifies this as the distinguishing characteristic of the extracted knowledge.
  • A.5 End-to-End Example: The coach appends experiential knowledge as context to the teacher model for on-policy context distillation after the policy generates a response.The pipeline begins with a prompt and rubrics, followed by policy generation, coaching, and context preparation.
Loading 2607.18110v1…