LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

• On open-ended tasks, scalar rewards discard evaluators’ textual feedback and make same-scored responses indistinguishable, including high-quality responses with subtle differences.These tasks lack a unique correct answer and involve multiple quality dimensions, such as factuality, relevance, completeness, organization, and style.Source: §1 · p. 1
• EL turns the evaluator into an LLM-as-a-Coach that converts on-policy responses into transferable guidance for producing better responses to similar tasks.The guidance captures reusable strategies rather than instance-specific corrections.Source: §1 · p. 2
• On-policy context distillation conditions a teacher on experiential knowledge, then trains the policy with token-level reverse KL so the knowledge is internalized without inference-time coaching.The policy is matched to the context-conditioned teacher along responses sampled from the policy.Source: §1 · p. 2
• A 1–10 score provides at most 3.3 bits/sample, versus 16 representational bits for bf16 or 17,600 theoretical bits for a 1,024-token experiential context.The context calculation assumes a vocabulary of 150,000 and represents a feedback-bandwidth intuition, not usable supervision; the paper says this is over 1,000× the scalar reward’s maximum bandwidth.Source: §1 · p. 2
• EL consistently improves over rubric-based RL across two policy families and feedback models on held-out and unseen open-ended tasks.Feedback comes either from the initial frozen policy checkpoint or a proprietary model accessed through an API.Source: §1 · pp. 2–3
• EL transfers more effectively beyond training distribution and mitigates reward hacking; in a simplified analysis distributional guidance preserves finer distinctions, while distilled experience beats raw critiques or rubrics.Iterative teacher updates improve task performance, and on-policy distillation over general-domain prompts mitigates forgetting in out-of-distribution capabilities.Source: §1 · p. 3

2.1 Preliminaries: RL with an LLM-as-a-Judge

• The RL baseline keeps only a scalar reward from each on-policy assessment and maximizes expected reward, using GRPO for implementation.Source: §2.1 · p. 4

2.2 Experiential Learning with an LLM-as-a-Coach

• For each on-policy response, EL distills the rubric-based assessment into experiential knowledge and trains the policy to match a context-conditioned teacher via reverse KL.The teacher can be the initial frozen policy or an iteratively updated checkpoint; on-policy context distillation internalizes the knowledge into policy weights.Source: §2.2 · p. 4

2.3 From Judge to Coach

• Both interfaces use the same feedback LLM and rubrics, but the judge supplies a score while the coach supplies transferable guidance that becomes dense token-level supervision.The distinction is what the optimizer consumes, not the feedback model’s intrinsic ability to produce textual analysis.Source: §2.3 · pp. 4–5

2.4 Feedback-Bandwidth Intuition

• EL’s text channel has a theoretical maximum of ≈17,600 bits/sample (L=1024, |V|=150,000), over 1,000× a bf16 scalar reward and 5,000× a discrete 1–10 reward.The corresponding RL maxima are 16 bits for bf16 and log2(10) ≈3.3 bits for discrete scoring; on non-verifiable tasks, EL can retain fine-grained preferences that scalar rewards lose when maximum-reward responses become indistinguishable.Source: §2.4 · p. 5

3.1 Setup

• Experiments train Qwen3-8B and OLMo-3-7B-Instruct on WildChat-IF, compare rubric-scored GRPO with coach-conditioned EL, and evaluate held-out prompts and four unseen benchmarks.WildChat-IF is a collection of 7,500 real user queries [ZRH+24] [HCZ+26] [BYC25]; GPT-4o evaluates the benchmarks, and GPT-4o and frozen-policy models each serve as feedback models.Source: §3.1 · pp. 5–6

3.2 Main Results

• EL consistently outperforms RL on held-out WildChat and transfers better to four unseen benchmarks, especially AlpacaEval v2.0 and WildBench, with either frozen-policy or GPT-4o feedback.Table 2 reports average scores or win rates across the top three checkpoints. GPT-4o feedback also improves WildChat scores for both methods relative to frozen-policy feedback.Source: Table 2 · p. 6, §3.2 · pp. 6–7

3.3 Experiential Learning Generalizes Better

• Despite smaller gains on a 1,000-example WildChat training subset, EL scores higher on the WildChat test set, AlpacaEval v2.0, and WildBench.The authors hypothesize that RL's scalar reward can encourage reward hacking, while EL's experiential-knowledge context steers learning toward more transferable behavior; they report similar results with Qwen3-8B and GPT-4o feedback.Source: §3.3 · p. 7, Figure 3 · p. 7

3.4 Ablations

• Iterative teacher updates raise WildChat score from 80.0 to 80.7 but reduce IFEval accuracy; general-prompt OPD raises IFEval from 74.9 to 79.9 while preserving the WildChat score.The general prompts use the initial frozen policy checkpoint as OPD teacher and are mixed into WildChat-IF at a 1:0.25 ratio.Source: Table 3 · p. 7, §3.4 · p. 7
• Default EL with extracted experiential knowledge performs best: Full Critique and Rubrics Only reduce IFEval accuracy, while Multiple-Choice preserves OOD performance but yields limited gains.Full Critique and Rubrics Only bias the teacher toward critiquing rather than task-solving; Multiple-Choice compresses feedback to log2 10 ≈ 3.3 bits, equivalent to a 1–10 scalar reward.Source: Table 4 · p. 8, §3.4 · p. 8

3.5 Analysis

• Feedback bandwidth scales as O(1) for RL, O(N) for OPD, and O(L) for EL, but EL's O(L) is a theoretical textual maximum, not a measure of effective supervision.A 1–10 reward has a theoretical maximum of 3.3 bits per sample; natural-language redundancy and imperfect extraction can make usable experiential information substantially smaller than the channel maximum.Source: §3.1 · p. 5, Table 5 · p. 8, §3.5 · p. 8
• In the toy analysis, RL matches a binary target about as well as distributional guidance, but five-level rewards yield a staircase on a bimodal target while distributional guidance recovers its smooth shape.This controlled construction illustrates feedback quantization rather than simulating the full training pipelines or proving a fundamental limit for all scalar-reward methods.Source: Figure 4 · p. 9, §3.5 · p. 9

Reward Hacking

• EL reduces reward hacking caused by scalar-reward bottlenecks through richer experiential knowledge, but does not fix feedback-model bias or miscalibration.Improving feedback-model calibration and accuracy remains complementary to EL, which addresses feedback bandwidth rather than feedback quality.Source: §4 · pp. 9–10

Distinction from Reinforcement Learning

• RL maximizes scalar reward, whereas EL minimizes divergence from a context-conditioned teacher distribution through on-policy context distillation.RL concentrates probability on the highest-reward responses and may exploit reward-correlated features; EL instead stays close to the teacher’s distributional shape.Source: §4 · p. 10

Training Cost and Efficiency

• EL’s teacher forward pass and token-level distillation add little cost relative to rollout sampling and feedback generation shared with RL.Both methods spend most end-to-end training time sampling policy rollouts and invoking the feedback model; RL uses reward-based GRPO updates.Source: §4 · p. 10

Learning from Experience

• Experience-based methods use environmental interaction, reflection, reusable memory, multi-turn feedback, and critique-conditioned RL, but remain prompt-off-policy and scalar-reward-bottlenecked [ZCL+25] [WWX25] [SCG+23] [ZHX+24] [YBB+24] [ATS+25] [LBF25] [ZZS+25] [HCZ+26].Prior work reports that experience accelerates future learning and supports strategy discovery through self-play and reflection.Source: §5 · p. 10

Context Distillation and Self-Distillation

• Context distillation compresses contextual knowledge into parameters [ABC+21] [CCL25]; on-policy distillation uses student-generated trajectories [GDWH24] [AVZ+24], and OPCD adds a context-conditioned teacher [YDW+26].Source: §5 · p. 10
• Related on-policy self-distillation typically uses the student as a context-conditioned teacher.Source: §5 · p. 10
• This paradigm has been extended to RLVR, agentic tasks, improved training algorithms, and online learning [YQS+26, LYF+26, KJLY26, WWX+26, WZS+26, YWL+26, XSZ+26, ZZS+26, ZDP+26, WCL+26, SXZ+26, YLX+26, YDD+26, BHP+26, WCJ+26].Source: §5 · p. 10
• Recent work investigates mechanisms behind self-distillation’s empirical success and potential degradation [LZH+26] [KLK+26] [ZBZ+26].Source: §5 · p. 10

6 Conclusion

• EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks across policy families and feedback models, with stronger generalization beyond the training distribution.The findings position experiential knowledge as an informative, transferable learning signal for non-verifiable tasks.Source: §6 · p. 10
• EL repurposes the feedback model as an LLM-as-a-Coach, distilling rubric-based assessments into transferable guidance and internalizing it through on-policy context distillation.This higher-bandwidth feedback provides dense supervision and preserves fine-grained preferences that scalar rewards discard.Source: §6 · p. 10
• The findings open directions for scalable iterative learning, improved knowledge extraction, and broader post-training of language models on complex open-ended tasks.Source: §6 · p. 10

A.1 Prompt Templates

• For EL, coach M receives the same inputs as the evaluator and is additionally asked to distill transferable experiential knowledge.The knowledge is extracted from the <experience> tags.Source: §A.1 · p. 15
• The evaluator prompt asks for an overall quality rating from 1 to 10 based holistically on how well the response satisfies the rubrics.It instructs lower scores for responses violating multiple rubrics and higher scores for responses satisfying all rubrics.Source: §A.1 · p. 15
• Figure 5 presents EL’s knowledge-extraction prompt, with text inside <experience> tags extracted as experiential knowledge e.The prompt asks the coach to distill experiential knowledge from its assessment.Source: Figure 5 · p. 15, §A.1 · p. 15
• The extracted experiential knowledge is prepended as context to the teacher model using the Figure 6 template.Figure 6 is the template for conditioning the teacher on experiential knowledge.Source: §A.1 · p. 15, Figure 6 · p. 16
• For RL, judge M evaluates each on-policy response against rubrics and produces a scalar score using the Figure 7 prompt template.The integer extracted from <score> tags is used as the scalar reward.Source: §A.1 · p. 15, Figure 7 · p. 16

Full Critique

• Full Critique uses the RL scoring prompt and gives the teacher M’s entire scoring output as context.The output includes per-rubric analysis and a score.Source: §A.2 · p. 16, Figure 8 · p. 17

Multiple-Choice

• Multiple-Choice extends the RL scoring prompt with a choice among 9 predefined directives, using the selected directive as experiential knowledge.If M outputs none of the directives, experiential knowledge is left empty, yielding 10 context choices; the context template matches default EL.Source: §A.2 · pp. 16–17

Rubrics Only

• Rubrics Only provides the teacher with rubrics as context, without response-specific feedback or a scoring prompt.Source: §A.2 · p. 16, Figure 9 · p. 17

A.3 Rubric Examples

• Each training prompt is paired with a set of evaluation rubrics generated by GPT-4o.Source: §A.3 · p. 17
• Each rubric includes a title, description, and weight indicating its importance.Source: §A.3 · p. 17
• The section presents three rubric examples spanning different task types, as shown in Figure 11.Source: §A.3 · p. 17

A.4 Experiential Knowledge Examples

• Three experiential-knowledge examples cover different task types and capture transferable strategies rather than instance-specific corrections.The examples include strategies such as maintaining an emotional arc, using accessible language, and avoiding misleading directions.Source: §A.4 · p. 17, Figure 12 · p. 20, §A.5 · p. 20

A.5 End-to-End Example

• The scoring prompt instructs the coach to select exactly one improvement directive, reproduce it unchanged, and wrap it in experience tags.Figure 10 describes a multiple-choice ablation with 10 context choices; when no required directive is output, experiential knowledge is left empty.Source: §A.5 · p. 18, Figure 10 · p. 18
• One example prompt asks for humanity’s most pressing issues and possible solutions, with reasoning and supporting evidence or examples.Its rubric includes weighted criteria for comprehensive issue coverage, solution proposals, evidence, reasoning clarity, factual accuracy, and empathetic tone.Source: §A.5 · p. 19
• The appendix’s rubric examples span an open-ended analytical question, a technical coding task, and recipe scaling with a negative-weight pitfall criterion.The end-to-end example’s rubrics assess issue coverage, solutions, evidence, reasoning, factual accuracy, and empathetic tone.Source: Figure 11 · p. 19, §A.5 · p. 19
• Another example prompt asks for a disclaimer clarifying that a mental wellness app is not clinical care or a substitute for professional clinical service.Source: §A.5 · p. 21
• The end-to-end example shows a policy responding to a prompt and rubrics, after which the coach extracts transferable knowledge for teacher-model context distillation.The extracted knowledge is prepended as context to the teacher model.Source: §A.5 · p. 17, Figure 13 · p. 21