PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

Yujin Kim, Namgyu Ho, Sangmin Hwang, Joonkee Kim, Yongjin Yang, Sangmin Bae, Seungone Kim, Jaehun Jung, Se-Young Yun, Hwanjun Song

arXiv:2607.04412v2cs.AI

TL;DR

Static training prompts can become misaligned with an improving policy, weakening reward discrimination in non-verifiable instruction following. LLM-as-a-Tutor detects non-challenging prompts through pairwise rollout comparisons and appends atomic constraints, outperforming policy-unaware and prior policy-adaptive baselines across three benchmarks.

  • Problem

    Static prompts can become misaligned with the evolving policy, causing rollout groups to offer little quality contrast for reward discrimination.

  • Method

    LLM-as-a-Tutor uses one LLM to compare rollout pairs, detect non-challenging prompts, and append atomic constraints that increase difficulty while preserving the seed prompt.

  • Results

    Across three instruction-following benchmarks, the method consistently outperforms policy-unaware and prior policy-adaptive baselines.

  • Takeaways & Limitations

    Prompt adaptation is a missing axis of policy-awareness in non-verifiable RL, complementary to rubric adaptation.

  • Takeaways & Limitations

    The append-only constraint design is aligned with constraint-additive instruction-following tasks, while other domains require different modification templates.

Abstract

from arXiv · show

Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.

1 Introduction

LLM-as-a-Tutor addresses policy-prompt mismatch in non-verifiable RL, where non-challenging prompts prevent LLM judges from producing discriminative rewards. It uses pairwise rollout comparison to detect such prompts and appends atomic constraints, outperforming policy-unaware and policy-adaptive baselines on three instruction-following benchmarks.

  • Motivation: When prompts are too easy or too hard for the current policy, rollouts lack quality variance and the rubric judge’s reward signal collapses.Prompt challengingness is a property of the policy-prompt pair, not the prompt alone.
  • Method: LLM-as-a-Tutor detects non-challenging prompts by pairwise comparison of policy rollouts, exploiting LLMs’ sharper and more human-aligned pairwise judgments.The method extends the LLM’s role from judge to tutor.
  • Method: For each seed prompt, the tutor preserves the base prompt and rubric while appending an atomic constraint and corresponding rubric criteria when rollout quality is indistinguishable.An examiner samples rollout pairs at each iteration, and a generator performs the prompt and rubric augmentation in a single call.
  • Evaluation: On FollowBench, AdvancedIF, and InfoBench, LLM-as-a-Tutor achieves the highest average score against fixed-prompt and prompt-modification baselines.The evaluation uses Qwen3-1.7B as the policy and Qwen3-8B as the tutor.
  • Analysis: Pairwise judgments correlate with empirical difficulty, appended constraints amplify the base rubric’s discriminative signal, and modification rates rise monotonically with policy scale.These findings support prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.

2 Related work

Prior work addresses reward-signal weakness through curriculum-based prompt selection, instance-specific rubric rewards, or policy-adaptive training distributions. However, existing prompt-evolving methods rely on scalar reward models or generic rubrics, motivating policy-adaptive prompt evolution with fine-grained rubric judges.

  • Curriculum and reward variation: RL curriculum methods target reward variation by selecting tasks near the agent’s competence frontier or filtering prompts with extreme empirical pass rates.These approaches are motivated by the absence of policy-gradient signal when sampled trajectories receive identical returns.
  • Rubric-based rewards: Standard scalar reward models are coarse and brittle, while RLCF, Rubrics as Rewards, and AdvancedIF use per-prompt checklists scored by LLMs for finer-grained on-policy rewards.Scalar signals are described as vulnerable to reward hacking, surface-feature bias, and instability under optimization.
  • Policy-adaptive training distributions: DR Tulu evolves rubrics, whereas EVA and LSP evolve prompts but use scalar reward models or generic prompt-agnostic rubrics instead of instance-specific rubric signals.The proposed direction extends policy-adaptive prompt evolution to the rubric-based setting with pairwise saturation checks and appended atomic constraints.

3 Method

LLM-as-a-Tutor adapts prompts and rubrics according to whether the current policy produces discriminative rollouts. A tutor compares rollouts, appends atomic constraints to non-discriminative prompts, and iteratively escalates difficulty while leaving the reward and GRPO objective unchanged.

  • Motivation: Prompt–policy alignment determines whether rollout groups provide sufficient score spread for GRPO advantages to deliver a meaningful learning signal.Static prompts can become misaligned as the policy improves, regardless of the quality of the base rubric or judge.
  • Constraint-based prompt adaptation: Discriminative prompt–rubric pairs remain unchanged, and adapted pairs use the same reward and GRPO objective as the baseline.Adaptation is the sole locus of change relative to the fixed-prompt baseline.
  • Pairwise discriminativeness judgment: At each adaptation interval, the tutor compares two current-policy rollouts and marks a prompt non-discriminative when their quality is indistinguishable.Pairwise comparison is used because the paper reports that LLMs are sharper and more human-aligned in pairwise than pointwise scoring.
  • Constraint-based prompt adaptation: For a non-discriminative prompt, the tutor appends one atomic constraint and adds matching rubric criteria, while jointly renormalizing the original and added rubric weights.The constraint imposes a single requirement along a dimension left unspecified by the original prompt.
  • Iterative difficulty escalation: Append-only adaptation monotonically increases difficulty because satisfying an adapted prompt also satisfies its predecessor, while retaining the original rubric criteria.Prompts can accumulate constraints as the policy improves, unlike rewriting strategies that provide no such guarantee.
  • Iterative difficulty escalation: The tutor is invoked at every chosen training interval, revisiting earlier adapted prompts and appending a new constraint whenever the current policy again makes one non-discriminative.The update is defined as x(t+1) = x(t) ⊕ c(t) when x(t) is non-discriminative for πθt.

4 Experiments

Across three instruction-following benchmarks, LLM-as-a-Tutor achieves the highest average and outperforms baselines on five of six metrics. Ablations show that policy-aware prompt selection and append-only modification are central to its gains.

  • Overall performance: LLM-as-a-Tutor achieves the highest average score across three benchmarks and outperforms all baselines on five of six metrics.The setup trains Qwen3-1.7B-Thinking as policy and Qwen3-8B-Thinking as tutor and judge, using three epochs of WildChat prompts.
  • Overall performance: Policy-adaptive methods outperform policy-unaware methods, while Evol-Instruct underperforms Base rubrics, 50.24 vs. 50.51.Policy-adaptive rubrics and EVA reach the same average (51.04), whereas policy-unaware methods yield more modest improvements.
  • Overall performance: Prompt adaptation outperforms rubric adaptation because changing prompts can restore rollout quality gaps when scoring rubrics cannot produce discriminative rewards.Rubrics affect scoring, whereas prompts affect the rollouts produced by the policy.
  • Selection ablation: Adaptive constraint selection outperforms applying constraints to all prompts or selecting them randomly, showing that additions must target the right prompts.The Random variant modifies 28% of prompts, matching the average constraint-added ratio across three epochs.
  • Selection ablation: Pairwise judgments outperform variance-based selection, whose threshold is τ = 3 and whose reward statistics serve as proxies for prompt learnability.The variance-based alternative selects prompts with reward standard deviation below the fixed threshold.
  • Modification ablation: Append outperforms Reset and Rewrite on all benchmarks because retaining prior atomic constraints supports monotonically increasing difficulty.Reset discards earlier constraints, while Rewrite regenerates the entire prompt and may drift from the original instruction.
  • Base-model robustness: On Qwen3-1.7B Base without instruction tuning, LLM-as-a-Tutor outperforms the policy-unaware baseline on average, 37.63 vs. 36.38.The method also outperforms baselines across all instruction-following benchmarks in this base-model setting.

5 Analysis

The tutor’s challenge decisions track empirical policy difficulty: as policy scale increases, more prompts require appended constraints, while non-challenging prompts exhibit higher reward and lower variance. Append-only atomic constraints raise difficulty and restore discriminative reward signals on saturated prompts.

  • Policy scaling and difficulty: Constraint augmentation rises monotonically from 8.1% to 40.5% as the policy scales from 0.6B to 4B with the tutor fixed at 8B.Larger policies make seed prompts relatively easier, producing high-quality rollouts with little variation and causing the tutor to identify more prompts as non-challenging.
  • Policy scaling and difficulty: Non-challenging prompts have higher reward mean (90.76 vs. 78.24) and lower standard deviation (12.96 vs. 27.07) than challenging prompts.Challenging prompts show the opposite pattern, indicating that the tutor’s decisions track empirical policy performance.
  • Constraint effects: Adding more appended constraints generally lowers reward mean under the base rubrics, because the policy must satisfy additional constraints simultaneously.This behavior is consistent with the append-only design’s monotonic increase in prompt difficulty.
  • Constraint effects: A single atomic constraint breaks saturation: all original rollouts score 100 with std = 0, whereas constrained rollouts have reward mean 92.5 with std 14.9.The added clause demands a specific named policy, separating rollouts that explicitly name one from those that defer to a supervisor without naming one.

6 Discussion

The discussion presents LLM-as-a-Tutor as a potential form of distillation that extends reward-guided adaptation from responses to prompt distributions. It also frames atomic-constraint appending as benchmark-aligned while describing the broader pipeline as task-agnostic.

  • LLM-as-a-Tutor as a potentially stronger form of distillation: LLM-as-a-Tutor may constitute a new form of distillation by using the tutor’s discriminative capability to identify when harder prompts are needed and construct them.The framework extends rubric-based RL’s principle from response rewards to the prompt distribution.
  • Generalizing the LLM-as-a-Tutor framework beyond constraint-additive tasks: Appending atomic constraints aligns with FollowBench-style instruction following, where difficulty is defined by the number of constraints a response must simultaneously satisfy.The discussion presents constraint count as the natural difficulty notion for non-verifiable instruction following, while describing the broader pipeline as task-agnostic.

7 Conclusion

LLM-as-a-Tutor addresses policy–prompt mismatch in rubric-based non-verifiable RL by reusing the judge LLM to adapt training prompts. Its append-only constraints create a self-calibrating curriculum that tracks policy capability, and it consistently outperforms policy-unaware methods across three instruction-following benchmarks.

  • 7 Conclusion: LLM-as-a-Tutor reuses the judge LLM to adapt training prompts when seed prompts fail to elicit rollout-quality variance.The tutor appends targeted constraints to such prompts.
  • 7 Conclusion: The append-only adaptation yields a self-calibrating curriculum that tracks policy capability without external difficulty schedules.Difficulty increases through targeted constraints appended to seed prompts.
  • 7 Conclusion: Across three instruction-following benchmarks, LLM-as-a-Tutor consistently outperforms policy-unaware methods.The supplied passage also indicates comparisons with prior methods, but its text is truncated before specifying them.

A Limitations … D Experimental setup

The paper reports limitations in evaluation scope, tutor capability, and inference cost, while noting dual-use risks and presenting the training procedure’s core steps. The method appends tutor-generated constraints and corresponding rubric updates when rollouts are judged saturated.

  • A Limitations: Main results use one Qwen3-1.7B policy–Qwen3-8B tutor pair, with broader model-family and policy-size evaluation omitted because of compute constraints.Figure 2a varies the policy from 0.6B to 4B while holding the tutor fixed.
  • A Limitations: The method depends on tutor reliability for pairwise discrimination and atomic constraint generation; weaker tutors yield noisier saturation judgments and less targeted constraints.The Wrong baseline illustrates this issue when judgments use a stronger model’s rollouts rather than the policy’s.
  • A Limitations: Each prompt uses two sample rollouts and one saturation-judgment call, plus one constraint-generation call when judged saturated, adding tutor inference during training.The existing rubric-based RL pipeline uses 8 policy rollouts and 8 × N judge invocations per prompt.
  • B Broader impacts: The method improves reliable instruction following for open-ended applications such as writing assistance and document drafting, and could combine with safety- or helpfulness-oriented objectives.Its training-time mechanism is described as general.
  • B Broader impacts: Because the task-agnostic adaptation procedure can strengthen compliance with harmful instructions, its behavior inherits values implicit in seed prompts and tutor judgments.The paper identifies dual-use potential when seed corpora or rubrics target undesirable objectives.
  • C Pseudocode: Algorithm 1 initializes base rubrics, then repeatedly generates two rollouts, invokes the tutor to judge saturation and propose a constraint, and appends accepted constraints and rubric updates.The updated dataset is subsequently used for rubric-based RL policy training.

D.1 Training

The experiments use the VeRL framework, with training hyperparameters summarized in Table 5.

  • All experiments are implemented using the VeRL framework.
  • Training hyperparameters are summarized in Table 5.
  • Table 5 is titled “Training hyperparameters.”

D.2 Evaluation · D.3 Licenses for existing assets · D.4 Compute resources

The paper evaluates policy models across three instruction-following benchmarks using repeated runs and a unified vLLM pipeline, while documenting asset licensing and H100-based compute requirements. Evaluation covers specified benchmark categories, publicly available assets, and approximately one-day training runs on 4×H100 GPUs.

  • D.2 Evaluation: Policy models are evaluated on FollowBench, AdvancedIF, and InfoBench using a unified vLLM-based generation pipeline.The evaluation uses three instruction-following benchmarks.
  • D.2 Evaluation: 5 independent evaluation runs are averaged for every benchmark to address stochasticity in LLM-based judging.The paper attributes this repetition to non-trivial scoring stochasticity from LLM-based judges.
  • D.2 Evaluation: FollowBench evaluation includes the content, situation, style, format, and mixed categories, excluding the example category.These are the categories specified for evaluation in the passage.
  • D.3 Licenses for existing assets: All assets are publicly available and used under their respective licenses for non-commercial academic research.The passage lists licenses for the models, dataset, and benchmarks used in the work.
  • D.3 Licenses for existing assets: Qwen3-1.7B, Qwen3-8B, and FollowBench use Apache 2.0, WildChat uses ODC-BY 1.0, AdvancedIF uses CC-BY-NC-4.0, and InfoBench uses MIT.AdvancedIF is used solely for evaluation in a non-commercial research context.
  • D.4 Compute resources: All experiments run on NVIDIA H100 GPUs, with each main Table 1 result using 4×H100 for approximately one day end-to-end.The stated duration includes rollout generation, tutor invocation, GRPO updates, and per-epoch tutor adaptation.
  • D.4 Compute resources: Baselines sharing the GRPO setup consume comparable compute per run.The passage names Base rubrics, WildChecklists, Policy-adaptive rubrics, and EVA as such baselines.
  • D.4 Compute resources: Evaluation uses the same H100 hardware through vLLM, with judge calls served by the gpt-5-mini API.The passage explicitly describes this evaluation serving setup.

E Baseline details · F Prompt templates

The paper details policy-agnostic and policy-adaptive baselines, including SFT distillation, Evol-Instruct, EVA, and rubric-only adaptation. It also specifies judge, base-rubric, constraint-generation, and policy-adaptive-rubric prompt templates with explicit scoring, comparison, grounding, and atomicity rules.

  • E Baseline details: SFT-distilled fine-tunes Qwen3-1.7B on one Qwen3-8B tutor response per WildChat prompt using cross-entropy.The teacher and student share sampling parameters, with additional hyperparameters summarized in Table 7.
  • E Baseline details: Evol-Instruct rewrites each WildChat seed prompt offline through four rounds, sampling among five mutation methods without consulting the policy.The operators include Add Constraints, Deepening, Concretizing, Increased Reasoning Steps, and one in-breadth operator; templates come from distilabel v1.3.2.
  • E Baseline details: EVA selects prompts by the scalar reward gap across current-policy responses, then completely rewrites selected prompts with Evol-Instruct templates.The rewriting step does not condition on sampled policy responses; the reimplementation uses GRPO, Qwen3-8B, and rubric-based RL infrastructure rather than the original setup.
  • E Baseline details: The policy-adaptive rubric baseline judges whether the current rubric discriminates among rollouts and appends criteria targeting observed weaknesses when it does not.This isolates rubric-level adaptation from the main method’s prompt-level adaptation.
  • F.1 Judge prompt: The judge template scores a response against one criterion on a 0–100 scale, permits any integer in that range, and returns -1 only when totally confused.It requires criterion-only evaluation, a justification of up to 100 words, and an explicit <score> output.
  • F.2.1 Base rubric generation: Base-rubric generation analyzes task scope and explicit constraints, then writes self-contained, atomic, non-redundant, and diverse criterion questions with 0–100 importance scores.The generated rubric is conditioned on a seed prompt and includes an analysis block plus criterion–importance pairs.
  • F.2.2 Constraint generation prompt: Constraint generation appends one atomic constraint only when two high-quality responses converge with no meaningful quality gradient.The prompt evaluates accuracy, completeness, depth, and execution quality, and requires any added constraint to be challenging, natural, additive, specific, grounded, and non-generic.
  • F.2.3 Policy-adaptive rubric generation: Policy-adaptive rubric generation adds a criterion only when the existing rubric misses concrete, instruction-relevant failures that are self-verifiable from the responses and instruction.Its criteria must be grounded in observed failures and target substantive quality dimensions rather than superficial formatting or word-choice features.

G Additional qualitative examples

The qualitative examples show that append-only constraints create rollout differences on tasks whose base rollouts are saturated or nearly uniform. These constraints expose distinct failure modes, including hallucination, formatting errors, omitted content, and narrative-frame collapse.

  • Constraint-induced rollout divergence: Adding an originator constraint differentiates saturated humanistic-technique rollouts, with one misattributing methods and another fabricating throughout.The unconstrained task has σbase = 0; Rollout A retains some correct anchors, while Rollout B compounds errors, including misattributing Suggestopedia to Ivan.
  • Constraint-induced rollout divergence: An explicit italicisation rule exposes formatting divergence: one rollout follows OSCOLA conventions, while another italicises every component and invents an [Author, Date] footnote.The original rollouts score consistently high on citation formatting before the constraint is added.
  • Constraint-induced rollout divergence: An added anecdote requirement competes with seven-item coverage, causing one rollout to omit the grandparents reflection despite near-uniformly strong base performance.The base rollouts perform near-uniformly well with µbase = 99.5; Rollout A preserves all seven actions, whereas Rollout B lets an extended dog-walking vignette crowd one out.
  • Constraint-induced rollout divergence: Requiring an actionable transformation mechanism destabilizes an otherwise ceiling-level alien narrative, with one rollout abandoning the required second-human-explainer frame.Base performance is µbase = 99.6 with σbase = 0.4; Rollout A preserves the frame through dialogue, while Rollout B shifts to first-person technical exposition.
Loading 2607.04412v2…