PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Train Your Advisor: Steering Black-Box LLMs with Advisor Models

Parth Asawa, Alan Zhu, Abigail O'Neill, Matei Zaharia, Alexandros G. Dimakis, Joseph E. Gonzalez

arXiv:2510.02453v3cs.LGcs.AIcs.CL

TL;DR

Frontier models are often black-box services, limiting customization to prompting when their weights cannot be modified. ADVISOR MODELS train a lightweight advisor to generate per-instance advice for a frozen model; they improve frontier-model performance across specialized reasoning, SWE, and personalization tasks, with reported gains transferring from low-cost students.

  • Problem

    Frontier models’ inaccessible weights make traditional fine-tuning impossible, while static prompts cannot adapt to diverse inputs.

    Source: §1 · p. 1

  • Method

    ADVISOR MODELS use a lightweight policy to generate instance-specific advice for a black-box student and train that advisor with reinforcement-learning rewards on the student’s final outputs.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across specialized reasoning, SWE, and personalization, advisors improve frontier models, including a 27.4% improvement in complex rule-following.

    Source: §1 · p. 1, §5 · p. 8

  • Takeaways & Limitations

    Advisors trained with low-cost students can transfer improvements to frontier systems, offering a practical, cost-effective way to customize black-box models.

    Source: §1 · p. 1, §5 · p. 9

    When a frontier model is already highly knowledgeable in a domain, the advisor may have nothing to learn that improves its capabilities; the paper observes no significant gain on MATH-500.

    Source: §4.4.4 · p. 8

Abstract

from arXiv · show

Frontier language models are deployed as black-box services, where model weights cannot be modified and customization is limited to prompting. We introduce Advisor Models, a method to train small open-weight models to generate dynamic, per-instance natural language advice that improves the capabilities of black-box frontier models. Advisor Models improve GPT-5.2's performance on RuleArena (Taxes) by 27.4%, reduce Gemini 3 Pro's steps taken in SWE agent tasks by 24.6%, and outperform static prompt optimizers in personalizing GPT-5 to user preferences (85-100% vs. 40-60%). We also find that advisors are transferable: an advisor trained with a low-cost student model still transfers improvements to a frontier model. Moreover, Advisor Models are robust: we observe no degradation on other benchmarks than the pipeline is trained on. Our method shows how to perform parametric optimization for black-box frontier models in a practical and cost-effective way.

1. Introduction

Frontier models are difficult to customize when their weights are inaccessible, while static prompts cannot adapt to diverse inputs. ADVISOR MODELS train a lightweight advisor to generate instance-specific guidance, improving frontier-model results across reasoning, SWE, and personalization tasks.

  • Frontier-model weights are inaccessible, making traditional fine-tuning impossible, while static prompts cannot adapt to diverse inputs.The paper highlights needs such as complex rule-following and personalization.Source: §1 · p. 1
  • ADVISOR MODELS train a lightweight advisor to generate dynamic, input-specific advice that steers a black-box model without access to its parameters or gradients.Task-specific rewards from the student model’s final outputs support reinforcement-learning optimization of the advisor.Source: §1 · p. 1
  • Advisors trained with low-cost students transfer gains to frontier models, and pipelines retain performance on benchmarks other than the one used for training.Source: §1 · pp. 1–2
  • On personalization, advisors raise reward from 40–60% to 85%+ while static prompt optimizers do not improve these tasks.Source: §1 · p. 2

2. Related Work

Static prompt optimization produces a fixed prompt for a task, while ADVISOR MODELS learn a lightweight policy that generates context-specific natural-language advice. The paper distinguishes this approach from trained prompt generators in its per-instance adaptation, training requirements, and advice format.

Static Prompt Optimization

  • Static prompt optimizers typically produce one prompt reused across task instances, limiting conditioning on tool output and user feedback in agentic settings.Source: §2 · p. 2
  • ADVISOR MODELS train a lightweight advisor to produce context-specific advice that adapts to individual inputs rather than reusing a fixed prompt.The advice is injected in-context into the black-box model.Source: §2 · p. 2

Trained Prompt Generators

  • Unlike other trained prompt generators, ADVISOR MODELS need no expert rewrites or expert labeler and produce per-instance natural-language advice without a constrained output format.The passage also contrasts them with methods limited to particular tasks or soft prompts.Source: §2 · p. 2

3. ADVISOR MODELS

ADVISOR MODELS place a trainable lightweight policy between an input and a black-box student, generating instance-specific natural-language advice. Reinforcement learning uses rewards on the student’s final output to train the advisor, which can also provide follow-up advice during multi-turn interaction.

  • A lightweight advisor generates input-conditioned natural-language guidance for a black-box student, and the learned policy can improve models beyond those used during training.Source: §3 · pp. 2–4

Design Variants

  • A three-step variant gives the advisor an initial student attempt, then uses its advice to guide a revised response; this focuses advice and simplifies the advisor’s role to verification.The paper reports better outcomes with this setup in complex tasks.Source: §3 · pp. 2–3
  • In multi-turn tasks, student actions and observations can be returned to the advisor for follow-up guidance, either regularly or when the student chooses to query.Several student actions may occur between advisor turns to save cost.Source: §3 · p. 3, Figure 2 · p. 3

Training

  • Reinforcement learning trains the advisor by sampling advice, evaluating the student’s output with a task-specific reward, and updating the advisor from observed rewards.The method uses Group Relative Policy Optimization (GRPO).Source: §3 · p. 3

Advisor Transferability

  • Trained advisors can be transferred to students beyond the model used during training, including frontier models, without further tuning.Training with a low-cost student is presented as a route to cost-efficient optimization.Source: §3 · p. 3, Figure 2 · p. 3
  • The advisor is updated while the black-box student remains unchanged, and the paper reports preserved capabilities on untrained tasks.Source: §3 · p. 4

4. Evaluation

The evaluation tests Advisor Models on specialized reasoning, personalization, and robustness, using low-cost students and transferring trained advisors to frontier models. Across these settings, trained advisors improve frontier performance and learn hidden preferences, while retaining unrelated capabilities and showing a boundary when the student already knows the domain.

4.1.1. RESULTS

  • On MTOB, the trained advisor raised GPT-4o mini chrF to 45.8, versus 33.1 for the student and 33.0 for GEPA; on RuleArena Taxes, GPT-4.1 mini reached 76.8% accuracy versus 64.8% alone.On SWE Agent Efficiency, training retained Gemini 2.5 Flash’s 61.2% resolve rate while reducing average steps from 19.1 to 14.4.Source: §4.1.1 · p. 5, Figure 3 · p. 6

4.2. ADVISOR MODELS lift the capabilities of frontier black-box models

  • GPT-5.2 RuleArena Taxes accuracy rose from 67.2% to 85.6%, Gemini 3 Pro used 5.4 fewer average SWE steps without affecting resolve rate, and GPT-5 MTOB chrF rose from 46.2 to 47.8 (p ≈0.03).The GPT-5 MTOB gain was statistically significant but smaller, with GPT-5 already near the 51.6 human expert baseline.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3.1. RESULTS

  • Advisor Models achieved 0.94 reward for review length, 99.6% accuracy for review level, and 0.948 reward for math solutions, while static optimizers offered marginal gains.The advisors also exceeded the in-context oracle; the personalization comparisons used GPT-4o mini students and Qwen2.5 7B Instruct advisors.Source: §4.3.1 · p. 6, Figure 5 · p. 7
  • Trained personalization advisors transferred to GPT-5 across all three domains, though Review Length and Math Solutions were less perfect than with GPT-4o mini.The authors attribute the weaker results to advice tailored especially to GPT-4o mini.Source: §4.3.1 · p. 7, Figure 6 · p. 7

4.4.1. ADVISOR PROMPT INITIALIZATION

  • Weak initialization reached the same 0.941 review-length performance as strong initialization, but required 20 epochs rather than 5.After 5 epochs, weak initialization was still learning at 0.749; the authors report that guidance can accelerate learning.Source: §4.4.1 · p. 7, Figure 7 · p. 7

4.4.2. CAPABILITY RETENTION

  • Pipelines trained on Math Solutions and Review Length performed essentially the same as the unadvised baseline on MATH-500, including where Math Solutions overlapped contextually with accuracy.The approach preserves student weights, unlike direct fine-tuning, which can degrade general capabilities.Source: §4.4.2 · p. 8, Figure 8 · p. 8

4.4.3. COMPARISON TO RL

  • On MTOB, direct RL scored 42.38 chrF and Advisor Models scored 38.98, versus a 31.17 baseline; direct RL remained better.The authors report that Advisor Models recovered most of the improvement over baseline as an alternative when student weights are unavailable.Source: §4.4.3 · p. 8, Figure 9 · p. 8

4.4.4. LIMITS OF ADVISOR MODELS

  • On MATH-500, Advisor Models provided no significant improvement because frontier models were already highly knowledgeable in this domain.The authors identify learnable information beyond what the student already knows as a requirement for effectiveness.Source: §4.4.4 · p. 8, Figure 10 · p. 8

5. Conclusion & Future Work

ADVISOR MODELS steer black-box foundation models with learned, instance-specific advice, supporting specialized reasoning and personalization beyond base capabilities. Experiments report gains in SWE efficiency and rule-following, hidden-preference learning, transfer, and robustness.

5. Conclusion & Future Work

  • ADVISOR MODELS steer black-box foundation models through learned dynamic advice, unlike static prompt optimization.Advisors generate instance-specific guidance for personalization and specialized reasoning beyond base-model capabilities.Source: §5 · p. 8
  • 24.6% faster SWE agents, 27.4% improvement in complex rule-following, and near-perfect hidden-preference learning are reported, while static methods fail on the latter.The method also transfers improvements from affordable advisors to frontier systems and preserves general capabilities.Source: §5 · pp. 8–9
  • The method provides a practical, cost-effective way to specialize black-box models while maintaining robustness.Source: §5 · p. 9

A. Training Details

The appendix reports training settings and discusses variants of the ADVISOR MODELS framework, including three-step advising, multi-turn advice, tool calls, and Monte Carlo reward estimation.

A. Training Details

B. ADVISOR MODELS Variants

B.1. 3-step Advisor System

  • In the three-step variant, the student attempts the task before the advisor evaluates that attempt and advises an updated response; this improved RuleArena learning over the two-step setup.The trade-off is twice as many student calls at inference, though training-set initial generations can be prepared offline and kept static.Source: §B.1 · p. 12

B.2. Multi-Turn Advising

  • Multi-turn advising lets the advisor provide guidance across a rollout; in SWE, the student takes five actions per step before their actions and outcomes are sent to the advisor.Longer intervals reduce advisor calls and can speed training, but make advice sparser.Source: §B.2 · pp. 12–13

B.2.1. ADVISOR AS TOOL CALL

  • In the dynamic-interval tool-call variant, students rarely requested advisor help because frontier models were loathe to ask without optimization pressure.Source: §B.2.1 · p. 13

B.3. Monte Carlo Reward Estimation

  • Monte Carlo reward estimation averages rewards across N student responses per advice to reduce variance, but multiplies student calls by N and was prohibitively expensive.Source: §B.3 · p. 13

C. Domain Specific Details

C.1. SWE Agent Training

SWE training uses filtered Tenacity-library problems and rewards resolved issues according to speed while assigning unresolved issues zero reward. The appendix also specifies a review-length reward based on matching the requested preference.

C.1. SWE Agent Training

  • Training problems came from the Tenacity library and were retained only if at least one of five unadvised Gemini 2.5 Flash trials resolved them; runs were capped at 40 student interactions.Unresolved issues receive zero reward, while resolved issues receive 0.5–1.0 according to resolution speed.Source: §C.1 · p. 13

C.2. Review Length

  • Review-length reward is 1 for an exact preference match and decays toward 0 as the length difference grows.Source: §C.2 · p. 13

D.1. Transfer Across Student Families

Advisors transfer across student families and providers, with reported gains on transferred tasks; a Llama advisor also improves MTOB performance over both baselines.

D.1. Transfer Across Student Families

  • 79.6% resolve rate versus the GPT-5 standalone baseline’s 78.4%, with 7.1 versus 8.1 average steps, followed transfer of a Gemini-trained SWE advisor to GPT-5.The step reduction was statistically significant (p ≈0.02).Source: §D.1 · p. 14, Figure 11 · p. 14
  • A GPT-4o mini-trained advisor transferred to Claude 4.5 Sonnet and raised Review Length average reward from 0.433 to 0.836.Source: §D.1 · p. 14
  • On MTOB, transfer to Claude 4.5 Haiku raised chrF from 40.2 to 45.1; transfer to a Qwen student also showed strong results across differing training distributions.Source: §D.1 · p. 14

D.2. Advisor Models w/ Llama

  • A Llama 3.1 8B Instruct advisor achieved 42.52 chrF on MTOB, above the GPT-4o mini baseline of 33.23 and untrained-pipeline baseline of 32.60.The result provides evidence that the method's utility is not restricted to the Qwen family.Source: §D.2 · p. 14, Figure 12 · p. 14

D.3. Scaling Personalization

In a 100-person Review Length task, Advisor Models improved personalization where a static oracle prompt did not. The experiment used fewer training rollouts per person than the original setup.

D.3. Scaling Personalization

  • The scaled task assigned 100 individuals preferences of 10–1000 words, with 10 training and 5 test examples per individual.This yielded 1,000 training and 500 test examples; the setup used a Qwen2.5 7B Instruct advisor for a GPT-4o mini student.Source: §D.3 · p. 15
  • Training used 4.5× fewer rollouts per individual than the original Review Length domain, and performance remained below that domain’s final performance.The authors note potential for further improvement as rollouts per individual reach parity.Source: §D.3 · p. 15
  • 0.85 reward with Advisor Models exceeded the student-alone score of 0.62 and the in-context oracle prompt score of 0.63.The oracle prompt was unable to improve performance, while Advisor Models provided meaningful improvement.Source: §D.3 · p. 15, Figure 13 · p. 15

D.4. Direct RL vs. ADVISOR MODELS

The MTOB comparison found direct RL learned faster early, while Advisor Models improved steadily; both were still improving at training’s end. Direct RL performed better, though Advisor Models provided a majority of its gains.

D.4. Direct RL vs. ADVISOR MODELS

  • The comparison trained a Qwen2.5 3B Instruct advisor to advise a Qwen2.5 7B Instruct student, versus directly training the 7B model on MTOB.Both methods followed the Appendix A training details.Source: §D.4 · p. 15
  • Direct RL performed better, but Advisor Models provided a majority of its gains on MTOB, with both methods’ scores still climbing at the end of training.Direct RL improved quickly early and then flattened; Advisor Models improved steadily throughout.Source: Figure 14 · p. 15, §D.4 · p. 15

D.5. Robustness of Frontier Models

An advisor trained on review length did not degrade frontier-model performance across several benchmarks. The appendix also presents generated advice examples for SWE Agent Efficiency tasks.

D.5. Robustness of Frontier Models

  • An advisor trained on an unrelated review-length task caused no degradation versus GPT-5 alone on MATH-500, MTOB, and LiveCodeBench Exec.The frontier-model experiments were intended to test robustness to irrelevant advice across domains.Source: Figure 15 · p. 16, §D.5 · p. 16

E.1. SWE Agent Efficiency

  • The SWE Agent Efficiency appendix illustrates advice for investigating unexpected retry-decorator statistics, including checking whether the statistics attribute is initialized.The examples are generated advice before and after training for the same evaluation problems.Source: §E.1 · p. 17

E.2. Review Length

Review Length examples contrast advice that misses a user’s stated length preference with advice that reflects it. The prompt examples show how advisor instructions and student prompts convey guidance.

E.2. Review Length

  • For a 10-word preference, one example recommends a 500–600-word review while another recommends 8–10 words, illustrating a shift toward matching the preference.The appendix presents examples of advice before and after training for the same evaluation problems.Source: §E.2 · p. 18

F. Prompt Examples

  • The strong advisor prompt explicitly asks for advice tailored to a person’s preferences and length, while the student prompt tells the model to follow advisor guidance.The weak advisor prompt asks for review advice without explicitly calling out preferences or length.Source: F. Prompt Examples · p. 19
Loading 2510.02453v3…