Source-linked AI summary
Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning
Minh-Ha Nguyen, Cathy Shyr
TL;DR
The paper asks whether fixed-weight pretrained executors can support reusable expert-guided policies for rare-disease diagnosis. PIHF revises a versioned policy-and-tool artifact through critic and expert review, increasing Recall@1 from 26.5% to 59.3% across 1,243 benchmark cases.
Problem
The paper asks whether fixed-weight pretrained executors can support expert-guided, reusable policy development for rare-disease diagnosis.
Method
PIHF iteratively revises a versioned policy-and-tool artifact through critic review of complete-panel trajectories and expert-controlled candidate admission or rollback.
Results
Recall@1 increased from 26.5% to 59.3% across 1,243 public rare-disease benchmark cases in the frontier proprietary executor.
Takeaways & Limitations
PIHF converted scarce expert reasoning into an inspectable, revisable execution policy reusable across model backbones while weights remained fixed.
Takeaways & Limitations
Current transfer evidence supports the composite policy-and-tool artifact, not stronger attribution to the optimized reasoning process.
Abstract
from arXiv · showhide
Generative pretraining established reusable task representations; later work on language-based task conditioning and in-context learning showed that a fixed model could adapt its behavior from instructions and demonstrations. Policy Iteration with Human Feedback (PIHF) builds on this development and the recurrent evaluate-and-improve structure of generalized policy iteration. PIHF uses a pretrained language model as its execution substrate and moves persistent revision to a versioned natural-language policy and tool set. A language-model critic and clinical expert review complete-panel reasoning and tool-use trajectories to localize recurrent failures and form candidate revisions; the expert may reinterpret the evidence and retains authority over admission and rollback, while Recall@1 and Recall@5 validate outcomes after candidate execution. Across cumulative ablations and ultra-rare-disease benchmarks, a PIHF-derived policy improved Recall@1 in one proprietary executor and three open-weight executors spanning 3 to 49 billion active parameters. Gains were 32.7 percentage points for GPT-5.4 and 31.1 points for Qwen3.6-35B, a difference of 1.7 points. These results support the feasibility of using pretrained language models as fixed-weight execution substrates for expert-guided policy development in rare-disease diagnosis.
1 Reinforcement-learning principles
Reinforcement learning improves a policy by maximizing evaluated response quality while regularizing departure from an anchor policy. In language-model feedback learning, this objective supports both an exponential-tilt interpretation and score-function updates that implement evaluate-and-improve.
- Objective: The objective combines expected evaluator score with a KL penalty against anchor policy πanc, weighted by β > 0.The evaluator supplies the reward-seeking term, while β controls pressure to remain near the anchor.
- Objective: Within one policy-update phase, prompts, evaluator, anchor policy, and β remain fixed, while learning changes θ and πθ(· | x).Across tasks or training phases, changes in prompt distribution D or evaluator R define a new environment and evaluation target.
- Closed-form optimizer: The optimizer reweights the anchor response distribution by exp{R(x, y)/β}, then normalizes the reweighted masses into a conditional distribution.Higher-reward responses receive more probability relative to other responses for the same prompt.
- Policy update: A score-function update converts baseline-adjusted reward and anchor pressure into changes in response probability, with the baseline preserving expected gradients while reducing variance.This update combines policy evaluation with improvement in a recurring evaluate-and-improve pattern.
2 Structural bridge to PIHF
PIHF transfers generalized policy iteration from weight updates to iterative revision of an external natural-language policy and tool artifact. It separates failure-localizing process feedback from terminal diagnostic validation while preserving expert control over which candidate persists.
- Structural correspondence: Generalized policy iteration alternates policy evaluation and improvement, whereas PIHF implements those functions through evidence from complete-panel trajectories and targeted artifact revisions.Candidate freezing, complete-panel comparison, and expert admission determine which artifact advances to the next iteration.
- Representation-level correspondence: PIHF represents iteration t with an external artifact A_t = (P_t, T_t), combining a natural-language policy P_t with available tools T_t.This contrasts with weight-space reinforcement learning, where θ denotes learned model parameters.
- Behavior-level correspondence: A frozen executor M applying A_t to clinical case z induces a distribution over complete trajectories τ, paralleling π_θ(y | x) over responses to prompts.The correspondence maps prompt-response behavior to case-instantiated prompts and complete tool-use trajectories.
- Feedback and validation: Critic and expert review localizes recurrent failures to policy stages or tool behavior, providing credit assignment for discrete artifact revision.This process feedback is distinct from terminal outcome measurement in weight-space reinforcement learning.
- Feedback and validation: Recall@1 and Recall@5 validate terminal diagnostic outcomes after a frozen candidate executes the complete panel.The broader iteration includes proposal formation, candidate freezing, outcome validation, and expert admission.
3 In-context policy definition
The in-context policy representation fixes an executor model while versioning a natural-language policy and available tool set. It instantiates case-specific prompts and produces reviewable, multi-call execution trajectories whose conformance and clinical adequacy can be assessed.
- Policy representation: At iteration t, the mutable artifact comprises a versioned natural-language policy P_t and executor tool set T_t.The executor model M is fixed while the policy and tools change across iterations.
- Execution trajectories: For each clinical case z, the policy instantiates the executor prompt and induces a distribution over complete rollout trajectories.A trajectory records model outputs, tool calls, and tool results, ending in a ranked differential.
- Execution trajectories: A complete trajectory may span several model calls connected by intervening tool calls and results.The trajectory is therefore an end-to-end record of model–tool execution rather than a single model response.
- Trajectory review: Review compares trajectories with the policy for conformance and evaluates whether the policy remains clinically sound and useful for adequacy.Critique localizes the earliest departure and attributes it to a policy rule, tool use, or returned evidence; adequacy incorporates clinical evidence, panel outcomes, critic analysis, and expert judgment.
- RL parallel: In the reinforcement-learning analogy, the artifact A_t is the policy representation and π_M,A_t(τ | z) is its induced behavior.This framing treats the in-context artifact as the policy while the fixed executor supplies the behavior.
4 Pretrained representations and in-context adaptation
Prior work established that pretrained language models can reuse task representations and condition behavior through language under fixed parameters. PIHF builds on this foundation by using a frozen pretrained executor while persistent task behavior resides in a versioned, expert-governed policy and tool artifact.
- Generative pretraining produced reusable representations across language tasks, while task-aware sequences enabled supervised adaptation of a shared pretrained Transformer.
- GPT-2 showed that language can encode the task, input, and output in one sequence to condition model behavior under fixed parameters.
- In-Context Policy Iteration used a frozen model, experience-buffer trajectories, sampled prompt context, and model-generated rollouts for greedy action selection in six small control tasks.
- PIHF treats the pretrained executor as a fixed-weight substrate and makes the persistent object a versioned artifact A_t = (P_t, T_t), combining a textual policy with available tools.
- During evaluation, the executor and current artifact remain fixed while model calls and tool interactions form complete trajectories; persistent improvement uses panel evaluation and expert admission.
5 PIHF operator and composed runs
PIHF takes a frozen executor, initial policy and tools, complete development panel, comparison protocol, and stopping condition, then returns the final admitted artifact. The study composed two invocations, with UDN development warm-started from the frozen LIRICAL artifact and policy-specific held-out exclusions defining evaluation claims.
- PIHF operator: PIHF operates on a frozen executor, initial policy, initial tools, and complete development panel under a fixed comparison protocol and stopping condition.The invocation outputs the final admitted policy-and-tool artifact (P⋆, T⋆).
- Composed runs: The study composed two invocations: public-policy development began from the initial artifact on a LIRICAL development panel, while UDN development warm-started from the frozen LIRICAL artifact.The warm start initializes a new improvement run from an admitted policy representation.
- Evaluation separation: Claims about each policy-and-tool artifact exclude its corresponding development panel from held-out evaluation.This exclusion is policy-specific: LIRICAL claims exclude Ddev_L, and UDN claims exclude Ddev_U.
- Evaluation separation: The warm start evaluates procedural reuse and artifact adaptation in discrete evaluations, while unchanged cross-cohort transfer is treated as a separate estimand.Within each invocation, the method evaluates complete-panel reasoning, forms proposals, compares candidates, admits revisions, and applies stopping operations.
6 Development-panel evaluation
Development-panel evaluation freezes panel composition and inference settings, then measures whether reference diagnoses appear in executors’ ranked differentials. Recall@1 and Recall@5 serve as post-revision validation checks after complete-panel execution.
- Evaluation setup: Evaluation uses a frozen development panel with fixed panel composition and inference settings for a declared run.During executor rollouts, reference outcomes and derived signals remain hidden from the executor.
- Evaluation metrics: Recall@k is the fraction of development cases whose reference diagnosis appears within the executor’s first k ranked diagnoses.The metric is computed empirically over the finite panel.
- Evaluation metrics: Recall@1 measures correct first-rank placement, whereas Recall@5 measures inclusion in the five-item differential.Because Top1 is a subset of Top5, Recall@1 cannot exceed Recall@5.
- Post-revision validation: The empirical mean of the case-level terminal outcome signal is computed after a frozen candidate executes the complete panel.This mean serves as a post-revision validation check of diagnostic accuracy.
7 Critic and expert proposal formation
PIHF forms candidate revisions through a critic’s analysis of sealed complete-panel trajectories, followed by expert review and authorization. Outcomes remain hidden during execution and are used only for outer-loop failure selection and terminal validation before admission or rollback.
- Critic and expert proposal formation: After outputs are sealed, an LLM critic identifies recurrent reasoning and tool-use failures and proposes policy, tool, and evaluation changes.The critic also suggests measures, constraints, and protected-win checks.
- Critic and expert proposal formation: The expert may accept, reject, revise, or replace the critic’s interpretation and can originate or revise the associated hypotheses, checks, and artifact changes.The expert’s review uses panel evidence and the incumbent artifact.
- Critic and expert proposal formation: The expert-authorized proposal advances to candidate testing, while candidate admission and rollback remain expert decisions after complete-panel evaluation.An empty proposal ends that proposal path.
- Outcome information: Outcomes stay hidden during executor reasoning, guide outer-loop failure selection after outputs are sealed, and validate frozen candidates using Recall@1 and Recall@5.Validation occurs after complete-panel execution and tests whether revisions preserve or improve diagnostic accuracy before admission.
8 Candidate freeze and expert admission
PIHF freezes each expert-proposed policy edit into a candidate and evaluates it against the incumbent on the complete panel under identical settings. Admission requires preserving both recall endpoints and satisfying the expert’s qualitative review, with failed candidates rejected or revised.
- Candidate freeze: Each authorized edit is applied to the incumbent and frozen as a candidate with fixed policy, tool versions, panel, inference settings, and output schema.The incumbent and candidate are then evaluated on the complete panel under the same protocol.
- Expert admission: The recall-preservation indicator equals one only when the candidate preserves both Recall@1 and Recall@5; regression at either endpoint sets it to zero.This comparison is made relative to the incumbent.
- Expert admission: The expert’s binary qualitative verdict covers clinical soundness, generalizable panel-supported revision, and compliance with information-boundary rules.The expert reviews candidate trajectories, complete-panel evidence, and protected prior wins.
- Expert admission: A candidate becomes the next incumbent only when both recall-preservation and qualitative expert indicators equal one.When either indicator is zero, the incumbent remains in force and the proposal is rejected or revised into a new candidate.
- Rollback: Rollback restores a previously admitted safe checkpoint as the incumbent, after which subsequent proposals are formed from that checkpoint.Every revised candidate is frozen and evaluated through the same two indicators.
9 Process-guided policy improvement and outcome validation
PIHF combines stage-localized process feedback with terminal outcome validation to improve a versioned external policy. Expert admission determines the next incumbent, with development stopping after performance plateaus for 10 or more completed iterations in the reported study.
- Process-guided policy improvement: PIHF’s critic and expert inspect sealed reasoning and tool-use trajectories, localize recurrent failures to stages or tools, interpret causes, and form targeted policy revisions.The expert may replace the critic’s interpretation or add proposal elements from panel evidence, assigning credit to the external policy component that should change.
- Outcome validation: Recall@1 measures correct first-rank placement, while Recall@5 measures inclusion within the five-item differential after complete-panel execution.These terminal metrics validate whether a process-guided revision preserves or improves diagnostic accuracy.
- Stopping and evaluation: 10 or more completed iterations marked the stopping point in the reported development study after diagnostic performance plateaued under finite compute.Each new invocation must declare its stopping condition before candidate evaluation, and the last admitted artifact is frozen for development-excluded evaluation.
- Policy iteration: Expert admission commits the resulting external policy as the next incumbent after process feedback and terminal outcome validation.This creates an RL-parallel evaluate-and-improve loop over the policy rather than only the model’s execution behavior.
10 Portability, invariance, and attribution · 11 Summary
Section 10 distinguishes portability from invariance and identifies the composite policy-and-tool artifact as the currently supported object of transfer. Section 11 summarizes PIHF’s expert-governed revision process and reports improved rare-disease diagnosis with fixed model weights across executors.
- 10 Portability, invariance, and attribution: Within-backbone benefit is defined as Δ_m,k = Recall_k(M_m, A⋆; D_transfer) − Recall_k(M_m, A∅; D_transfer) for k ∈ {1, 5}.The comparison holds the transfer dataset and endpoint fixed while contrasting the frozen composite artifact with the matched no-artifact condition.
- 10 Portability, invariance, and attribution: Positive Δ_m,k across declared backbones supports portability of the frozen artifact.Portability is assessed through the sign of the within-backbone benefit rather than through absolute performance alone.
- 10 Portability, invariance, and attribution: Low dispersion supports similarity of benefit across backbones, making invariance a separate claim requiring separate uncertainty analysis.Both portability and invariance allow backbone-specific trajectories and absolute performance.
- 10 Portability, invariance, and attribution: Holding A⋆ fixed while changing M_m evaluates how the same policy representation induces behavior under different executors.This is the reinforcement-learning parallel used to frame cross-backbone transfer.
- 10 Portability, invariance, and attribution: The identified transfer object is the composite policy-and-tool artifact, because policy content, tool access, and coordination differ between A⋆ and A∅.Stronger attribution to the optimized reasoning process requires a content-only versus with-critique ablation separating expert-written clinical content from iteratively revised process structure.
- 11 Summary: PIHF persistently improves a versioned in-context policy-and-tool artifact executed by a frozen pretrained model.A language-model critic reviews complete-panel reasoning and tool-use trajectories, while the expert may accept, revise, replace, or originate changes and controls candidate formation, admission, and rollback.
- 11 Summary: 32.8 percentage points: Recall@1 increased from 26.5% to 59.3% across 1,243 public rare-disease benchmark cases in the frontier proprietary executor.The PIHF-derived policy was developed from 50 cases, transferred across proprietary and open-weight executors, and reused while model weights remained fixed.