PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortized Off-Policy Evaluation for LLMs

Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee

arXiv:2610.10848v1cs.LG

TL;DR

Offline LLM evaluation must estimate candidate value from logs that may differ in both responses and reward definitions, while conventional methods require per-task fitting. PFN-OPE amortizes evaluation across a task prior built to represent both shifts, and achieves lower MSE than the best baselines on the reported benchmarks.

  • Problem

    Offline evaluation estimates a candidate LLM's reward from deployed-model logs, although candidate responses and the reward definition may differ, and conventional direct-method models must be refit for new datasets or reward definitions.

    Source: §1 · pp. 1–2

  • Method

    PFN-OPE is pretrained on contextual-bandit tasks built from a pool of LLM responses and multiple reward functions, then estimates target value from logged data and sampled target responses in one forward pass without per-task fitting.

    Source: §1 · p. 2

  • Results

    On HelpSteer2 and UltraFeedback, PFN-OPE has the lowest MSE under all five reward definitions, by 2.0 to 9.3 times over the best baselines, and the lowest MSE in 11 of 12 pairs with both target LLM and reward model unseen.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results position PFN-OPE as an approach for reusing one pretraining run across the stream of target policies produced by deployment.

    Source: §5 · p. 9

    Pretraining and evaluation rewards come from ArmoRM or GRM, making the setting semi-synthetic rather than based on human rubrics; human-rated logs were unavailable at the required scale.

    Source: E LIMITATIONS · p. 17

Abstract

from arXiv · show

Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams therefore evaluate candidates offline, on data produced by already-deployed models. This is off-policy evaluation (OPE), and it faces two distribution shifts: as a model is updated in post-training, its responses diverge from the logged ones (policy shift), and the reward definition under which it is judged changes with business requirements (reward shift). Classical OPE methods are ill-suited to this continual-deployment setting because they are defined per task and require fitting from scratch on every new logged dataset or reward definition. To address this, we propose PFN-OPE, a prior-data fitted network that amortizes OPE across a distribution of contextual-bandit tasks. We pretrain it once on tasks constructed from a pool of LLM responses scored by several reward functions, in which both shifts occur. At test-time it maps a logged dataset and one sampled target response per prompt to a value estimate in a single forward pass, with no per-task fitting. On HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies, PFN-OPE achieves 2.0 to 9.3 times lower error than the best baselines across all tested configurations in the reward-shifted settings.

1 INTRODUCTION

Offline LLM evaluation estimates candidate value from logs whose responses and reward definitions may differ from deployment. PFN-OPE amortizes this task across those shifts and outperforms baselines in the reported benchmarks.

1 INTRODUCTION

  • LLM OPE faces policy shift when candidate responses differ from logged responses and reward shift when the rubric changes with company requirements.The logs measure the deployed assistant, not the candidate.Source: §1 · p. 1
  • The direct method requires a new reward-model fit for each logged dataset or reward definition and must extrapolate to target responses whose rewards are unobserved.Both costs arise from fitting one model to one logged dataset at a time.Source: §1 · pp. 1–2
  • PFNs take a new dataset as input rather than requiring new training, and their prior determines which task shifts they can generalize across.For OPE, the proposed prior is intended to represent the reward and policy shifts encountered in LLM evaluation.Source: §1 · p. 2
  • PFN-OPE maps a logged dataset and one sampled target response per prompt to a value estimate in one forward pass, with no per-task fitting.Its input is a set of paired responses, one with a reward and one without, and its label is a functional of the whole set.Source: §1 · p. 2
  • Its task prior uses a response pool scored by several reward functions, with affine reward combinations and reward-tilted or routed policies representing the two shifts.Target values are computed from the pool, where policies are distributions over stored responses.Source: §1 · p. 2
  • On HelpSteer2 and UltraFeedback, PFN-OPE has the lowest MSE under all five reward definitions, 2.0 to 9.3 times over the best baselines, and in 11 of 12 pairs with both target LLM and reward model unseen.Four of the five reward definitions were absent from pretraining.Source: §1 · p. 2

2 BACKGROUND

The background frames LLM off-policy evaluation as contextual-bandit value estimation from behavior-policy logs, where target-response rewards are unobserved. It reviews conventional estimators and PFNs, whose task prior shapes their predictions.

2 BACKGROUND

  • OPE estimates a target policy's value, defined as its expected reward over contexts and actions sampled from that policy.The logged data are sampled by the behavior policy, which produced the observed responses and rewards.Source: §2 · pp. 2–3
  • Distribution shift leaves rewards for target-policy actions unobserved, and the problem becomes more severe as behavior and target policies diverge.In the LLM application, prompts are contexts, generated responses are actions, and human annotators assign rewards.Source: §2 · pp. 2–3
  • IPS, DM, and DR differ in their trade-offs: IPS is unbiased given the behavior policy but can have high variance, DM has low variance but shift-related bias, and DR combines them.DR is unbiased when either IPS or DM is unbiased and has lower variance than IPS.Source: §2 · p. 3
  • Conventional estimators do not amortize across datasets; IPS and DR need policy probabilities and per-pair weights, while DM must be retrained when the reward changes.The work focuses on improving DM for amortized OPE, motivated by IPS's high variance in LLM response spaces.Source: §2 · p. 3
  • A PFN is a transformer pretrained across datasets from a synthetic task prior; a new dataset and query are inputs for a prediction in one forward pass.In the standard PFN, tasks relate inputs to labels, and datasets are sampled from the task prior.Source: §2 · p. 3
  • PFN pretraining by negative log-likelihood yields a posterior predictive whose mean is the minimum-MSE estimate for tasks from the prior; that prior supplies the model's inductive bias.The prior determines which tasks the PFN infers from small datasets and how far it transfers.Source: §2 · p. 3

3 PFN-OPE: PRIOR-DATA FITTED NETWORK FOR OFF-POLICY EVALUATION

PFN-OPE turns LLM off-policy evaluation into in-context prediction, using a pretrained estimator and tasks synthesized from a reusable scored response pool to address policy and reward shifts.

3.1 OPE AS IN-CONTEXT PREDICTION

  • PFN-OPE treats OPE as in-context prediction from behavior-logged data and sampled target responses, predicting correction C = V(π) − mean logged reward because target-response rewards are unobserved.The correction centers the target on the logged mean reward.Source: §3.1 · pp. 3–4
  • At test time, one forward pass predicts correction quantiles and averages them toward a posterior-mean value estimate, processing both shifts through the observation without parameter updates.Averaging quantiles approximates the mean of the correction’s predictive distribution.Source: §3.1 · p. 4

3.2 CONSTRUCTING TASK PRIOR FROM A RESPONSE POOL

  • Pretraining reuses a finite response pool scored by reward functions, enabling task values to be computed from the pool without new LLM generation or reward-model evaluation.Tasks provide known target-policy values for supervision.Source: §3.1 · p. 4, §3.2 · p. 5
  • The task prior varies evaluation rewards through affine combinations of standardized stored rewards and varies policies through reward tilting and routing between target and behavior policies.Selection weights for policy steering are drawn independently of the weights defining the evaluation reward.Source: §3.2 · p. 5
  • At n = 512, PFN-OPE’s MSE is 2.0 to 9.3 times lower than the best baseline’s across six pairs of seen LLMs.Figure 1 compares the ArmoRM aggregate, the seen reward, with four rewards absent from pretraining.Source: Figure 1 · p. 6

3.3 PFN-OPE ARCHITECTURE

  • The transformer adapts PFN input representation, attention, and readout: target tokens attend to logged and target tokens, and a summary token predicts correction quantiles.Logged tokens attend only to logged tokens.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments test whether PFN-OPE handles reward and policy shifts without per-task fitting, across unseen rewards and targets, large corrections, limited data, and prior variants.

4.3 PFN-OPE GENERALIZES TO UNSEEN TARGET LLMS UNDER AN UNSEEN REWARD MODEL

  • PFN-OPE has the lowest MSE in 11 of 12 behavior–target pairs when both the target LLM and reward model are unseen.At n = 512 under GRM, Table 1 reports MSE (×10−2) and the larger half-width of the 95% interval.Source: Table 1 · p. 7, §4.3 · p. 8

4.1 COMMON EXPERIMENTAL SETUP

  • The setup evaluates held-out prompts with Qwen, Llama, and Gemma policies, five reward definitions, and 100 tasks per setting, comparing PFN-OPE with six estimators.PFN-OPE is one frozen checkpoint; direct-method baselines refit on each logged dataset, while DM-TabPFN and DM-TabICLv2 amortize reward regression.Source: §4.1 · p. 7

4.4 PFN-OPE IS ROBUST TO LARGE CORRECTION AND LESS LOGGED DATA

  • Above |Cstd| = 0.2, baseline MSE reaches 0.21 to 0.32 in the largest bin, while PFN-OPE’s remains between 0.04 and 0.07; PFN-OPE with 64 rows beats baselines with 2,048.Figure 2 compares MSE with correction size and logged rows; the advantage is 3.9 times for seen targets and 1.6 times for unseen targets.Source: Figure 2 · p. 8, §4.4 · pp. 8–9

4.2 PFN-OPE GENERALIZES TO FOUR UNSEEN REWARDS

  • PFN-OPE has the lowest MSE under all five rewards on both datasets, by 2.0–9.3 times over the best baseline, which sees the new reward through the same logged rewards and refits to it.The gain shrinks with distance from the prior but does not vanish: it is 3.7–9.3 times for rewards built from ArmoRM’s objectives, which the prior’s reward combinations span, and 2.0–4.1 times under GRM, which lies outside them.Source: §4.2 · p. 8

4.5 PRIOR ABLATION: THE POLICY AXIS OF THE PRIOR CARRIES THE GAIN

  • The full policy prior lowers MSE by 28% to 36% at every n, while reward combinations alone improve it by only 2% to 3% with pool policies.Figure 3 ablates reward and policy priors in a 2×2 design.Source: Figure 3 · p. 8, §4.5 · p. 9

4.6 PRIOR COVERAGE IS IMPORTANT IN PRETRAINING

  • The pool-only PFN has its highest error where its prior has no tasks, while the full prior covers that region and has low error there.Those evaluation tasks are Llama-3.2-1B→Llama-3.2-3B shifts, described as closest to a post-trained checkpoint of the logged model.Source: Figure 4 · p. 9, §4.6 · p. 9

5 CONCLUSION

The conclusion presents PFN-OPE as amortized LLM off-policy evaluation, pretrained once on response-pool tasks with known target values and used without per-task fitting.

5 CONCLUSION

  • On HelpSteer2 and UltraFeedback, PFN-OPE has 2.0 to 9.3 times lower MSE than the best baseline and the lowest MSE in 11 of 12 pairs with unseen targets and rewards.The authors frame reusing one pretraining run across shifting deployment policies as a step toward evaluation that keeps pace with changing post-training objectives and feedback signals.Source: §5 · p. 9
  • The authors report using generative AI tools for methodology and experiment feedback, implementation, dataset reformatting, qualitative analysis, and paper drafting and editing.They state that AI-assisted work was reviewed, with qualitative analyses independently verified by two authors.Source: AI USE STATEMENT · p. 10

C Experimental details

The experimental-details appendix lists sections on the response pool, evaluation protocol, and estimators.

C Experimental details

E Limitations 17

The supplied appendix passages cover PFN-related work, text-embedding OPE, in-context bandits, and pretraining algorithms.

A RELATED WORK

  • The related-work discussion places PFNs in prior-data fitted prediction and describes this work as extending them to LLM off-policy evaluation.It also lists applications to tabular prediction, Bayesian optimization, and causal effect estimation.Source: A RELATED WORK · p. 15
  • Embedding-based MIPS methods address text-action OPE, but still require per-dataset fitting.The passage also notes challenges extending kernel-based IPS to high-dimensional text embeddings.Source: A RELATED WORK · p. 15

B ALGORITHMS

  • Algorithm 2 gives every step of the procedure summarized in Algorithm 1: it samples a task, forms a policy, computes its value over the pool, forms observations and a correction label, then updates θ.Source: B ALGORITHMS · p. 15, E Limitations 17 · p. 15

C EXPERIMENTAL DETAILS

The appendix records the evaluation protocol and estimator settings used for Section 4.

C EXPERIMENTAL DETAILS

  • The appendix provides the evaluation protocol and estimator settings behind Section 4.It introduces the procedural and estimator details that follow.Source: C EXPERIMENTAL DETAILS · p. 15

C.1 POOL

Evaluation prompts are separated from pretraining prompts, and responses from each LLM are sampled and scored with two reward models.

C.1 POOL

  • Evaluation uses 1,024 randomly split prompts, with no evaluation prompt or associated behavior or target response appearing in pretraining.Every Section 4 evaluation task draws prompts only from the held-out evaluation set.Source: §C.1 · p. 15
  • For each prompt and LLM, the pool contains eight nucleus-sampled responses, each scored by ArmoRM and GRM.Sampling uses temperature 0.7 and top-p 0.9 for both seen and unseen LLMs.Source: §C.1 · p. 15

C.2 EVALUATION PROTOCOL

The protocol builds a response pool, samples contextual-bandit tasks and target policies from it, and trains on logged observations paired with a correction target.

C.2 EVALUATION PROTOCOL

  • Each pretraining example hides target-response rewards from its observation and pairs that observation with a correction target.The target is C = Vτ(π) − r̄, and the standardized correction divides C by logged-reward standard deviation floored at 10^-6.Source: §C.2 · pp. 15–16
  • The response pool stores sampled LLM responses, scores each with every reward function, standardizes each reward, and defines each LLM’s pool policy uniformly.The pool is built once before task sampling.Source: §C.2 · p. 16
  • Tasks draw reward-selection weights independently of the prompt distribution, then sample behavior and target LLMs and a policy mechanism.Pool, selected, and routed mechanisms are drawn with probabilities 0.4, 0.4, and 0.2, respectively.Source: §C.2 · p. 16
  • For each task, the procedure computes the target value over the pool, samples prompts and logged responses, then emits observations with the correction.The observation contains prompts, responses, and logged rewards only; it contains no task parameter.Source: §C.2 · p. 16
  • Within each setting, 100 candidate tasks are ranked by the absolute standardized correction, which varies across tasks even when the target value is fixed.The correction also includes sampling deviation in the logged-reward mean, so it is not simply the target-minus-behavior value gap.Source: §C.2 · p. 16

C.3 ESTIMATORS

The appendix specifies how estimators are compared and defines prior-coverage diagnostics based on a fixed linear-ridge reward probe.

C.3 ESTIMATORS

  • PFN-OPE estimates value as the logged mean plus the average of 99 predicted correction quantiles, while baselines use refitting, frozen weights, or density-ratio estimation.All estimators receive the same observations, and target rewards are used only to compute the target value.Source: §C.3 · p. 16

D PRIOR-COVERAGE DIAGNOSTICS

  • The diagnostics use a fixed linear-ridge probe on frozen embeddings, and neither Figure 4 axis is PFN-OPE prediction error.The probe uses prompt-grouped folds of logged data and computes expectations over stored responses.Source: D PRIOR-COVERAGE DIAGNOSTICS · pp. 16–17
  • Probe correction error is squared value-gap error scaled by logged-reward variance, with shared errors on logged and target responses canceling.It mixes finite-sample error, regularization, and linear-probe misspecification, and is not error on the target value itself.Source: D PRIOR-COVERAGE DIAGNOSTICS · p. 17
  • Value-relevant extrapolation measures ridge-regularized leverage of the mean feature shift, giving more weight to directions with little logged-feature variation.The shift is value-relevant because a linear probe’s value correction depends on it.Source: D PRIOR-COVERAGE DIAGNOSTICS · p. 17

E LIMITATIONS

The evaluation is bounded by its target policies, reward sources, and the breadth of the LLM pool. In particular, rewards are semi-synthetic, and evidence for transfer beyond the pool includes only one unseen family.

E LIMITATIONS

  • Evaluation targets are other LLMs, and tilted policies are used to represent tasks like those posed by post-trained checkpoints.The prior-coverage analysis reports that these tilted policies reach the kind of task a post-trained checkpoint would pose.Source: E LIMITATIONS · p. 17
  • Every reward comes from ArmoRM or GRM, so reward definitions are reward-model heads or combinations rather than human rubrics.Human-rated logs would test the described reward shift, but were unavailable at the scale required for the pool.Source: E LIMITATIONS · p. 17
  • The prior is built from three LLMs; transfer evidence beyond them includes one unseen family, which the authors describe as limited evidence.Expanding to more families and actual post-training checkpoints would require a one-time cost of generating and scoring their responses.Source: E LIMITATIONS · p. 17
Loading 2610.10848v1…