PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Policy Improvement from Observational Data

Jonas Schweisthal, Dennis Frauen, Stefan Feuerriegel, Emma Brunskill

arXiv:2610.23436v1cs.AIcs.LG

TL;DR

Policy improvement from observational data is difficult when actions are complex structured outputs and action-level overlap is limited. PolicyLift learns a policy over discrete action codes and lifts it through behavior-system sampling; the paper proves improvement transfers to the original action space and reports gains over baselines in LLM-based text generation.

  • Problem

    For complex structured actions, limited action-level overlap makes off-policy learning difficult, while the target policy must generate raw actions from logged interactions.

    Source: §1 · pp. 1–2

  • Method

    PolicyLift learns discrete action codes, optimizes a context-dependent policy over codes, and deploys it by sampling behavior-system actions matching the selected code.

    Source: §1 · p. 2

  • Results

    PolicyLift proves that code-level policy improvement guarantees improvement of the induced original-action policy and reports gains over baselines in LLM-based text generation.

    Source: §1 · p. 2, §5 · p. 5

  • Takeaways & Limitations

    Behavior-consistent lifting transfers true code-level value improvements to deployed structured-action policies when the behavior system is sampled until a matching code is found.

    Source: §1 · p. 2, §5 · p. 5

    Deployment requires sampling access to the behavior system and preserves its within-code action distribution rather than optimizing among actions with the same code.

    Source: §7 · p. 10

Abstract

from arXiv · show

We study how to improve decision policies from observational data to generate better actions when actions are complex, structured objects, such as text messages or images. This task is subject to two major challenges: (i) action-level overlap is often too weak for reliable off-policy evaluation and improvement, and (ii) generating new structured actions often requires expensive training or fine-tuning (e.g., of an LLM) and can be ill-defined due to information loss in action representations. To address both, we propose PolicyLift, a three-stage framework for generative policy improvement with structured actions from observational data. In Stage 1, PolicyLift learns a small set of discrete action codes that capture high-level strategies in the observed action space and are tailored to support policy improvement under limited overlap. In Stage 2, it learns a context-dependent policy over these action codes. At deployment time, Stage 3 "lifts" the learned code-level policy back to the original action space by generating actions from the historical behavior system in a code-conditioned way. We show theoretically that improving the policy over action codes guarantees improvement of the induced policy in the original action space. We demonstrate PolicyLift across experiments in synthetic and real-world text generation tasks with consistent policy improvements over the behavior policy.

1 INTRODUCTION

The paper studies policy improvement from logged interactions when actions are complex structured outputs that must be generated at deployment. It proposes PolicyLift to address weak action-level overlap and information loss in representations, combining code-level policy learning with behavior-consistent lifting.

1 INTRODUCTION

  • For complex actions, the behavior system may cover only a vanishingly small, locally varying subset, making action-level overlap implausible.Existing approaches constrain policies or use conservative objectives in the original action space, but do not resolve this coverage gap.Source: §1 · p. 2
  • A coarse action representation may make overlap more plausible but can lose information needed to map the learned decision back to a deployable raw action.If the representation does not mediate the action's effect on outcomes, decoding can create a deployment gap between optimized and delivered value.Source: §1 · p. 2
  • PolicyLift maps observed actions to discrete high-level strategy codes, learns a context-dependent policy over them, then lifts that policy using the behavior system as a generator.The framework targets generated structured actions learned from logged context-action-outcome triples.Source: §1 · pp. 1–2
  • PolicyLift proves that improving a policy over action codes guarantees improvement of the induced policy in the original action space, and reports empirical gains over baselines in LLM-based text generation.Its behavior-consistent lifting eliminates the deployment gap when the behavior system is sampled until an action with the target code is found.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses off-policy learning, overlap, or action representations, but the cited approaches do not cover generative policy learning for structured actions. PolicyLift learns overlap-oriented codes and deploys them through behavior-system sampling.

2 RELATED WORK

  • Standard off-policy methods and overlap-focused approaches largely target simple or low-dimensional actions, not structured-action policy learning.Methods modify or regularize policies toward better-supported interventions, or penalize uncertain value estimates.Source: §2 · p. 2
  • Embedding-based OPE/OPL methods reduce propensity challenges or select clusters, but generally assume fixed enumerable actions or sufficient overlap in a learned representation.Other learned action groups and embeddings target OPE rather than OPL.Source: §2 · p. 3
  • Kiyohara et al. (2025) study sentence-generation OPL with logged prompts, whereas PolicyLift learns from logged structured actions without prompts and uses only behavior-system sampling at deployment.The cited approach uses a frozen behavior generator to map discrete prompts to text.Source: §2 · p. 3

3 PROBLEM SETUP

The setup uses logged context, structured-action, and outcome data to learn a target policy that generates raw actions, with sampling access to the historical behavior policy at deployment. Identification relies on standard consistency, ignorability, and overlap assumptions.

3 PROBLEM SETUP

  • The data are i.i.d. logged context-action-outcome triples from a behavior policy, with no fixed finite action set assumed.Actions may be structured outputs such as LLM-generated text.Source: §3 · p. 3
  • The target policy must generate complex raw actions at deployment, and the learner has black-box sampling access to the behavior policy but not its weights, likelihoods, or training data.For context x, candidate actions can be drawn from the historical behavior system.Source: §3 · p. 3
  • Consistency and ignorability identify conditional expected outcomes, while value identification also requires the behavior policy to cover the target policy's support.Weak overlap inflates variance for importance-weighted estimators and extrapolation error for regression-based estimators.Source: §3 · p. 3

4 POLICYLIFT FRAMEWORK

PolicyLift separates code-level policy learning from action generation in three stages. It learns predictive, overlap-oriented action codes, optimizes a code policy under an overlap constraint, and deploys by sampling behavior-system actions that match the selected code.

4 POLICYLIFT FRAMEWORK

  • PolicyLift separates policy learning from action generation, with its learned code policy made generative by behavior-system sampling in the original action space.For each context, a code is selected first and a matching structured action is then returned.Source: §4 · pp. 3–4
  • Stage 1 learns discrete action codes that should predict outcomes and have code-conditional context distributions close to the marginal context distribution.Predictiveness determines the potential improvement over behavior, while balance makes code propensities less context-dependent and reduces extrapolation.Source: §4 · p. 4
  • The Stage 1 encoder is trained with a trade-off between attainable value and finite-sample reliability, using distributional distance and marginal code balance.The marginal-balance weight prevents globally rare or empty codes and avoids a trivial single-code minimizer of the overlap term.Source: §4 · p. 4
  • Stage 2 re-encodes logged actions and learns a context-dependent policy over codes, restricting choices to codes whose behavior propensity is at least τ.The Stage 1 balance regularizers are surrogates for this pointwise overlap condition; distributional balance alone does not guarantee a uniform propensity lower bound.Source: §4 · p. 4
  • Stage 2 maximizes a regression-adjusted value estimator over the feasible code-policy class.The framework uses this estimator because Stage 1 regularizes context regions where code-level regressors are evaluated.Source: §4 · pp. 4–5
  • Stage 3 selects a target code and returns the first behavior-system sample with that code, falling back to an ordinary behavior sample if the draw cap is reached.For supported codes, expected draws scale as 1/µC(c|x), and the threshold bounds this quantity.Source: §4 · p. 5

5 THEORETICAL ANALYSIS

The analysis justifies PolicyLift’s design through value-preserving behavior-consistent lifting, the role of code predictiveness in attainable improvement, and an error bound for regression-adjusted learning under limited overlap.

5.1 BEHAVIOR-CONSISTENT LIFTING

  • Under consistency and ignorability, behavior-consistent lifting preserves code-policy value, so code-level improvements transfer; propensity threshold τ bounds expected behavior-system draws by 1/τ.A draw budget Tmax additionally safeguards worst-case sampling cost.Source: §5.1 · p. 6

5.2 PREDICTIVENESS AND THE ATTAINABLE CODE-LEVEL VALUE

  • More predictive action codes increase attainable improvement: the signal is the reduction in outcome-prediction error beyond context alone, which underlies the oracle-value lower bound.A code constant given context has zero signal and permits no improvement over the behavior policy.Source: §5.2 · p. 6

5.3 OVERLAP AND RA ESTIMATION ERROR

  • Theorem 5.6 bounds regression-adjusted estimation error by code-conditional prediction error, an overlap penalty, and sample complexity under its stated shift and regularity assumptions.The theorem assumes bounded outcomes and predictions, an independent evaluation sample, a finite code-policy class, and the shift bound.Source: §5.3 · p. 7

6 NEURAL IMPLEMENTATION

The neural implementation learns balanced, predictive discrete codes, optimizes a propensity-constrained policy over them, then deploys it by sampling matching actions from the behavior system.

6 NEURAL IMPLEMENTATION

  • Stage 1 uses an action encoder with Gumbel–Softmax codes and a context embedding for outcome prediction, plus sliced-Wasserstein overlap and code-balance penalties.The overlap distance stops gradients through the context embedding; the balance penalty discourages collapsing actions into one code.Source: §6 · pp. 7–8
  • Stage 2 freezes the encoder, fine-tunes the outcome-prediction head, and selects the highest-scoring code among those with estimated propensity at least τ.The propensity proxy is fit by cross-entropy on logged context-code pairs.Source: §6 · p. 8
  • Stage 3 samples behavior-system actions until the selected code matches, with Tmax limiting worst-case deployment cost.Because Stage 2 restricts selection to supported codes, the implementation uses no additional overlap fallback.Source: §6 · p. 8

7 EXPERIMENTS

Experiments validate POLICYLIFT’s modeling choices and report improvements across controlled and text-generation settings. On OfflinePrompts, it improves policy value despite using less information than prompt-based comparators.

7 EXPERIMENTS

  • Evaluation progresses from a controlled synthetic DGP to headline generation and OfflinePrompts, while varying code-learning and policy-learning objectives to test modularity.OfflinePrompts uses a public reward simulator and does not require POLICYLIFT to observe the generating prompt.Source: §7 · p. 8
  • Synthetic diagnostics show that behavior-consistent lifting preserves code-policy value, outcome-predictive codes increase value, and overlap creates a predictiveness–estimation-error trade-off.Figure 2 varies deployment decoding, code predictiveness, and logging-policy concentration while fixing the other stages.Source: Figure 2 · p. 8, §7 · p. 9
  • 0.1031 to 0.2363: POLICYLIFT raises OfflinePrompts policy value by approximately 129%; its RA objective leads on synthetic data and OfflinePrompts and remains competitive on headlines.On OfflinePrompts, its mean exceeds prompt-based regression and is similar to DSO, which has higher variation across seeds.Source: §7 · p. 9
  • Deployment requires sampling access to the behavior system and preserves, rather than optimizes, the behavior distribution among actions sharing a code.The authors identify less conservative lifting and online evaluation as future work.Source: §7 · p. 10

A EXTENDED DISCUSSION AND LIMITATIONS

POLICYLIFT learns a policy in a supported code space and deploys it in the original structured-action space through behavior-consistent lifting. The discussion situates this approach among related methods and identifies its operational limits and a technical assumption.

A EXTENDED DISCUSSION AND LIMITATIONS

  • Theoretical and empirical results indicate that behavior-consistent lifting removes the deployment gap and transfers code-level improvement to deployed improvement over the behavior policy.The approach separates optimization in a supported code space from action generation by sampling from the behavior system.Source: A EXTENDED DISCUSSION AND LIMITATIONS · p. 15
  • The method requires sampling access to the behavior system at deployment and does not optimize among actions assigned the same code.It also depends on codes that balance outcome predictiveness with sufficient support; when no such representation exists, attainable improvement is limited.Source: A EXTENDED DISCUSSION AND LIMITATIONS · p. 15

B EXTENDED RELATED WORK

  • POLICYLIFT addresses a gap left by methods that handle only parts of generative policy improvement from observational data.Related approaches may require preference data, reward models, model weights, learned generators, fixed actions, or treatment-effect estimation rather than offline policy learning.Source: B EXTENDED RELATED WORK · pp. 16–17

C.1 A TECHNICAL ASSUMPTION ON WITHIN-CODE CONDITIONALS

  • The technical setup assumes regular conditional distributions for within-code behavior.The stated condition applies for contexts and codes with positive code probability under the behavior policy.Source: §C.1 · p. 18

C.2 DEPLOYMENT GAP AND CONTEXT-FREE DECODING

This appendix analyzes when code-level policy value aligns with deployed value, including failures from context-free decoding and support mismatch. It also establishes value-transfer results, describes acceptance sampling and its capped variant, and extends value equivalence to stochastic encoders.

C.2 DEPLOYMENT GAP AND CONTEXT-FREE DECODING

  • Perfect marginal within-code fit does not ensure zero deployment gap, because marginal decoding averages across contexts while deployment requires context-specific mixtures.The mismatch is expressed through differences between within-code outcome means under the marginal and context-specific behavior distributions.Source: §C.2 · p. 18
  • If the target decoder selects an action unsupported by behavior at a context-code pair, its expected outcome—and policy value when mismatch has positive measure—is not identified.The proof constructs observationally indistinguishable processes that assign different outcomes to the unobserved action.Source: §C.2 · p. 19

C.3 VALUE EQUIVALENCE AND IMPROVEMENT TRANSFER

  • With behavior-consistent lifting, improving the code policy over the behavior code policy transfers to at least as much value improvement in the deployed policy.The cited proof gives V(π↑) = VC(πC) and uses code-policy improvement over µC.Source: §C.3 · p. 19

C.4 ACCEPTANCE SAMPLING

  • Ideal acceptance sampling returns an action distributed as the behavior policy conditional on the target code, with expected draws bounded by 1/τ on ΠC,τ.The first matching code is accepted from repeated context-specific behavior draws.Source: §C.4 · pp. 19–20
  • Capping acceptance sampling at Tmax draws trades computation for approximation error, with failure probability decaying exponentially in Tmax.The capped procedure uses at most Tmax + 1 behavior draws, and its deployed-value deviation is O(pfail) when |Y| ≤ B.Source: §C.4 · p. 20

C.5 STOCHASTIC ENCODER EXTENSION

  • Value equivalence also holds for stochastic encoders under consistency, ignorability, and Assumption A3 applied to the joint (A, C).The result applies to code policies in ΠC,τ.Source: §C.5 · p. 20

C.6 PROOF OF THEOREM 5.5; C.7 PROOF OF THEOREM 5.6

  • The proof analysis uses total variance for code-conditioned outcomes and cross-fitting with Hoeffding bounds and a union bound for sample fluctuation.The proof fragments also separate a code-level prediction residual from the sample fluctuation term.Source: §C.6 · p. 21, §C.7 · p. 22

EVALUATION BENCHMARKS; D SYNTHETIC DIAGNOSTICS

  • The evaluation uses controlled benchmarks to isolate mechanisms: synthetic diagnostics vary within-code effects, predictive signal, and code-level overlap.The synthetic setup is intended to isolate behavior-consistent lifting, predictive action codes, and overlap-aware code learning.Source: EVALUATION BENCHMARKS · p. 23, D SYNTHETIC DIAGNOSTICS · p. 23

D.1 DATA-GENERATING PROCESS

This section specifies the synthetic and semi-synthetic benchmarks and evaluation setup for studying predictive signal, overlap, and deployment behavior, including a process for generating strategy-conditioned headlines from real article excerpts. It also describes code validation by rejection sampling before outcome assignment, with units discarded when retries fail.

D.1 DATA-GENERATING PROCESS

  • The synthetic simulator hides context type and code from learners, while action signals, within-code variation, and nuisance features test code recovery, deployment consistency, and geometric-clustering failure.The nuisance block does not affect outcomes but dominates Euclidean action geometry; deployment rules that alter the within-code action distribution incur a penalty.Source: §D.1 · pp. 23–24

D.2 ORACLE QUANTITIES

  • Oracle quantities include code values, support-constrained values, and behavior-system draws; learned-representation diagnostics estimate these by noise-free simulator Monte Carlo and table lookup.This avoids training additional oracle models.Source: §D.2 · p. 25

D.3 CODE REPRESENTATIONS

  • The representation comparison spans ground-truth and balanced oracle codes, random balanced codes, raw-action k-means, predictive codes, and predictive-plus-balanced codes.The final learned representation adds overlap and marginal-balance regularizers to prediction loss, testing whether it retains predictive signal while improving finite-sample reliability.Source: §D.3 · p. 25

D.4 DIAGNOSTIC PROTOCOLS

  • Three diagnostics isolate deployment instantiation, representation predictiveness, and weak-overlap reliability by varying decoder rules, code representations, support constraints, and overlap.They report deployment loss or policy value and improvement, with representation regret and prediction error for predictiveness, and RA error, unsupported-code rate, and expected draws for overlap.Source: §D.4 · p. 26

D.5 DEFAULT PARAMETERS

  • Table 5 is identified as the default-parameter table for the synthetic diagnostics; the cited passage provides the table caption but no parameter values.Source: Table 5 · p. 27

E SEMI-SYNTHETIC BENCHMARK CONSTRUCTION

  • The semi-synthetic benchmark pairs Upworthy article excerpts with LLM-generated headlines and simulator-assigned rewards; learners receive shared numerical representations, while natural-text OfflinePrompts is evaluated separately.The simulator provides known counterfactual rewards in a noisy structured-text setting with limited code-level overlap and recoverable reward-relevant structure; learner inputs exclude the context and action annotators.Source: E SEMI-SYNTHETIC BENCHMARK CONSTRUCTION · p. 27, §E.1 · p. 27, §E.2 · p. 27

E.2 SIMULATOR CONTEXT AND ACTION SPACES

  • The benchmark’s hidden context combines topic, tone, and specificity need, while headline codes combine strategy and tailoring level; reward ranks strategies by topic and tone and favors context-appropriate tailoring.The additive context baseline does not affect the optimal code, and reported policy values use a noise-free oracle reward table.Source: §E.2 · pp. 27–28, §E.3 · p. 28, §E.4 · pp. 28–29

E.5 CODE-LEVEL LOGGING POLICY

  • Logging factorizes strategy by topic and tone and tailoring by specificity need, with configurable preference for plausible but suboptimal strategies; κS = 0.60 and κR = 0.50 yield preferred-code probabilities of 0.70 and 0.75, respectively.Prompt-based construction uses a context annotator, headline generator, and action validator, with all prompts returning structured outputs.Source: §E.5 · pp. 29–30, §E.6 · p. 30

E.6 PROMPT-BASED CONSTRUCTION

  • The headline prompt requests exactly one concise Upworthy-style headline, specifying a strategy and generic or tailored treatment while requiring fidelity to the excerpt.Strategies are summary, curiosity, emotion, or direct; generic headlines avoid article-specific details, while tailored headlines include them.Source: §E.6 · p. 30

E.7 ACCEPTED ACTION GENERATOR

  • Generated headlines are accepted only when the validator confirms both the intended simulator code and faithfulness to the article excerpt.The accepted generator is implemented by rejection sampling before outcomes are assigned; validation does not use the outcome Y.Source: §E.7 · p. 31
  • If no valid headline is obtained within the fixed retry budget Lmax, the unit is discarded.An observed headline may serve only as a stylistic anchor and must not be copied verbatim.Source: §E.6 · p. 30, §E.7 · p. 31

E.8 LOGGED DATA AND ORACLE EVALUATION

The logged-data and oracle-evaluation protocol evaluates deployed headlines against an exact code-level benchmark. It validates realized simulator codes and evaluates only valid, faithful headlines.

E.8 LOGGED DATA AND ORACLE EVALUATION

  • Deployed text actions are checked for their realized simulator code before evaluation.Only valid and faithful deployed headlines are evaluated, using noise-free deployed reward.Source: §E.8 · p. 31
  • The main code-sufficient benchmark reduces the deployed-reward calculation to its code-level form.The passage identifies this reduction but does not provide its expression here.Source: §E.8 · p. 31
  • The benchmark provides an exact code-level oracle while learned policies return generated headlines.The oracle code policy is evaluated without outcome noise.Source: §E.8 · p. 31

F OFFLINEPROMPTS BENCHMARK

The OfflinePrompts evaluation uses real MovieLens-derived contexts, generated movie-related text, and a reward simulator trained on real user ratings. The synthetic diagnostics isolate deployment, predictiveness, and overlap effects, while a sensitivity analysis varies loss parameters β and γ.

F OFFLINEPROMPTS BENCHMARK

  • OfflinePrompts tests the method on raw generated text without injected simulator features, using a released personalized reward simulator trained on real user ratings.The task uses real MovieLens-derived user and movie contexts, and POLICYLIFT learns without observing the generating prompt.Source: F OFFLINEPROMPTS BENCHMARK · p. 32

G.1 SYNTHETIC DIAGNOSTICS

  • Figure 3 varies within-code reward effects, predictive signal, and logging-policy concentration across its three columns.Its top panel reports policy improvement; the bottom panels report deployment loss, code regret, and RA estimation error.Source: §G.1 · p. 33, Figure 3 · p. 33

G.1.1 DEPLOYMENT

  • Behavior-consistent lifting preserves code-level value, whereas alternative instantiation rules change within-code action distributions and incur deployment loss.The deployment diagnostic fixes ground-truth codes, the oracle code-level policy, and the true support rule while varying the instantiation rule.Source: §G.1.1 · p. 33, Table 10 · p. 33

G.1.2 PREDICTIVENESS

  • The predictiveness diagnostic fixes oracle Stage 2 optimization and behavior-consistent lifting, so differences across representations isolate information in the codes.Oracle true codes are the learner-inaccessible upper benchmark; oracle balanced codes are a support-aware proxy.Source: §G.1.2 · pp. 33–34, Table 11 · p. 34

G.1.3 OVERLAP

  • The overlap diagnostic varies code representations under increasing logging-policy concentration and distinguishes attainable policy improvement from value-estimation accuracy.All rows use a regression-adjusted objective, the τ support constraint, and behavior-consistent lifting; expected draws measure lifting cost.Source: §G.1.3 · p. 34, Table 12 · p. 34

G.2 SENSITIVITY TO LOSS PARAMETERS

  • Figure 4 varies β in the top row and γ in the bottom row, reporting policy improvement on the left and expected draws on the right.Higher policy improvement and lower expected draws are preferred.Source: §G.2 · p. 34, Figure 4 · p. 35

H IMPLEMENTATION DETAILS

  • The implementation-details passage identifies code for the synthetic diagnostics and directs readers to implementation choices for the synthetic and semi-synthetic runs.It does not specify those choices in the supplied text.Source: H IMPLEMENTATION DETAILS · p. 35

H.1 IMPLEMENTATION OF THE EXPERIMENTAL COMPARISONS

The experiments implement PolicyLift as a three-stage process: learn action codes, fit code-level outcome and propensity models and a policy, then deploy by lifting the selected code through the behavior policy. They compare code and objective alternatives across synthetic and text benchmarks, with benchmark-specific configurations and shared generators and evaluators.

Code substitutions

  • Code comparisons use K = 8; random codes are outcome-independent, k-means codes cluster action features, and predictive-only codes set β = γ = 0 while retaining the learned architecture and prediction loss.Code substitutions retain the RA objective and evaluation protocol; objective substitutions share the proposed encoder within each seed.Source: §H.1 · p. 35

Stage 2 objectives

  • Code-IPS uses inverse propensity weighting on observed codes, Code-DR adds a code-level reward predictor and residual correction, and Kernel-IS uses soft memberships based on distances to code centroids.Kernel-IS estimates conditional means with a soft-label propensity model and uses a median-distance bandwidth heuristic.Source: §H.1 · p. 35

Algorithm

  • PolicyLift learns the code encoder and predictor, freezes the encoder to assign hard codes, fits code-level outcome and propensity models with a policy, then deploys a code-conditioned behavior-policy sampler.The implementation minimizes prediction, overlap, and code-balance losses before freezing the Stage-1 model; deployment falls back to the behavior policy after an acceptance cap.Source: §H.1 · p. 36

Evaluation and compute

  • Within each dataset, baselines share the candidate generator and deployment evaluator, and experimental runs use commodity CPUs without GPU acceleration, typically finishing in under one hour.Semi-synthetic dataset construction also uses OpenAI API calls for headline generation and validation, completed within approximately one hour.Source: §H.1 · p. 37

Benchmark configurations

  • Synthetic runs use 5,000 logged observations and 10,000 noise-free evaluation samples, while headline runs retain MiniLM features and evaluate across five random seeds.The synthetic configuration uses αpred = 1, αdep = 0, κov = 0.6; the headline configuration uses (β, γ) = (10−4, 0.2).Source: Table 13 · p. 36, §H.1 · p. 37, Table 14 · p. 37

Prompt-based reference methods

  • On OfflinePrompts, prompt-based regression selects among 100 catalog prompts, while a DSO-style policy learns a context-dependent distribution over that catalog using kernel importance weighting.Both baselines share generated sentences, logged rewards, test contexts, and the reward simulator with PolicyLift, but additionally access prompt information unavailable to PolicyLift.Source: §H.1 · p. 37

Related papers

Loading 2610.23436v1…