Source-linked AI summary
A Generalized Acquisition Function for Preference-based Reward Learning
Evan Ellis, Gaurav R. Ghosal, Stuart J. Russell, Anca Dragan, Erdem Bıyık
TL;DR
Reward learning from human feedback can be costly, and parameter-focused query selection may miss distinctions that matter for downstream behavior. The paper proposes a tractable way to query for reward alignment under behaviorally meaningful metrics. Across three task settings, its method outperforms the state-of-the-art information-gain method, including on reward transfer.
Problem
Method
The framework selects queries to learn rewards up to an equivalence class defined by a user-chosen alignment metric, with a tractable approximation.
Source: §I · p. 1Results
The method outperforms state-of-the-art performance by up to 85% in learning rewards that transfer well to new domains, across synthetic, assistive-robotics, and NLP tasks.
Source: §I · p. 1Takeaways & Limitations
Alignment metrics can encode functional reward characteristics and deployment-domain distributions, allowing query selection to target behavior-relevant distinctions.
Source: §I · p. 1, §VI · p. 4
Abstract
from arXiv · showhide
Preference-based reward learning is a popular technique for teaching robots and autonomous systems how a human user wants them to perform a task. Previous works have shown that actively synthesizing preference queries to maximize information gain about the reward function parameters improves data efficiency. The information gain criterion focuses on precisely identifying all parameters of the reward function. This can potentially be wasteful as many parameters may result in the same reward, and many rewards may result in the same behavior in the downstream tasks. Instead, we show that it is possible to optimize for learning the reward function up to a behavioral equivalence class, such as inducing the same ranking over behaviors, distribution over choices, or other related definitions of what makes two rewards similar. We introduce a tractable framework that can capture such definitions of similarity. Our experiments in a synthetic environment, an assistive robotics environment with domain transfer, and a natural language processing problem with real datasets demonstrate the superior performance of our querying method over the state-of-the-art information gain method.
I. INTRODUCTION
Reward learning from human feedback can be costly, while parameter-focused active querying may target distinctions that do not matter to downstream behavior. This paper centers query selection on reward alignment and behaviorally meaningful equivalence classes, building on work on feedback, query informativeness, and reward metrics.
I. INTRODUCTION
- Parameter-focused objectives can waste costly human queries because distinct reward parameters may induce equivalent downstream behavior, while reward-function errors can lead to undesirable consequences.Prior objectives may ask queries with little benefit to the downstream application.Source: §I · p. 1
- The proposed active-learning objective learns the true reward up to an equivalence class of statistics over induced behavior, such as trajectory rankings, and skips irrelevant questions.The framework supports alignment metrics that capture functional characteristics and can encode deployment-domain distributions, with a tractable approximation under mild assumptions.Source: §I · p. 1
- The method outperforms state-of-the-art performance by up to 85% in learning rewards that transfer well to new domains, across synthetic, assistive-robotics, and NLP tasks, for linear and nonlinear rewards.The experiments use log-likelihood, EPIC distance, and trajectory rankings as reward-alignment measures.Source: §I · p. 1
II. RELATED WORK
- Related work studies query informativeness through volume removed, mutual information, and regret, and reward metrics including EPIC and reward projection [19].EPIC respects equivalence between rewards inducing the same optimal policy; reward projection measures similarity in likelihood on expert demonstrations.Source: §II · p. 2
III. PROBLEM FORMULATION
The problem formulation models reward learning in a finite-horizon MDP with human feedback and an adaptive querying policy. Its objective is to learn a reward aligned with the true reward under a chosen metric using a small number of queries.
III. PROBLEM FORMULATION
- The system is modeled as a finite-horizon MDP with state and action spaces, an initial-state distribution, transition dynamics, and a set of dynamically feasible trajectories.The reward is defined over states and actions, with cumulative reward evaluated over trajectories.Source: §III · p. 2
- The formulation uses pairwise trajectory comparisons and a response model conditioned on the query and reward, while allowing other feedback types when they have an appropriate response model.The paper names rankings, ordinal feedback,, and corrections, as examples.Source: §III · p. 2
- Reward functions are learned Bayesianly from query-response data, assuming responses are conditionally independent given the reward; the prior can incorporate domain knowledge or demonstrations.The experiments use an uninformative prior.Source: §III · p. 2
- The objective is to choose an adaptive policy that uses few queries to make the learned reward align with the true reward under a specified metric.Each query is selected based on the previous query-response history.Source: §III · p. 2
IV. EXISTING SOLUTIONS
The existing mutual-information policy greedily reduces uncertainty about reward parameters. The paper describes cases where that target diverges from downstream behavior, motivating queries that account for the intended alignment objective and trajectory distribution.
IV. EXISTING SOLUTIONS
- πMI selects each query to greedily maximize mutual information between the response and reward parameters, equivalent to minimizing posterior entropy.The method had previously outperformed volume-removal methods on metrics including cosine similarity to w* and validation-set log-likelihood,.Source: §IV · pp. 2–3
V. FAILURE CASES OF MUTUAL INFORMATION BASED QUERYING POLICY
- When the goal is finding an optimal trajectory, parameter entropy can waste queries because many reward parameters induce the same optimum; maximum-regret querying can help but requires optimizing trajectories for arbitrary rewards.The maximum-regret policy is reported to find optimal trajectories better and faster than mutual-information querying.Source: §V · p. 3
- For comparing arbitrary trajectories, πMI weights parameters equally even when some affect only a small subset of comparisons.Such parameters matter less to most trajectory comparisons, according to the passage.Source: §V · p. 3
- In domain transfer, πMI is agnostic to the trajectory distribution shift and can therefore produce suboptimal queries.The paper gives simulation-to-robot deployment and comparing paragraphs rather than scientific articles as motivating examples.Source: §V · p. 3
- These failure cases motivate a querying approach that lets designers specify the true alignment objective.The formulation targets alignment under a chosen metric rather than parameter uncertainty alone.Source: §V · p. 3
VI. OUR APPROACH
The approach selects queries to learn the reward only up to equivalence under an alignment statistic f, rather than identify exact reward parameters. It makes this goal tractable with a greedy proxy and illustrates f with response likelihood, EPIC distance, and trajectory-ranking metrics.
VI. OUR APPROACH
- The proxy rests on the assumption that the best-aligned reward's response predictions approximate those of the true reward, excluding adversarial alignment metrics.The paper describes this as a mild assumption that f is an alignment metric to be maximized.Source: §VI · p. 3
- The proxy identifies rewards only up to equivalence under f: once two rewards induce the same ranking, f becomes zero and further queries are no longer helpful.The paper frames this as learning a reward up to an equivalence class of rewards that produce the same behavior.Source: §VI · pp. 3–4
- Optimizing the adaptive policy over K queries with stochastic user responses is intractable, so the method uses a greedy objective that maximizes expected posterior alignment under f.The simplified objective requires evaluating f(Rw, Rw′) and response probabilities, with expectations computed by sampling from the prior once for each query.Source: §VI · p. 4
- The framework instantiates f with log-likelihood response agreement, EPIC distance, and ρ-projection, which compares reward-induced trajectory rankings.The ρ-projection comparison is defined over a set of trajectories.Source: §VI · p. 4
- The query set for πLL and trajectory set for πρ should represent the deployment environment, including real-robot trajectories when preferences are collected in simulation.The paper gives simulator feedback followed by real-robot deployment as an example for selecting the query set.Source: §VI · p. 4
VII. ANALYSIS
The analysis recovers mutual-information querying as a special case of the framework and gives a conditional near-optimality guarantee.
VII. ANALYSIS
- When expected alignment is adaptive monotone and adaptive submodular, the greedy solution is near-optimal.The analysis states an expected guarantee of at least (1 − 1… of OPTK after K queries, but the supplied passage truncates its bound.Source: §VII · pp. 4–5
- The mutual-information policy πMI is a special case of the framework when f(Rw, R) = log P(w | Dk).The paper notes this slightly abuses the notation by letting f depend on Dk.Source: §VII · p. 4
- Common active reward-learning acquisition functions do not satisfy adaptive submodularity, so the stated guarantee does not generally apply to them.The paper notes possibilities among adaptive-submodular objectives in active learning.Source: §VII · p. 5
VIII. EXPERIMENTS
The experiments cover three domains and use simulated preference responses from a probabilistic human model.
VIII. EXPERIMENTS
- The evaluation spans a synthetic environment, simulated Assistive Gym robotics, and an NLP task using Reddit-curated datasets.Simulated human responses follow a probabilistic model used in prior work.Source: §VIII · p. 5
A. Human Response Model
The experiments model simulated human preference answers with a Boltzmann-rational response model.
A. Human Response Model
- The human response model is Boltzmann rational, parameterized by β and tuned so about 95% of simulated answers align with the reward function.The model gives response probabilities conditioned on the reward function.Source: A. Human Response Model · p. 5
B. Metrics
The method variants are evaluated against alignment metrics, and in the synthetic experiment πLL and πρ outperform πMI on both reported measures.
B. Metrics
- The variants pair loglikelihood, EPIC distance, and ρ-projection distance with their corresponding alignment objectives.The paper expects each algorithm variant to perform best under its corresponding metric.Source: B. Metrics · p. 5
A. Synthetic Environment
- πρ outperforms all other methods on ρ-projection alignment, while πLL outperforms all others on loglikelihood.The comparison is reported over 50 seeds (mean±se).Source: Figure 1 · p. 5, A. Synthetic Environment · p. 5
- The synthetic comparison uses 50 randomly generated true reward parameters and excludes πEPIC because the data lack its required state-action-next-state detail.The reward is a linear combination of features.Source: A. Synthetic Environment · p. 5
B. Assistive Gym
In Assistive Gym, the methods are evaluated on learning a nonlinear reward that transfers from a Sawyer arm to a Jaco arm. The alignment-based policies outperform mutual information across three alignment measures.
B. Assistive Gym
- The task learns a spoon-placement goal for a Jaco arm from preference queries in a Sawyer-arm source domain.The reward is the negative distance between the end-effector and the goal position, which is unknown to the robot.Source: B. Assistive Gym · p. 5
- πLL, πEPIC, and πρ outperform πMI on log-likelihood, rho-projection alignment, and EPIC-distance alignment for the target domain.Figure 2 reports results over 100 seeds (mean±se).Source: B. Assistive Gym · p. 5, Figure 2 · p. 6
- The inefficiency of πMI on this simple nonlinear reward suggests that deployment-environment information matters for more complex active-learning problems.Source: B. Assistive Gym · pp. 5–6
C. NLP Task
In the NLP task, preference learning on r/askvet is evaluated for alignment with preferences in r/askphilosophy. The paper reports advantages over πMI and concludes that its method learns both linear and nonlinear rewards, while identifying future extensions beyond Bayesian learning.
C. NLP Task
- The task learns a writing-quality reward from r/askvet preferences and checks alignment with true preferences in r/askphilosophy.Comments are represented using sentiment, readability, and relevance features; queries are restricted to comments on the same main post.Source: C. NLP Task · p. 6
- πρ and πLL outperform πMI on log-likelihood and ρ-projection alignment in the NLP task; πEPIC is excluded because granular trajectory information is unavailable.Source: C. NLP Task · p. 6
- πLL also outperforms πρ under the ρ alignment metric, although not early in training; the authors suggest greedy approximation or domain-specific metric fit may explain this.Source: C. NLP Task · p. 6
X. CONCLUSION
- The conclusion reports advantages over the state-of-the-art baseline for learning both linear and nonlinear rewards using three alignment metrics.Source: §X · p. 6
- The framework is Bayesian; future work may extend it to gradient-based learning for rewards with many parameters, such as deep neural networks.Source: §X · p. 6
APPENDIX
The appendix derives Equation (10) by expanding expectations, applying conditional-independence relationships, and rewriting the result as expectations.
A. Derivation of Equation (10)
- The derivation starts from Equation (9) and expands expectations over the entire parameter space for w.Source: A. Derivation of Equation (10) · p. 6
- Bayes rule and the stated conditional independences replace the posterior term with factors involving the prior posterior and query likelihoods.Source: A. Derivation of Equation (10) · p. 6
- The derivation rewrites the integrals as expectations and substitutes the resulting expression into the objective to obtain Equation (10).Source: A. Derivation of Equation (10) · pp. 6–7