Source-linked AI summary
Asking Easy Questions: A User-Friendly Approach to Active Reward Learning
Erdem Bıyık, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, Dorsa Sadigh
TL;DR
Robots can learn a user’s reward preferences by asking questions, but queries optimized for robot uncertainty may be difficult for people to answer. This paper selects questions by balancing robot information gain with human answer confidence and derives a stopping rule for query costs. Simulations and a user study report easier questions and faster reward learning.
Problem
Reward-learning queries optimized for robot uncertainty may not account for whether humans can answer them confidently.
Source: §1 · p. 1Method
The approach maximizes information gain to balance robot uncertainty about responses with the human’s uncertainty when choosing, and stops asking when expected value no longer outweighs query cost.
Source: §1 · p. 2, §5.1 · p. 4Results
The approach generated easier questions and was more data-efficient than state-of-the-art methods in simulations and a user study.
Source: §8 · p. 8Takeaways & Limitations
Greedy information gain and optimal stopping are presented as ways to ask a right number of easy, informative reward-learning questions.
Source: §8 · p. 8The approach’s ability to identify questions humans can answer confidently depends on the accuracy of its cognitive model of the human.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Robots can learn the right reward function by querying a human expert. Existing approaches attempt to choose questions where the robot is most uncertain about the human's response; however, they do not consider how easy it will be for the human to answer! In this paper we explore an information gain formulation for optimally selecting questions that naturally account for the human's ability to answer. Our approach identifies questions that optimize the trade-off between robot and human uncertainty, and determines when these questions become redundant or costly. Simulations and a user study show our method not only produces easy questions, but also ultimately results in faster reward learning.
1 Introduction
Personalized reward learning can use questions, but queries optimized only for robot uncertainty may be indistinguishable to the human. The paper accounts for human answer confidence, balances it with information gain, and reports easier questions and faster reward learning.
1 Introduction
- Volume removal can ask humans to distinguish nearly indistinguishable options, such as trajectories or speeds of 4.9 versus 5.1 m/s.Figure 1 contrasts those questions with the easier-to-answer questions generated by the proposed approach.Source: §1 · p. 1, Figure 1 · p. 2
- Information gain balances information from a correct answer against the human’s ability to answer confidently, and naturally favors questions users can answer confidently regardless of their individual preferences.The authors report that this approach has the same computational complexity as volume removal within their settings.Source: §1 · pp. 1–2
- The method derives an optimal stopping condition to ask questions only while their expected value outweighs their cost, with extensions for personalized, query-dependent costs.The stated objective is to minimize the number of queries the user must answer.Source: §1 · p. 2
- Across simulations, information-gain questions led to faster robot learning than volume removal, and users reported easier questions in simulation and Fetch-robot experiments.The approach asked particularly easy questions at the start of learning.Source: §1 · p. 2
2 Related Work
Related work covers reward learning from demonstrations and preferences, active query selection, and approaches to user-friendly questions. The paper combines these concerns by focusing on query selection that is both informative and easy to answer.
2 Related Work
- Inverse reinforcement learning recovers rewards from expert demonstrations, while preference-based learning asks people to choose or rank candidate trajectories as a more user-friendly alternative.Demonstrations can be difficult to provide, particularly for robots with many degrees of freedom.Source: §2 · p. 2
- Active learning selects informative queries to improve data efficiency; volume removal is popular in preference learning, while information gain is used in other active-learning settings.The paper builds on both approaches with a focus on asking easy questions.Source: §2 · p. 2
- Prior work addresses difficult questions, human uncertainty, and the time or effort cost of queries through natural questions, an “About Equal” option, or stopping rules.The paper presents its approach as incorporating all three challenges.Source: §2 · p. 2
3 Problem Formulation
The problem formulation represents the human’s reward as a linear function of trajectory features and aims to learn its parameters using as few preference queries as possible. It models the human’s responses probabilistically and updates a reward-parameter distribution after each response.
3 Problem Formulation
- The system is modeled as deterministic and fully observable, with states, actions, and finite trajectories over a time horizon.For a deterministic system, a trajectory can be represented by its initial state and sequence of actions.Source: §3 · pp. 2–3
- The human’s reward is a linear combination of trajectory features, so learning the reward reduces to learning its parameter vector.The reward is written as R(ξ) = ω⊤Φ(ξ).Source: §3 · p. 3
- A preference query is a set of K trajectories from which the human selects the most preferred, and the robot learns the reward parameter from a sequence of such queries..Source: §3 · p. 3
- The goal is to maximize expected information about the reward parameter while limiting query count and accounting for answer difficulty; the general problem is NP-hard.The paper therefore alternates greedily between generating a query and updating the parameter distribution from the human’s response.Source: §3 · p. 3
4 Volume Removal Solution
Volume removal selects queries by maximizing an expected change between the prior and unnormalized posterior, but its formulation has a trivial global optimum and can favor difficult choices. The authors note that non-convex optimization can lead practical solvers to more informative local optima.
4 Volume Removal Solution
- Volume removal maximizes the expected difference between the prior and unnormalized posterior to select a query of K trajectories.Computing expectations over the complex reward-parameter distribution requires sampling.Source: §4 · p. 3
- The volume-removal formulation’s global solution is a query containing K identical trajectories, which provides no information about the human’s preferences.The authors explain that practical methods can instead converge to more informative local optima because the optimization problem is non-convex.Source: §4 · p. 3
- Even when the options are not identical, volume removal may select difficult queries whose alternatives are equally plausible to the human.The paper contrasts a query with equal response probabilities across reward parameters against one where the response is nearly certain for each parameter region.Source: §4 · p. 4
5 Information Gain Solution
The paper selects preference queries by maximizing information gain, balancing what the robot can learn with how confidently the human can answer. The approach avoids volume removal’s identical-options failure while retaining comparable computational complexity and greedy-query guarantees.
5 Information Gain Solution
- Information gain selects queries that trade off information about the reward parameter against the human’s ability to answer confidently.The objective considers both the robot’s uncertainty about the human’s response and the human’s uncertainty when choosing an option.Source: §5 · p. 4, §5.1 · pp. 4–5
- Unlike volume removal, information gain makes identical-trajectory queries a global minimum rather than a global solution, with equivalent-order computational complexity.Source: §5 · p. 4
5.2 Theoretical Guarantees
- The mutual-information objective is monotone submodular, which supports a greedy-query guarantee relative to the optimal sequence.Source: §5.2 · p. 5
5.3 Extensions
- Batch optimization and warm-starting extensions can be applied because they require greedy query generation and a distribution over the reward parameter.Source: §5.3 · p. 5
6 Optimal Stopping
The robot subtracts query cost from expected information gain, favoring informative, low-cost questions. It stops when the global objective is negative, meaning query cost outweighs value.
6 Optimal Stopping
- Each query receives a cost that can represent answer time, repeated similar questions, or question interpretability.Subtracting this cost from information gain biases query selection toward low-cost questions.Source: §6 · p. 5
- The robot should stop asking questions if and only if the global solution to the cost-adjusted information-gain objective is negative.At each timestep, it asks another query when the solution is nonnegative; otherwise, it stops.Source: §6 · p. 5
7 Experiments
The experiments compare information gain with volume removal through simulated reward learning and user studies, using strict and weak preference queries. They evaluate learning rate, answer difficulty, and learned trajectories.
7.1 Human Choice Model
- Weak preference queries add an “About Equal” response, modeled with a minimum perceivable-difference parameter δ; δ = 0 recovers the strict-choice model.The model can also allow δ to be user-specific and unknown, while extending weak queries to more than two trajectories complicates the choice model.Source: §7.1 · p. 6
7.2 Simulations
- Information gain significantly improved learning rate over volume removal across four tasks, with both strict and weak preference queries.The simulations used 100 randomly sampled reward parameters; “About Equal” improved volume removal but was not required for information gain’s improvement.Source: §7.2 · p. 6, Figure 3 · p. 7
- Information gain reduced wrong answers, especially early in the query sequence, while weak preference queries consistently reduced wrong answers relative to strict queries.Information gain was slightly better than uniformly random querying on wrong answers and “About Equal” choices.Source: Figure 4 · p. 7, §7.2 · p. 7
- Optimal stopping retained near-optimal cumulative active-learning rewards across environments.The simulation averaged cumulative rewards over 100 test runs and recorded when the stopping condition was met.Source: Figure 5 · p. 7, §7.2 · p. 7
7.3 User Studies
- Users significantly preferred trajectories optimized with information-gain-learned rewards over those from volume removal in both study environments.The comparison used a one-sample t-test (p < 0.05).Source: Figure 6 · p. 8, §7.3 · p. 8
- Users rated information-gain queries easier in all environments, and volume removal produced more indistinguishable options in Driver and Tosser.Easiness differences were significant (two-sample t-test, p < 0.005); indistinguishable-option differences were significant for Driver (paired-sample t-test, p < 0.05) and Tosser (p < 0.005).Source: Figure 6 · p. 8, §7.3 · p. 8
- Each participant answered 30 actively generated questions—15 per method—in randomized order and rated query ease and trajectory preference.The ease survey used a 7-point scale; participants also compared trajectories optimized using the two learned reward estimates.Source: §7.3 · p. 8
8 Conclusion
The paper concludes that information gain can produce easy, informative reward-learning queries, while optimal stopping limits costly or redundant questions. The approach’s ability to identify answerable questions is constrained by the accuracy of its human cognitive model.
8 Conclusion
- Greedy information-gain queries trade off robot and human uncertainty, and optimal stopping accounts for questions that become prohibitively expensive.Simulations and a user study found the approach more data-efficient and its queries easier, as judged by humans, than state-of-the-art methods.Source: §8 · p. 8
- The approach’s ability to identify questions users can answer confidently depends on cognitive-model accuracy.The authors identify learning and personalizing that model for the current user as future work.Source: §8 · p. 8
9 Appendix
The appendix extends the information-gain analysis to unknown user parameters, continuous action spaces, and additional query and stopping conditions. Its simulations report faster learning with information gain and weak-preference queries, and near-optimal cumulative rewards under the tested stopping rule.
9.1 Derivation of Information Gain Solution
- The information-gain objective is the difference between expected log-posterior terms before and after a query, with integration over possible ω values.The appendix derives this objective as the information-gain solution.Source: §9.1 · p. 11
9.2 Proof of Theorem 3
- Terminating is optimal if and only if the global solution to (6) is negative.The proof uses submodularity of mutual information to show that a longer horizon has negative expected reward when the global optimum is negative.Source: §9.2 · p. 11
9.3 Extension to User-Specific and Unknown δ
- The alternative of jointly learning (ω, ν) is parallelizable, but causes performance loss because the learning goal is ω alone.The formulation can be derived by replacing ω with (ω, ν).Source: §9.3 · p. 12
- With unknown user parameter δ, weak-preference queries still outperform strict queries, though computation is heavier; jointly learning (ω, ν) loses performance but is parallelizable.The heavier objective takes M reward-parameter samples for each ν sample; joint learning incurs performance loss because the target is ω.Source: §9.3 · p. 12
9.4 Feature Transformations and Action Spaces of the Environments
- Table 1 gives environment details; the user-study Fetch task removes average speed and moves directly to the final position instead of following 19 control inputs.The appendix also includes simulation results for information gain with unknown δ.Source: §9.4 · p. 12, Figure 7 · p. 13, Table 1 · p. 14
9.5 Comparison of Models without Query Space Discretization
- Without query-space discretization, information gain outperforms volume removal with both query types, and weak-preference queries learn faster than strict-preference queries.The comparison covers LDS, Driver, and Tosser; Fetch is excluded because trajectory optimization over its large action space is prohibitive.Source: §9.5 · p. 13, Figure 8 · p. 13
9.6 Effect of Information from “About Equal” Responses
- “About Equal” responses improve learning in Driver, Tosser, and Fetch for both formulations, while their effect is very small in LDS.The simulations compare using those responses with ignoring them and removing the query to prevent repetition.Source: §9.6 · p. 13, Figure 9 · p. 13
9.7 Optimal Stopping under Query-Independent Costs
- Under query-independent costs, the optimal stopping rule achieves near-optimal cumulative active-learning rewards in all environments.The tests use a cost c(Q) = ϵ tuned from simulated users.Source: §9.7 · p. 13, Figure 10 · p. 14