Source-linked AI summary
Active Reward Learning from Multiple Teachers
Peter Barnett, Rachel Freedman, Justin Svegliato, Stuart Russell
TL;DR
Reward learning uses human feedback to infer objectives, but commonly treats feedback as coming from one teacher even when multiple teachers with different rationality levels are queried. This paper models that variation and actively selects teachers by expected value of information. It reports eventual belief convergence and finds that less rational teachers can sometimes be more informative than more rational ones.
Problem
Reward learning systems often assume one feedback source despite querying multiple teachers whose expertise, focus, and intelligence can vary, making accurate human decision modeling important.
Source: §1 · p. 1Method
Results
The learner’s posterior converges to the true reward under random queries to Boltzmann-rational teachers with positive, finite β, and a more rational teacher does not always yield faster convergence.
Source: §5 · p. 5Takeaways & Limitations
Less rational teachers can be more informative because their mistakes can reveal the magnitude of reward differences between alternatives.
Source: §1 · p. 1, §9 · p. 7The analysis assumes Boltzmann-rational teachers with constant, known β values and does not account for systematic biases in human judgment.
Source: §8 · p. 7
Abstract
from arXiv · showhide
Reward learning algorithms utilize human feedback to infer a reward function, which is then used to train an AI system. This human feedback is often a preference comparison, in which the human teacher compares several samples of AI behavior and chooses which they believe best accomplishes the objective. While reward learning typically assumes that all feedback comes from a single teacher, in practice these systems often query multiple teachers to gather sufficient training data. In this paper, we investigate this disparity, and find that algorithmic evaluation of these different sources of feedback facilitates more accurate and efficient reward learning. We formally analyze the value of information (VOI) when reward learning from teachers with varying levels of rationality, and define and evaluate an algorithm that utilizes this VOI to actively select teachers to query for feedback. Surprisingly, we find that it is often more informative to query comparatively irrational teachers. By formalizing this problem and deriving an analytical solution, we hope to facilitate improvement in reward learning approaches to aligning AI behavior with human values.
1. Introduction
Reward learning can infer objectives from human feedback, but modeling feedback from multiple teachers as one source risks overlooking meaningful differences in their expertise and rationality. The paper develops a teacher-selection approach that uses those differences to choose informative feedback.
1. Introduction
- Reward learning can infer objectives from human feedback, helping address tasks whose rewards are difficult to specify manually.Misspecified objectives can produce useless or dangerous outcomes.Source: §1 · p. 1
- Treating multiple teachers as one feedback source disregards differences in expertise, focus, and intelligence and increases the risk of human-model misspecification.The paper notes that reward-learning systems may query multiple teachers to gather sufficient feedback.Source: §1 · p. 1
- The proposed Bayesian algorithm actively selects a teacher using feedback noisiness and the learner’s current belief; a less rational teacher can reveal the size of a reward gap through occasional mistakes.A perfectly rational teacher consistently selects the higher-valued alternative, but does not reveal how large the value difference is.Source: §1 · p. 1
- The paper formalizes value-of-information-based teacher selection and reports eventual belief convergence, greater informativeness from less rational teachers in some cases, and better performance than simple heuristics.The stated heuristic comparison includes always querying the most rational teacher.Source: §1 · p. 2
2. Related Work
Reward learning infers reward functions from human behavior or feedback, with preference comparisons a focus because of their reported success across domains. This section situates that approach among other feedback modalities and reward-learning applications.
2. Related Work
- Preference-based reward learning fits a model to paired comparisons and uses the learned reward to guide system behavior.Inverse reinforcement learning can instead infer rewards from observed human behavior or policies.Source: §2 · p. 2
- Preference comparisons have been applied in continuous control and language generation, while reward learning also uses demonstrations, natural language, numeric values, corrections, and proxy rewards.The paper focuses on preference comparisons because of their recent success.Source: §2 · p. 2
Active Reward Learning
Prior active preference-learning work selects which queries to send to teachers, while this paper focuses on choosing the teacher. Its teacher model treats people as Boltzmann-rational, with different rationality levels.
Active Reward Learning
- Active query selection matters because human feedback is expensive and time-consuming to generate.Prior work includes Bayesian query synthesis and efficient approximations, as well as empirical comparisons of query-selection heuristics.Source: Active Reward Learning · p. 2
- Earlier active-learning approaches select preference queries, whereas this work selects which teacher receives a query.The cited approaches include posterior-volume removal and uncertainty-based sampling strategies.Source: Active Reward Learning · p. 2
- The paper models teachers as Boltzmann-rational, following prior work on noisy or probabilistic human decision-making.Earlier models also considered perfect rationality, pedagogic behavior, and systematic bias.Source: Active Reward Learning · p. 2
3. Active Reward Learning
The method represents items by features, models teachers’ preference choices probabilistically, and updates a Bayesian belief over reward weights. It selects teachers by expected improvement in belief accuracy while assuming queries are sampled uniformly at random for the analysis.
Active Reward Learning
- The method samples item pairs and selects a teacher using the current reward belief, then uses that teacher’s preference to update the reward model.The selection process repeats as the learner seeks convergence to the correct reward model.Source: §3 · pp. 2–3
- For the analysis, item pairs are sampled uniformly at random; existing query-selection methods can be combined with the teacher-selection approach.Query selection means choosing which items to present to the teacher.Source: §3 · p. 3
- Each item is represented by a feature vector, and its reward is the dot product of that vector with the learned weight vector.Items without a feature representation can use one-hot vectors, with weights corresponding to item rewards.Source: §3 · p. 3
- Teachers have known rationality parameters β; higher β makes them more likely to choose the higher-reward item, especially when the reward gap is larger.The preference model is logistic, and the learner updates its weight belief by Bayes’ theorem after observing a comparison.Source: §3 · p. 3
- Belief accuracy is measured with mean squared error (MSE) and log loss (LL), for which lower scores indicate a more accurate distribution.MSE measures distance from the true belief, while LL measures the belief distribution’s height at the true value.Source: §3 · p. 4, Table 1 · p. 4
- Teacher selection seeks the best immediate expected improvement in a belief-accuracy metric, using expected scores over the learner’s current belief and possible teacher preferences.The method chooses the teacher with the lowest expected metric score; the paper notes that greater belief-volume removal need not mean greater accuracy.Source: §3 · p. 4
4. Teacher Selection
The algorithm estimates each teacher’s expected metric score using the learner’s current reward belief and the teacher’s preference probabilities, then selects the teacher with the lowest score. It updates the reward belief from that teacher’s preference and repeats until entropy falls below a specified threshold.
4. Teacher Selection
- The expected metric averages over the learner’s current reward belief and each teacher’s probability of preferring either query item.The metrics measure similarity between the posterior belief and the ground-truth reward; lower scores indicate improvement.Source: §4 · p. 4
- The learner selects the teacher whose expected metric score is lowest, using the current belief distribution to evaluate each teacher.The selected rationality parameter is the one expected to produce the largest reduction in the metric.Source: §4 · p. 4
- After generating a paired-item query, the learner asks the selected teacher for a preference and uses that response to update its reward-function belief.The algorithm begins with an initial belief distribution over reward-function parameterizations.Source: §4 · p. 4
- The algorithm repeats until the entropy of the reward belief falls below the specified threshold 𝜖.The procedure uses the expected metric function and teachers’ Boltzmann rationality parameters.Source: §4 · p. 5
5. Theoretical Analysis
The analysis proves posterior convergence under positive finite teacher rationalities, then shows that the most informative teacher can depend on the learner’s prior beliefs.
Convergence
- As N→∞, random queries to Boltzmann-rational teachers with positive, finite β make the posterior over w converge to the true value.The proof shows that for positive finite β, each expected term is positive and the accumulated sum diverges, driving posterior probability to zero for every w ≠ w_true.Source: §5 · p. 5
Teacher informativeness
- A lower-β teacher can be more informative because mistakes reveal the magnitude of w⊤𝜙𝑖𝑗, unlike perfectly rational choices that always select the higher-valued item.The comparison depends on the learner’s current belief, rather than rationality alone.Source: §5 · pp. 5–6
- For Gaussian priors, larger β is better when μ = 0 or σ is large, while narrow asymmetric beliefs can make lower-β teachers more informative.Figure 2 maps the most informative β under MSE and log loss across prior means μ and spreads σ; purple regions indicate β≈1.Source: §5 · pp. 5–6, Figure 2 · p. 6
6. Restaurant Recommendation
The restaurant recommendation example applies the teacher-selection approach to learning user preferences from pairwise comparisons, with known user rationality parameters.
Scope
- The authors state that the approach is compatible with recommendation tasks beyond restaurants, including entertainment, news, and shopping.Source: §6 · p. 6
Domain
- The example represents restaurants with cleanliness, vegan-friendliness, and spiciness features, and learns reward weights from multiple users’ preferences.The reward for each restaurant is expressed as w⊤𝜌𝑖.Source: §6 · p. 6
Application
- For each randomly selected restaurant pair, the approach chooses the user expected to yield the lowest metric score after one update, then updates the belief from that user’s preference.The process repeats until the belief distribution converges.Source: §6 · p. 6
7. Experiments
Active teacher selection outperforms the evaluated baselines in reward inference. The most informative teacher’s rationality decreases as the learner’s belief becomes more confident.
7. Experiments
- The experiment compares 21 teachers with β values from 0 to 4 over 100 steps, using randomly generated restaurant features and the same feature vectors across methods.The discretized weight domain is W = [−10, 10]^3, with a uniform prior and a newly sampled true weight for each of 100 runs.Source: §7 · p. 6
- Equation 8 selection outperforms choosing the largest β, random β, or β = 1 on expected mean squared error and expected log loss.MSE-based selection most effectively minimizes MSE, while log-loss-based selection most effectively minimizes log loss.Source: §7 · pp. 6–7, Figure 3 · p. 7
- The selected β decreases as queries accumulate: larger β values help determine preference signs early, while smaller β values help determine preference magnitudes later.The decrease accompanies a narrower belief distribution; the most informative β is averaged across 100 runs for MSE and log loss.Source: §7 · p. 7, Figure 3 · p. 7
8. Limitations and Future Work
The analysis uses relatively simple human decision-making and reward models, with teachers assumed to have constant, known rationality parameters. The authors identify richer human models, unknown teacher parameters, nonlinear rewards, and differing teacher reward functions as future directions.
8. Limitations and Future Work
- The paper characterizes its human decision-making and reward models as relatively simple, and recommends increasing model complexity in future work.The authors frame this simplification as serving conceptual clarity and mathematical formalism.Source: §8 · p. 7
- The analysis assumes Boltzmann-rational teachers with constant, known β values, a model that does not account for systematic judgment biases [40].The authors suggest allowing β to vary across the state space or explicitly modeling cognitive biases.Source: §8 · p. 7
- The most informative β decreases as the learner’s belief distribution over w becomes more confident.Figure 4 reports the trend averaged across 100 runs for expected mean squared error and expected log loss.Source: Figure 4 · p. 7
- The learner is assumed to have the teachers’ β parameters, although an agent may not have access to them in practice.The authors call for methods that model this uncertainty alongside reward learning.Source: §8 · p. 7
- The authors identify nonlinear reward models, such as neural-network ensembles, as an extension beyond the models studied.They also propose exploring convergence and querying when teachers have different reward functions, including differing restaurant preferences.Source: §8 · p. 7
9. Conclusion
The paper concludes that active teacher selection can improve reward learning when multiple teachers have varying rationality. Its algorithm selects the teacher expected to be most informative, and the authors prove and demonstrate eventual belief collapse to the true reward function.
9. Conclusion
- The algorithm queries the teacher expected to be most informative, and that teacher is not always the most rational one.Teachers are modeled as Boltzmann-rational with known β parameters.Source: §9 · p. 7
- Under the algorithm, the reward learner’s belief eventually collapses to the true reward function, a result the authors prove and demonstrate.Source: §9 · p. 7
- The proposed method and analysis are intended for reward learning where feedback comes from multiple teachers with varying rationality.Source: §9 · p. 7